Computational Psycholinguistics, Spring 2020
1 Course information
| Lecture Times | Mondays & Wednesdays 9:30-11:00am |
| Lecture Location | 46-3310 |
| Class website | https://stellar.mit.edu/S/course/9/sp20/9.19/ |
| Syllabus | http://www.mit.edu/~rplevy/teaching/2020spring/9.19 |
2 Instructor information
| Instructor | Roger Levy (rplevy@mit.edu) |
| Instructor's office | 46-3033 |
| Instructor's office hours | Mondays 2-4pm |
| Teaching Assistants | Jon Gauthier (jgauthie@mit.edu), Jennifer Hu (jennhu@mit.edu), Ethan Wilcox (ethanwil@mit.edu) |
| TA Office | 46-3027C (Jon, Jennifer) 46-3027D (Ethan) |
| TA Office Hours | Tu 9:30-10:30am, 2-3pm (Ethan), 3-5pm (Jenn), W 11am-12pm, 1-2pm (Jon) |
3 Course Description
Over the last two and a half decades, computational linguistics has been revolutionized as a result of three closely related developments: increases in computing power, the advent of large linguistic datasets, and paradigm shifts toward probabilistic modeling and deep learning. At the same time, similar theoretical developments in cognitive science have led to a view of major aspects of human cognition as instances of rational statistical inference. These developments have set the stage for renewed interest in computational approaches to how humans use language. Correspondingly, this course covers some of the most exciting developments in computational psycholinguistics over the past decade. The course spans human language comprehension, production, and acquisition, and covers key phenomena spanning phonetics, phonology, morphology, syntax, semantics, and pragmatics. Students will learn technical tools including probabilistic models, formal grammars, neural networks, and decision theory, and how theory, computational modeling, and data can be combined to advance our fundamental understanding of human language acquisition and use.
4 Course organization
We'll meet twice a week; the course format will be a combination of lecture, discussion, and in-class exercises as class size, structure, and interests permit.
5 Intended Audience
Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. The undergraduate section is 9.19, the graduate section is 9.190. Postdocs and faculty are also welcome to participate!
The course prerequisites are:
- One semester of Python programming (fulfillable by 6.00/6.0001+6.0002, for example), plus
- Either:
- one semester of probability/statistics/machine learning (fulfilled by, for example, 6.041B or 9.40), or
- one semester of introductory linguistics (fulfilled by 24.900).
If you think you have the requisite background but have not taken the specific courses just mentioned, please talk to the instructor to work out whether you should take this course or do other prerequisites first.
We will be doing Python programming in this course, and also using programs that must be run from the Unix/Linux/OS X command line. If you have less than one semester of Python programming experience and/or would like to strengthen your Python programming background, I have listed some resources recommended by others here.
6 Readings & Textbooks
Readings will frequently be drawn from the following textbooks:
- Daniel Jurafsky and James H. Martin. Speech and Language Processing. Draft chapters for the third edition can be found here. (I refer to this book as "SLP3" in the syllabus.)
This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics. Note: we will also be using some chapters from the second edition (2008); these will be referred to as J&M2008 on the syllabus, and will be available on Stellar.
- Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media. (I refer to this book as "NLTK" in the syllabus.)
This is the book for the Natural Language Toolkit (or NLTK), which we will be using extensively to do programming We will also be doing some of our programming in the Python programming language, and will make quite a bit of use of Python. You can buy this book, or you can freely access it on the Web at http://www.nltk.org/book.
- Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "M&S" in the syllabus.)
This is an older but still very useful book on natural language processing (NLP).
We'll also occasionally draw upon other sources for readings, including original research papers in computational linguistics, psycholinguistics, and other areas of the cognitive science of language.
7 Syllabus (subject to modification!)
| Week | Day | Topic | Readings | Related readings | Problem sets |
|---|---|---|---|---|---|
| Week 1 | Mon 3 Feb | Course intro; intro to probability theory; speech perception | M&S 2.1 | Goldsmith, 2007; Clayards et al., 2008 | Pset 1 out |
| Wed 5 Feb | Word sequences, language models and \(N\)-grams | SLP3 3 | |||
| Week 2 | Mon 10 Feb | Psycholinguistic methods; prediction in human language understanding; surprisal theory | Kutas et al. 2011; TBD | McMurray et al., 2008; Smith & Levy, 2013; Rayner, 1998; Dahan et al., 2001 | Pset 2 out |
| Wed 12 Feb | Regular expressions | J&M2008 2.1-2.1.6 | Pset 1 due | ||
| Week 3 | Tue 18 Feb | Finite-state machines | J&M2008 2.2-2.4, 3; Eisenstein, 2018, Section 9.1 | ||
| Wed 19 Feb | Is human language finite state? | SLP3 11; J&M2008 16 (under Readings on Stellar); | Chomsky, 1956 | Pset 3 out | |
| Week 4 | Mon 24 Feb | Context-free grammars; syntactic analysis. | SLP3 12; Eisenstein, 2018, Section 9.2; NLTK 8.1-8.5; Levy & Andrew, 2006 | Gazdar, 1981 (esp. Section 2); Müller, 2018 (esp. Section 5.4); Joshi et al., 1991 (on formalisms beyond context-free) | Pset 2 due |
| Wed 26 Feb | Context-free grammars and syntactic analysis continued. | ||||
| Week 5 | Mon 2 Mar | Probabilistic context-free grammars (PCFGs), incremental parsing, human syntactic processing | SLP3 10 & 13; NLTK 8.6; Levy, 2013 | Jurafsky, 1996; Hale, 2001; Levy, 2008 | Pset 4 out |
| Wed 4 Mar | Bayes Nets; the perceptual magnet | Russell & Norvig, 2010, chapter 14 (on Stellar); Feldman et al., 2009; Levy in progress, Directed Graphical Models appendix; | Bayes Nets lecture notes; Kraljic et al., 2008 | Pset 3 due | |
| Week 6 | Mon 9 Mar | Multi-factor models: logistic regression; word order preferences in language. Hierarchical models; binomial construction. | SLP3 5; Graphical models intro ; Morgan & Levy, 2015 | ||
| Wed 11 Mar | Word embeddings | SLP3 6; Eisenstein, 2018, Ch. 14 Young et al., 2018, Section IV; | Goldberg, 2015; Collobert et al., 2011; Levy, Goldberg, & Dagan, 2015; | ||
| Week 7 | Mon 16 Mar | No class | |||
| Wed 18 Mar | No class | ||||
| Spring Break | 23-27 Mar | Spring break, no class | |||
| Week 8 | Mon 30 Mar | Logistic regression review; basic multi-layer neural networks | SLP3 7.1-7.4; Eisenstein, 2018, 3.1-3.3 | Mikolov et al., 2013, Pennington et al., 2014, Gutierrez et al., 2016 | |
| Wed 1 Apr | Midterm review | Pset 4 due Friday 3 April | |||
| Week 9 | Mon 6 Apr | Midterm exam | Pset 5 out | ||
| Wed 8 Apr | Neural networks for natural language | SLP3 7.5, 9.1-9.4; Eisenstein 2018, 6.3 | |||
| Week 10 | Mon 13 Apr | What do neural networks learn about language structure – and what don't they learn? | Linzen et al., 2016; Gulordava et al., 2018 | Linzen 2019 | Pset 6 out |
| Wed 15 Apr | Bringing together grammar and deep learning; controlled syntactic evaluation of neural language models | Dyer et al., 2016; Kuncoro et al., 2017; Futrell et al., 2019 | Choe & Charniak, 2016; Dyer et al., 2015 | Pset 5 due Friday 17 April | |
| Week 11 | Mon 20 Apr | Patriots Day, no class | |||
| Wed 22 Apr | Transformers; filler-gap dependencies | Vaswani et al., 2017; Wilcox et al., 2018 | Radford et al., 2018; Radford et al., 2019 | Pset 7 out | |
| Week 12 | Mon 27 Apr | Noisy-channel language comprehension | Levy et al., 2009 | Pset 6 due | |
| Wed 29 Apr | Noisy-channel language comprehension II | Gibson et al., 2013; Futrell & Levy, 2017 | |||
| Week 13 | Mon 4 May | Statistical word learning in humans; modeling with nonparametric Bayes | Saffran et al., 1996; Goldwater, Griffiths, & Johnson, 2009 | ||
| Wed 6 May | Computational pragmatics | Frank & Goodman, 2016 | Pset 7 due Thursday May 7 | ||
| Fri 8 May | (no class but:) 9.190 Course projects due | Course projects due Friday, May 8 | |||
| Week 14 | Mon 11 May | End-of-semester review | Use this class to review readings from the rest of the semester! | ||
| Final Exam | Tue 19 May | Final exams to be distributed by 8pm Sunday 5/17 and due 8pm Tuesday 5/19 |
8 Requirements & grading
You'll be graded on:
| Work | Grade percentage (9.19) | Grade percentage (9.190) |
| A number of homework assignments throughout the semester | 60% | 48% |
| A midterm exam | 15% | 12% |
| A final exam | 25% | 20% |
| If you are enrolled in 9.190, a course project | -- | 20% |
Active participation in the class is also encouraged and taken into account in borderline grade cases!
8.1 Homework late policy
Homework assignments can be turned in up to 7 days late; 10% of your score will be deducted for each 24 hours of lateness (rounded up). For example, if a homework assignment is worth 80 points, you turn it in 3 days late, and earn a 70 before lateness is taken into account, your score will be (1-0.3)*70=49.
8.2 Personal or medical circumstances impacting psets, exams, or projects
If personal or medical circumstances such as illness impact your work on a pset or project, or your ability to take an exam on the scheduled date with adequate preparation, please work with Student Support Services (S3) to verify these circumstances and be in touch with the instructor. We are happy to work with you in whatever way is most appropriate to your individual circumstances to help ensure that you are able to achieve your best performance in class while maintaining your health, happiness, and well-being.
8.3 Mapping of class score to letter grade
We use new homework and exam materials every year, and it can be hard to perfectly predict the difficulty of assignments and exams. Therefore I determine standards for end-of-semester letter grades in light of student performance throughout the class (though I do not grade on a curve). However, I guarantee minimum grades on the basis of the following thresholds:
| Threshold | Guaranteed minimum grade |
| >=90% | A- |
| >=80% | B- |
| >=70% | C- |
| >=60% | D |
So, for example, an overall score of 90.0001% of points guarantees you an A-, but you could well wind up with a higher grade depending on the ultimate grade thresholds determined at the end of the semester.