Computational Psycholinguistics, Fall 2021
1 Class information
| Lecture Times | Mondays & Wednesdays 9:30–11:00am |
| Recitation Time | Fridays 1-2pm |
| Lecture & Recitation Location | 46-3189 |
| Class website | https://canvas.mit.edu/courses/11393 |
| Syllabus | http://www.mit.edu/~rplevy/teaching/2021fall/9.19 |
2 Instructor information
| Instructor | Roger Levy (rplevy@mit.edu) |
| Instructor's office hours | Wednesdays 11am-12pm |
| Teaching Assistants | Rujul Gandhi (rujul@mit.edu); Hang Jiang (hjian42@mit.edu); Adnan Rebei (rebeia@mit.edu); Mycal Tucker (mycal@mit.edu); |
| TA Office Hours | Hang: Wednesdays 3-4pm @ 46-4199; Adnan: Thursdays 5-6pm @ TBD; Rujul: Fridays 11am-12pm @ TBD; Mycal: Tuesdays 11am-12pm @ TBD |
3 Class Description
Over the last two and a half decades, computational linguistics has been revolutionized as a result of three closely related developments: increases in computing power, the advent of large linguistic datasets, and paradigm shifts toward probabilistic modeling and deep learning. At the same time, similar theoretical developments in cognitive science have led to a view of major aspects of human cognition as instances of rational statistical inference. These developments have set the stage for renewed interest in computational approaches to how humans use language. Correspondingly, this course covers some of the most exciting developments in computational psycholinguistics over the past decade. The course spans human language comprehension, production, and acquisition, and covers key phenomena spanning phonetics, phonology, morphology, syntax, semantics, and pragmatics. Students will learn technical tools including probabilistic models, formal grammars, neural networks, and decision theory, and how theory, computational modeling, and data can be combined to advance our fundamental understanding of human language acquisition and use.
4 Class organization
The fall 2021 edition of 9.19/9.190 will operate as a "flipped" classroom. We have pre-recorded lecture videos by the instructor available in advance of the class meeting time; part of your out-of-class work will involve watching the video for the upcoming class and preparing questions about it. In-class activities will include in-class exercises, reviewing answering questions about lecture and exercise content, and open discussion.
We also have a weekly recitation session led by TAs, which will review relevant material on programming, linguistics, probability, and machine learning.
5 Intended Audience
Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. The undergraduate section is 9.19, the graduate section is 9.190. Postdocs and faculty are also welcome to participate!
The course prerequisites are:
- One semester of Python programming (fulfillable by 6.00/6.0001+6.0002, for example), plus
- Either:
- one semester of probability/statistics/machine learning (fulfilled by, for example, 6.041B or 9.40), or
- one semester of introductory linguistics (fulfilled by 24.900 or equivalent).
If you think you have the requisite background but have not taken the specific courses just mentioned, please talk to the instructor to work out whether you should take this course or do other prerequisites first.
We will be doing Python programming in this course, and also using programs that must be run from the Unix/Linux/OS X command line. If you have less than one semester of Python programming experience and/or would like to strengthen your Python programming background, I have listed some resources recommended by others here.
If you are unsure about whether this class is a good fit for you given your background and interests, please consult the class FAQ; if you are still unsure, please contact Roger.
6 Well-being and flexibility in teaching during COVID
The 9.19 instructional staff acknowledge that we're teaching in an extraordinary time, and that there are many sources of uncertainty and fatigue. We are committed to providing you a supportive, stable, and safe educational environment during this time. In addition to the pre-recorded that we're working with, live class sessions will be recorded and made available afterwards.
If you are notified that you must isolate or quarantine because of a COVID-related issue – please know that we am committed to working with S3 or GradSupport to support your continued learning this semester. we understand how difficult these times are and we will all need to remain flexible and adjust to the situation. The details of what may be required will vary depending on the timing and duration of your absence. Please feel free to reach out to us at any point if you have any questions or concerns.
7 Readings & Textbooks
Readings will frequently be drawn from the following textbooks:
Daniel Jurafsky and James H. Martin. Speech and Language Processing. Draft chapters for the third edition can be found here. (I refer to this book as "SLP3" in the syllabus.)
This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics. Note: we will also be using some chapters from the second edition (2008); these will be referred to as J&M2008 on the syllabus, and will be available on Stellar.
Jacob Eisenstein. 2019. Introduction to Natural Language Processing. MIT Press. Pre-publication PDF available here.
This is an excellent recent textbook on natural language processing, with contemporary deep learning thoroughly integrated. We will be using the pre-publication version freely available, under the Creative Commons CC BY-NC-ND license, here.
Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media. (I refer to this book as "NLTK" in the syllabus.)
This is the book for the Natural Language Toolkit (or NLTK), which we will be using extensively to do programming We will also be doing some of our programming in the Python programming language, and will make quite a bit of use of Python. You can buy this book, or you can freely access it on the Web at http://www.nltk.org/book.
Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "M&S" in the syllabus.)
This is an older but still very useful book on natural language processing (NLP).
We'll also occasionally draw upon other sources for readings, including original research papers in computational linguistics, psycholinguistics, and other areas of the cognitive science of language.
8 Syllabus (subject to modification!)
| Week | Day | Topic | Readings | Related readings | Problem sets |
|---|---|---|---|---|---|
| Week 1 | Wed 8 Sep | Course intro; intro to probability theory | M&S 2.1 | . Goldsmith, 2007; Clayards et al., 2008 | Pset 1 out |
| Fri 10 Sep | Recitation 1: Probability theory | ||||
| Week 2 | Mon 13 Sep | Speech perception and introductory rational analysis | Selection from Anderson 1990; PMSL 5.2.4 | ||
| Wed 15 Sep | Word sequences, language models and $N$-grams | . SLP3 3 | |||
| Fri 17 Sep | Recitation 2: Python programming | ||||
| Week 3 | Mon 20 Sep | Psycholinguistic methods; prediction in human language understanding; surprisal theory | . Kutas et al. 2011; TBD | . McMurray et al., 2008; Smith & Levy, 2013; Rayner, 1998; Dahan et al., 2001 | Pset 2 out |
| Wed 22 Sep | Regular expressions and finite-state machines | J&M2008 2.1-2.4, 3; Eisenstein, 2018, Section 9.1 | Pset 1 due | ||
| Fri 24 Sep | Recitation 3: Regular expressions in practice | ||||
| Week 4 | Mon 27 Sep | Is human language finite state? | SLP3 11; J&M2008 16 (under Readings on Stellar); | . Chomsky, 1956 | Pset 3 out |
| Wed 29 Sep | Context-free grammars; syntactic analysis. | SLP3 12; Eisenstein, 2018, Section 9.2; NLTK 8.1-8.5; Levy & Andrew, 2006 | . Gazdar, 1981 (esp. Section 2); Müller, 2018 (esp. Section 5.4); Joshi et al., 1991 (on formalisms beyond context-free) | ||
| Fri 1 Oct | Recitation 4: Linguistics fundamentals | ||||
| Week 5 | Mon 4 Oct | Context-free grammars and syntactic analysis continued. | Pset 2 due | ||
| Wed 6 Oct | Probabilistic context-free grammars (PCFGs), incremental parsing, human syntactic processing | SLP3 10 & 13; NLTK 8.6; Levy, 2013 | . Jurafsky, 1996; Hale, 2001; Levy, 2008 | Pset 4 out | |
| Fri 8 Oct | Recitation 5: Writing grammar fragments | ||||
| Week 6 | Mon 11 Oct | Indigenous Peoples' Day, no class | |||
| Wed 13 Oct | Theory, models, and data for human language comprehension | Pset 3 due | |||
| Fri 15 Oct | Recitation 6: Designing experiments & interpreting experimental data | ||||
| Week 7 | Mon 18 Oct | Bayes Nets; the perceptual magnet | Russell & Norvig, 2010, chapter 14 (on Stellar); Feldman & Griffiths, 2006; Levy in progress, Directed Graphical Models appendix; | Pset 5 out | |
| Wed 20 Oct | Multi-factor models: logistic regression; word order preferences in language. Hierarchical models; binomial construction. | SLP3 5; Graphical models intro ; Morgan & Levy, 2015 | . Bayes Nets lecture notes; Kraljic et al., 2008 | ||
| Fri 22 Oct | Recitation 7: Midterm review | Pset 4 due | |||
| Week 8 | Mon 25 Oct | Midterm exam (take-home, 24 hours, designed to take no more than 80 minutes) | |||
| Wed 27 Oct | Logistic regression review; basic multi-layer neural networks | SLP3 7.1-7.4; Eisenstein, 2018, 3.1-3.3 | . Mikolov et al., 2013, Pennington et al., 2014, Gutierrez et al., 2016 | ||
| Fri 29 Oct | Recitation 8: feed-forward neural networks | ||||
| Week 9 | Mon 1 Nov | Word embeddings | SLP3 6; Eisenstein, 2018, Ch. 14; Young et al., 2018, Section IV; | . Goldberg, 2015; Collobert et al., 2011; Levy, Goldberg, & Dagan, 2015; | Pset 6 out |
| Wed 3 Nov | Neural networks for natural language | SLP3 7.5, 9.1-9.4; Eisenstein 2018, 6.3 | Pset 5 due | ||
| Fri 5 Nov | Recitation 8: language modeling with deep learning | ||||
| Week 10 | Mon 8 Nov | What do neural networks learn about language structure – and what don't they learn? | . Linzen & Baroni, 2020 | . Linzen et al., 2016; Gulordava et al., 2018, Linzen 2019 | |
| Wed 10 Nov | Bringing together grammar and deep learning; controlled syntactic evaluation of neural language models | . Dyer et al., 2016; Futrell et al., 2019 | . Choe & Charniak, 2016; Kuncoro et al., 2017; Dyer et al., 2015 | Pset 7 out | |
| Fri 12 Nov | Recitation 9: comparing models with human data | ||||
| Week 11 | Mon 15 Nov | Transformers; filler-gap dependencies | . Vaswani et al., 2017; Sasha Rush's "The Annotated Transformer"; Wilcox et al., 2018 | . Radford et al., 2018; Radford et al., 2019 | Pset 6 due |
| Wed 17 Nov | Noisy-channel language comprehension | . Levy et al., 2009 | |||
| Fri 19 Nov | Recitation 10: how attention works | ||||
| Week 12 | Mon 22 Nov | Noisy-channel language comprehension II | . Gibson et al., 2013; Futrell & Levy, 2017 | Pset 8 out | |
| Wed 24 Nov | Unsupervised learning and native language acquisition I | . Feldman et al., 2013 | Pset 7 due | ||
| Fri 26 Nov | Thanksgiving holiday, no recitation | ||||
| Week 13 | Mon 29 Nov | Unsupervised learning and native language acquisition II | . Saffran et al., 1996; Goldwater, Griffiths, & Johnson, 2009 | ||
| Wed 1 Dec | Computational semantics & pragmatics I | . Frank & Goodman, 2016 | Pset 8 due | ||
| Fri 3 Dec | Recitation 11: hierarchical Bayesian models | 9.190 Class projects due Friday, December 3 | |||
| Week 14 | Mon 6 Dec | Computational semantics & pragmatics II | Reading TBD | ||
| Wed 8 Dec | End-of-semester review | Use this class to review readings from the rest of the semester! | |||
| Final Exam | Mon 13 Dec | Final exam (take-home, 48 hours, designed to take no more than 2 hours) |
9 Requirements & grading
You'll be graded on:
| Work | Grade percentage (9.19) | Grade percentage (9.190) |
| A number of homework assignments throughout the semester | 60% | 48% |
| A midterm exam | 15% | 12% |
| A final exam | 25% | 20% |
| If you are enrolled in 9.190, a class project | -- | 20% |
Active participation in the class is also encouraged and taken into account in borderline grade cases!
9.1 Pset late policy
Psets can be turned in up to 7 days late; 10% of your score will be deducted for each 24 hours of lateness (rounded up, so e.g. you lose 10% of your score for an assignment turned in 12 hours late). For example, if a homework assignment is worth 80 points, you turn it in 3 days late, and earn a 70 before lateness is taken into account, your score will be (1-0.3)*70=49.
9.2 Personal or medical circumstances impacting psets, exams, or projects
If personal or medical circumstances such as illness impact your work on a pset or project, or your ability to take an exam on the scheduled date with adequate preparation, please work with Student Support Services (S3) to verify these circumstances and be in touch with the instructor. We are happy to work with you in whatever way is most appropriate to your individual circumstances to help ensure that you are able to achieve your best performance in class while maintaining your health, happiness, and well-being.
9.3 Mapping of class score to letter grade
We use new homework and exam materials every year, and it can be hard to perfectly predict the difficulty of assignments and exams. Therefore I determine standards for end-of-semester letter grades in light of student performance throughout the class (though I do not grade on a curve). However, I guarantee minimum grades on the basis of the following thresholds:
| Threshold | Guaranteed minimum grade |
| >=90% | A- |
| >=80% | B- |
| >=70% | C- |
| >=60% | D |
So, for example, an overall score of 90.0001% of points guarantees you an A-, but you could well wind up with a higher grade depending on the ultimate grade thresholds determined at the end of the semester.