Computational Psycholinguistics, Fall 2021

1 Class information

Lecture Times Mondays & Wednesdays 9:30–11:00am
Recitation Time Fridays 1-2pm
Lecture & Recitation Location 46-3189
Class website

2 Instructor information

Instructor Roger Levy (
Instructor's office hours Wednesdays 11am-12pm
Teaching Assistants Rujul Gandhi (; Hang Jiang (; Adnan Rebei (; Mycal Tucker (;
TA Office Hours Hang: Wednesdays 3-4pm @ 46-4199; Adnan: Thursdays 5-6pm @ TBD; Rujul: Fridays 11am-12pm @ TBD; Mycal: Tuesdays 11am-12pm @ TBD

3 Class Description

Over the last two and a half decades, computational linguistics has been revolutionized as a result of three closely related developments: increases in computing power, the advent of large linguistic datasets, and paradigm shifts toward probabilistic modeling and deep learning. At the same time, similar theoretical developments in cognitive science have led to a view of major aspects of human cognition as instances of rational statistical inference. These developments have set the stage for renewed interest in computational approaches to how humans use language. Correspondingly, this course covers some of the most exciting developments in computational psycholinguistics over the past decade. The course spans human language comprehension, production, and acquisition, and covers key phenomena spanning phonetics, phonology, morphology, syntax, semantics, and pragmatics. Students will learn technical tools including probabilistic models, formal grammars, neural networks, and decision theory, and how theory, computational modeling, and data can be combined to advance our fundamental understanding of human language acquisition and use.

4 Class organization

The fall 2021 edition of 9.19/9.190 will operate as a "flipped" classroom. We have pre-recorded lecture videos by the instructor available in advance of the class meeting time; part of your out-of-class work will involve watching the video for the upcoming class and preparing questions about it. In-class activities will include in-class exercises, reviewing answering questions about lecture and exercise content, and open discussion.

We also have a weekly recitation session led by TAs, which will review relevant material on programming, linguistics, probability, and machine learning.

5 Intended Audience

Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. The undergraduate section is 9.19, the graduate section is 9.190. Postdocs and faculty are also welcome to participate!

The course prerequisites are:

  1. One semester of Python programming (fulfillable by 6.00/6.0001+6.0002, for example), plus
  2. Either:
    • one semester of probability/statistics/machine learning (fulfilled by, for example, 6.041B or 9.40), or
    • one semester of introductory linguistics (fulfilled by 24.900 or equivalent).

If you think you have the requisite background but have not taken the specific courses just mentioned, please talk to the instructor to work out whether you should take this course or do other prerequisites first.

We will be doing Python programming in this course, and also using programs that must be run from the Unix/Linux/OS X command line. If you have less than one semester of Python programming experience and/or would like to strengthen your Python programming background, I have listed some resources recommended by others here.

If you are unsure about whether this class is a good fit for you given your background and interests, please consult the class FAQ; if you are still unsure, please contact Roger.

6 Well-being and flexibility in teaching during COVID

The 9.19 instructional staff acknowledge that we're teaching in an extraordinary time, and that there are many sources of uncertainty and fatigue. We are committed to providing you a supportive, stable, and safe educational environment during this time. In addition to the pre-recorded that we're working with, live class sessions will be recorded and made available afterwards.

If you are notified that you must isolate or quarantine because of a COVID-related issue – please know that we am committed to working with S3 or GradSupport to support your continued learning this semester. we understand how difficult these times are and we will all need to remain flexible and adjust to the situation. The details of what may be required will vary depending on the timing and duration of your absence. Please feel free to reach out to us at any point if you have any questions or concerns.

7 Readings & Textbooks

Readings will frequently be drawn from the following textbooks:

  1. Daniel Jurafsky and James H. Martin. Speech and Language Processing. Draft chapters for the third edition can be found here. (I refer to this book as "SLP3" in the syllabus.)

    This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics. Note: we will also be using some chapters from the second edition (2008); these will be referred to as J&M2008 on the syllabus, and will be available on Stellar.

  2. Jacob Eisenstein. 2019. Introduction to Natural Language Processing. MIT Press. Pre-publication PDF available here.

    This is an excellent recent textbook on natural language processing, with contemporary deep learning thoroughly integrated. We will be using the pre-publication version freely available, under the Creative Commons CC BY-NC-ND license, here.

  3. Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media. (I refer to this book as "NLTK" in the syllabus.)

    This is the book for the Natural Language Toolkit (or NLTK), which we will be using extensively to do programming We will also be doing some of our programming in the Python programming language, and will make quite a bit of use of Python. You can buy this book, or you can freely access it on the Web at

  4. Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "M&S" in the syllabus.)

    This is an older but still very useful book on natural language processing (NLP).

We'll also occasionally draw upon other sources for readings, including original research papers in computational linguistics, psycholinguistics, and other areas of the cognitive science of language.

8 Syllabus (subject to modification!)

Week Day Topic Readings Related readings Problem sets
Week 1 Wed 8 Sep Course intro; intro to probability theory M&S 2.1 . Goldsmith, 2007; Clayards et al., 2008 Pset 1 out
  Fri 10 Sep Recitation 1: Probability theory      
Week 2 Mon 13 Sep Speech perception and introductory rational analysis Selection from Anderson 1990; PMSL 5.2.4    
  Wed 15 Sep Word sequences, language models and $N$-grams . SLP3 3    
  Fri 17 Sep Recitation 2: Python programming      
Week 3 Mon 20 Sep Psycholinguistic methods; prediction in human language understanding; surprisal theory . Kutas et al. 2011; TBD . McMurray et al., 2008; Smith & Levy, 2013; Rayner, 1998; Dahan et al., 2001 Pset 2 out
  Wed 22 Sep Regular expressions and finite-state machines J&M2008 2.1-2.4, 3; Eisenstein, 2018, Section 9.1   Pset 1 due
  Fri 24 Sep Recitation 3: Regular expressions in practice      
Week 4 Mon 27 Sep Is human language finite state? SLP3 11; J&M2008 16 (under Readings on Stellar); . Chomsky, 1956 Pset 3 out
  Wed 29 Sep Context-free grammars; syntactic analysis. SLP3 12; Eisenstein, 2018, Section 9.2; NLTK 8.1-8.5; Levy & Andrew, 2006 . Gazdar, 1981 (esp. Section 2); Müller, 2018 (esp. Section 5.4); Joshi et al., 1991 (on formalisms beyond context-free)  
  Fri 1 Oct Recitation 4: Linguistics fundamentals      
Week 5 Mon 4 Oct Context-free grammars and syntactic analysis continued.     Pset 2 due
  Wed 6 Oct Probabilistic context-free grammars (PCFGs), incremental parsing, human syntactic processing SLP3 10 & 13; NLTK 8.6; Levy, 2013 . Jurafsky, 1996; Hale, 2001; Levy, 2008 Pset 4 out
  Fri 8 Oct Recitation 5: Writing grammar fragments      
Week 6 Mon 11 Oct Indigenous Peoples' Day, no class      
  Wed 13 Oct Theory, models, and data for human language comprehension     Pset 3 due
  Fri 15 Oct Recitation 6: Designing experiments & interpreting experimental data      
Week 7 Mon 18 Oct Bayes Nets; the perceptual magnet Russell & Norvig, 2010, chapter 14 (on Stellar); Feldman & Griffiths, 2006; Levy in progress, Directed Graphical Models appendix;   Pset 5 out
  Wed 20 Oct Multi-factor models: logistic regression; word order preferences in language. Hierarchical models; binomial construction. SLP3 5; Graphical models intro ; Morgan & Levy, 2015 . Bayes Nets lecture notes; Kraljic et al., 2008  
  Fri 22 Oct Recitation 7: Midterm review     Pset 4 due
Week 8 Mon 25 Oct Midterm exam (take-home, 24 hours, designed to take no more than 80 minutes)      
  Wed 27 Oct Logistic regression review; basic multi-layer neural networks SLP3 7.1-7.4; Eisenstein, 2018, 3.1-3.3 . Mikolov et al., 2013, Pennington et al., 2014, Gutierrez et al., 2016  
  Fri 29 Oct Recitation 8: feed-forward neural networks      
Week 9 Mon 1 Nov Word embeddings SLP3 6; Eisenstein, 2018, Ch. 14; Young et al., 2018, Section IV; . Goldberg, 2015; Collobert et al., 2011; Levy, Goldberg, & Dagan, 2015; Pset 6 out
  Wed 3 Nov Neural networks for natural language SLP3 7.5, 9.1-9.4; Eisenstein 2018, 6.3   Pset 5 due
  Fri 5 Nov Recitation 8: language modeling with deep learning      
Week 10 Mon 8 Nov What do neural networks learn about language structure – and what don't they learn? . Linzen & Baroni, 2020 . Linzen et al., 2016; Gulordava et al., 2018, Linzen 2019  
  Wed 10 Nov Bringing together grammar and deep learning; controlled syntactic evaluation of neural language models . Dyer et al., 2016; Futrell et al., 2019 . Choe & Charniak, 2016; Kuncoro et al., 2017; Dyer et al., 2015 Pset 7 out
  Fri 12 Nov Recitation 9: comparing models with human data      
Week 11 Mon 15 Nov Transformers; filler-gap dependencies . Vaswani et al., 2017; Sasha Rush's "The Annotated Transformer"; Wilcox et al., 2018 . Radford et al., 2018; Radford et al., 2019 Pset 6 due
  Wed 17 Nov Noisy-channel language comprehension . Levy et al., 2009    
  Fri 19 Nov Recitation 10: how attention works      
Week 12 Mon 22 Nov Noisy-channel language comprehension II . Gibson et al., 2013; Futrell & Levy, 2017   Pset 8 out
  Wed 24 Nov Unsupervised learning and native language acquisition I . Feldman et al., 2013   Pset 7 due
  Fri 26 Nov Thanksgiving holiday, no recitation      
Week 13 Mon 29 Nov Unsupervised learning and native language acquisition II . Saffran et al., 1996; Goldwater, Griffiths, & Johnson, 2009    
  Wed 1 Dec Computational semantics & pragmatics I . Frank & Goodman, 2016   Pset 8 due
  Fri 3 Dec Recitation 11: hierarchical Bayesian models     9.190 Class projects due Friday, December 3
Week 14 Mon 6 Dec Computational semantics & pragmatics II Reading TBD    
  Wed 8 Dec End-of-semester review Use this class to review readings from the rest of the semester!    
Final Exam Mon 13 Dec Final exam (take-home, 48 hours, designed to take no more than 2 hours)      

9 Requirements & grading

You'll be graded on:

Work Grade percentage (9.19) Grade percentage (9.190)
A number of homework assignments throughout the semester 60% 48%
A midterm exam 15% 12%
A final exam 25% 20%
If you are enrolled in 9.190, a class project -- 20%

Active participation in the class is also encouraged and taken into account in borderline grade cases!

9.1 Pset late policy

Psets can be turned in up to 7 days late; 10% of your score will be deducted for each 24 hours of lateness (rounded up, so e.g. you lose 10% of your score for an assignment turned in 12 hours late). For example, if a homework assignment is worth 80 points, you turn it in 3 days late, and earn a 70 before lateness is taken into account, your score will be (1-0.3)*70=49.

9.2 Personal or medical circumstances impacting psets, exams, or projects

If personal or medical circumstances such as illness impact your work on a pset or project, or your ability to take an exam on the scheduled date with adequate preparation, please work with Student Support Services (S3) to verify these circumstances and be in touch with the instructor. We are happy to work with you in whatever way is most appropriate to your individual circumstances to help ensure that you are able to achieve your best performance in class while maintaining your health, happiness, and well-being.

9.3 Mapping of class score to letter grade

We use new homework and exam materials every year, and it can be hard to perfectly predict the difficulty of assignments and exams. Therefore I determine standards for end-of-semester letter grades in light of student performance throughout the class (though I do not grade on a curve). However, I guarantee minimum grades on the basis of the following thresholds:

Threshold Guaranteed minimum grade
>=90% A-
>=80% B-
>=70% C-
>=60% D

So, for example, an overall score of 90.0001% of points guarantees you an A-, but you could well wind up with a higher grade depending on the ultimate grade thresholds determined at the end of the semester.

Author: Roger Levy (

Created: 2021-10-12 Tue 15:43