Computational Psycholinguistics, Spring 2019

Lecture Times	Mondays & Wednesdays 9:30-11:00am
Lecture Location	46-3310
Class website	https://learning-modules.mit.edu/class/index.html?uuid=/course/9/sp19/9.19#info
Syllabus	http://www.mit.edu/~rplevy/teaching/2019spring/9.19

Instructor	Roger Levy (rplevy@mit.edu)
Instructor's office	46-3033
Instructor's office hours	Thursdays 9-11am (subject to change)
Teaching Assistants	Peng Qian (pqian@mit.edu), Veronica Boyce (vboyce@mit.edu)
TA Office	46-3027B (Peng) 46-3027F (Veronica)
TA Office Hours	Thursdays 1-3pm (Peng); Mondays 11am-1pm (Veronica)

Over the last two and a half decades, computational linguistics has been revolutionized as a result of three closely related developments: increases in computing power, the advent of large linguistic datasets, and a paradigm shift toward probabilistic modeling. At the same time, similar theoretical developments in cognitive science have led to a view major aspects of human cognition as instances of rational statistical inference. These developments have set the stage for renewed interest in computational approaches to human language use. Correspondingly, this course covers some of the most exciting developments in computational psycholinguistics over the past decade. The course spans human language comprehension, production, and acquisition, and covers key phenomena spanning phonetics, phonology, morphology, syntax, semantics, and pragmatics. Students will learn technical tools including probabilistic models, formal grammars, neural networks, and decision theory, and how theory, computational modeling, and data can be combined to advance our fundamental understanding of human language acquisition and use.

We'll meet twice a week; the course format will be a combination of lecture, discussion, and in-class exercises as class size, structure, and interests permit.

Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. The undergraduate section is 9.19, the graduate section is 9.190. Postdocs and faculty are also welcome to participate!

The course prerequisites are:

1. One semester of Python programming (fulfillable by 6.00/6.0001+6.0002, for example), plus
2. Either:
   • one semester of probability/statistics/machine learning (fulfilled by, for example, 6.041B or 9.40), or
   • one semester of introductory linguistics (fulfilled by 24.900).

If you think you have the requisite background but have not taken the specific courses just mentioned, please talk to the instructor to work out whether you should take this course or do other prerequisites first.

We will be doing Python programming in this course, and also using programs that must be run from the Unix/Linux/OS X command line. If you have less than one semester of Python programming expereince and/or would like to strengthen your Python programming background, I have listed some resources recommended by others here.

Readings will frequently be drawn from the following textbooks:

1. Daniel Jurafsky and James H. Martin. Speech and Language Processing. Third edition (draft). Draft chapters can be found here. (I refer to this book as "SLP" in the syllabus.)

   This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics.

2. Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media. (I refer to this book as "NLTK" in the syllabus.)

   This is the book for the Natural Language Toolkit (or NLTK), which we will be using extensively to do programming We will also be doing some of our programming in the Python programming language, and will make quite a bit of use of for Python. You can buy this book, or you can freely access it on the Web at http://www.nltk.org/book.

3. Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "M&S" in the syllabus.)

   This is an older but still very useful book on NLP.

4. Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "MRS" in the syllabus.)

We'll also occasionally draw upon other sources for readings, including original research papers in computational linguistics, psycholinguistics, and other areas of the cognitive science of language.

Week	Day	Topic	Readings	Related readings	Problem sets
Week 1	Wed 6 Feb	Course intro; intro to probability theory;		Feldman et al., 2009	Pset 1 out
Week 2	Mon 11 Feb	Speech perception and the perceptual magnet		MRS Chapter 13
	Wed 13 Feb	Word sequences, language models and \(N\)-grams	SLP 3
Week 3	Tue 19 Feb	Psycholinguistic methods: human eye movements and language understanding	None	McMurray et al., 2008; Rayner, 1998; Dahan et al., 2001
	Wed 20 Feb	Prediction in human language understanding; surprisal	Kutas et al. 2011; Piantadosi et al., 2011	Smith & Levy, 2013	Pset 1 due (Fri 22 Feb)
Week 4	Mon 25 Feb	Word embeddings and neural networks	SLP 6; Levy, Goldberg, & Dagan, 2015	Mikolov et al., 2013, Pennington et al., 2014, Gutierrez et al., 2016
	Wed 27 Feb	Recurrent neural network models for language	SLP 7; Young et al., 2018, Section IV; Eisenstein, 2018, Section 6.3	Goldberg, 2015; Collobert et al., 2011	Pset 2 out (Fri 1 Mar)
Week 5	Mon 4 Mar	Regular expressions	SLP 2.1-2.1.6
	Wed 6 Mar	Finite-state machines	SLP 2.2-2.4, 3
Week 6	Mon 11 Mar	Finite-state machines II
	Wed 13 Mar	Is human language finite state?	SLP 11; SLP 2nd edition chapter 16 (under Readings on Stellar)	Chomsky, 1956
Week 7	Mon 18 Mar	Context-free grammars; Syntactic analysis.	SLP 12; NLTK 8.1-8.5; Levy & Andrew, 2006	Gazdar, 1981 (esp. Section 2); Müller, 2018 (esp. Section 5.4); Joshi et al., 1991 (on formalisms beyond context-free)	Pset 2 due
	Wed 20 Mar	Midterm Exam
Spring Break	25-29 Mar	Spring break, no class
Week 8	Mon 1 Apr	Probabilistic context-free grammars, incremental parsing, human syntactic processing	SLP 12; NLTK 8.6; Levy, 2013	Jurafsky, 1996; Hale, 2001; Levy, 2008	Pset 3 out
	Wed 3 Apr	Searching treebanks, estimating PCFGs, and accounting for syntactic comprehension effects	SLP 10	SLP 13
Week 9	Mon 8 Apr	What do neural networks learn about language structure – and what don't they learn?	Linzen et al., 2016; Wilcox et al., 2018	Linzen 2019
	Wed 10 Apr	Artificial neural networks as psycholinguistic subjects	Futrell et al., 2019		Pset 3 due (Fri 12 Apr)
Week 10	Mon 15 Apr	Patriots Day, no class (student holiday due to Patriots Day)
	Wed 17 Apr	Implicit associations in word embeddings	Caliskan et al., 2017
Week 11	Mon 22 Apr	Bayes Nets and interventions	Kraljic et al., 2008; Russell & Norvig, 2010, chapter 14 (on Stellar); Levy in progress, Directed Graphical Models appendix;	Bayes Nets lecture notes	Pset 4 out
	Wed 24 Apr	Multi-factor models: logistic regression; word order preferences in language. Hierarchical models; binomial construction.	SLP 7; Graphical models intro ; Morgan & Levy, 2015
Week 12	Mon 29 Apr in 56-114	Statistical word learning in humans; modeling with nonparametric Bayes	Saffran et al., 1996; Goldwater, Griffiths, & Johnson, 2009
	Wed 1 May	The emergence of syntactic productivity in language development	Meylan et al., 2017
Week 13	Mon 6 May	Combining grammars and neural networks	Dyer et al., 2016; Kuncoro et al., 2017	Choe & Charniak, 2016; Dyer et al., 2015
	Wed 8 May	Comparing the grammatical capabilities of sequence-based and hybrid neuro-symbolic models	Futrell et al., 2019; Wilcox et al., 2019		Pset 4 due (Fri 10 May)
Week 14	Mon 13 May	Noisy-channel language comprehension	Levy et al., 2009; Gibson et al., 2013; Futrell & Levy, 2017
	Wed 15 May	End-of-semester review	Use this class to review readings from the rest of the semester!		Final projects due Thursday, May 16
Final Exam	Tue 21 May 2:15-4:30pm

You'll be graded on:

Work	Grade percentage (9.19)	Grade percentage (9.190)
A number of homework assignments throughout the semester	50%	37.5%
A midterm exam	20%	15%
A final exam	30%	22.5%
If you are enrolled in 9.190, a final project	–	25%

Active participation in the class is also encouraged and taken into account in borderline grade cases!

There will be a mailing list for this course, which you can access at https://mailman.mit.edu:444/mailman/listinfo/9.19-2019-spring. Please make sure you're signed up for it! This list is both for discussion of ideas in the class and for communications about organizational issues.