**Independence Day, University Holiday (Thursday, July 4)**

**Lecture 1 (Monday, July 8): Introduction**

Machine learning timeline: from Least Squares to AlphaZero, milestones of neural networks and deep learning,
linear algebra review, fully connected neural networks, forward propagation as a composition of functions, each with linear and non-linear component, nonlinear activation functions, network loss functions.

**Lecture 2 (Tuesday, July 9): Backpropagation**

Backpropagation algorithm using chain rule in reverse mode differentiation, computing gradient of total loss with respect to all network parameters in a single backward pass using local computations, derivative of activation functions, chain rule for differentiation, equations for backpropagation, input normalization, weight initialization, computational graph, differential programming.

**Lecture 3 (Wednesday, July 10): Optimization**

Optimization methods using first order and second order derivatives, comparison, analytic and numerical computation of gradients, stochastic gradient descent, adaptive gradient descent methods, finding descent direction based on gradients and selecting the step size.
Evolution strategies for efficient distributed optimization.

**Tutorial 1: TensorFlow**

**Lecture 4 (Thursday, July 11): CNNs**

Representations sharing weights, convolution as filters or matrix multiplication, 2D and 3D convolutions as filters or block Toeplitz matrices, convolution and max pooling layers, CNNs.

**Tutorial 2: PyTorch**

**Project proposal meetings (Friday, July 12)**

**Exercise 1 (Friday, July 12): due Thursday, July 18**

**Lecture 5 (Monday, July 15): CNNs**

ResNet and ODENet, DenseNet, SENet

**Tutorial 3: Keras**

**Lecture 6 (Tuesday, July 16): Regularization**

Regularization of neural networks, bias and variance, generalization error, learning and fitting curves, vector and matrix norms, and cross validation, regularized loss functions, dropout regularization, data augmentation, batch normalization.

**Lecture 7 (Wednesday, July 17): Sequence Models**

Recurrent neural networks (RNNs), bidirectional RNNs, deep RNNs, backpropagation through time, limitations, long-short term memory (LSTM), gated-recurrent unit (GRU), stochastic beam search.

**Lecture 8 (Thursday, July 18): Sequence Models**

Encoder-decoder attention, self-attention, embeddings, transformers, BERT, paraphrasing, word transformations.

**Tutorial 4: AllenNLP**

**Exercise 2 (Friday, July 19): due Thursday, July 25**

**Lecture 9 (Monday, July 22): Generative Models**

Variational inference (VI), stochastic variational inference (SVI), black-box variational inference (BBVI), score function and reparameterization gradients, probability functional descent, amortized variational inference, variational auto-encoder (VAE) algorithm, vector quantized variational autoencoder (VQ-VAE).

**Tutorial 5: Pyro**

**Lecture 10 (Tuesday, July 23): Generative Models**

Generative adversarial networks (GANs) from a game theoretic and co-evolution perspectives, DCGANs, limitations, conditional GANs, Cycle GANs, 3D-GANs, style GAN.

**Tutorial 6: TF-GAN**

**Lecture 11 (Wednesday, July 24): Generative Models**

Wasserstein GAN, Flows, generative flow (Glow), continuous normalizing flow (CNF), invertible neural networks.

**Lecture 12 (Thursday, July 25): Graph Neural Networks**

Node and subgraph embeddings, neighborhood aggregation, graph convolutional networks (GCNs), graph neural networks (GNNs), gated graph neural networks (GGNNs).

**Project milestone meetings (Friday, July 26)**

**Exercise 3 (Friday, July 26): due Thursday, August 1**

**Lecture 13 (Monday, July 29): Adversarial Examples, Training Poisoning, DMI**

**Tutorial 7: TF-Probability**

**Lecture 14 (Tuesday, July 30): Reinforcement Learning**

Markov Decision Processes, value-based methods, policy-based methods.

**Lecture 15 (Wednesday, July 31): Reinforcement Learning**

Bellman equations, dynamic programming, planning, policy iteration and evaluation.

**Lecture 16 (Thursday, August 1): Reinforcement Learning**

Monte Carlo sampling, temporal difference sampling, Q-learning, Sarsa.

**Tutorial 8: TF-Agents**

**Exercise 4 (Friday, August 2): due Thursday, August 8**

**Lecture 17 (Monday, August 5): Deep Reinforcement Learning**

Deep Q-Networks (DQN), double DQN,model-based reinforcement learning, combining model free and model based methods.

**Lecture 18 (Tuesday, August 6): Deep Reinforcement Learning**

Policy gradient methods, REINFORCE, Actor-Critic methods, natural policy gradients, trust region optimization, combining Q-learning and policy gradient.

**Lecture 19 (Wednesday, August 7): Imperfect Information Games**

Imprefect information games, infosets, safe subgame solving, counterfactual regret minimization (CFR), MC-CFR, Deep CFR.

**Lecture 20 (Thursday, August 8): Automatic Deep Learning**

Automatic machine learning (AutoML), neural network architecture search, meta learning.

**Exercise 5 (Friday, August 9): due Thursday, August 15**

**Lecture 21 (Monday, August 12): Deep Learning for Self Driving Cars**

**Lecture 22 (Tuesday, August 13): Fair and Private Deep Learning**

**Lecture 23 (Wednesday, August 14): Quantum Neural Networks**

Spin, polarization, qubits, entanglement, Bellâ€™s inequality, quantum gates and circuits, no cloning theorem, Bell circuit, superdense coding and quantum teleportation, error correction, tensor networks, quantum neural networks.

**Lecture 24 (Thursday, August 15): Project Presentations**

5:30-5:50 ♣Deep CFR vs. Pluribus♠

5:50-6:00 3D protein structure prediction using conditional GANs

6:00-6:10 AutoML using dataset embeddings

6:10-6:20 AutoML using pipeline embeddings

Break 🥤 🍕

6:30-6:40 Explainable Deep Soduko

6:40-6:50 BERT for lyrics genre and author identification

6:50-7:00 SPADE for face video synthesis

7:00-7:10 Motion transfer

7:10-7:20 Adversarial examples for silhouette-based gait recognition

**Project report meetings (Friday, August 16)**

**Last Day of Summer Classes (Friday, August 16)**