00230212: Introduction to Morden AI (现代人工智能导论) (Fall 2024)

Course Information

When: Tuesday 9:50 pm – 12:15 pm.
Where: 清华大学旧水利馆303
Instructor: Jianzhu Ma (马剑竹) Email: majianzhu at tsinghua dot edu dot cn.
Office Hour: Wednesday 19:30 pm – 20:30 pm.or by appointment (send e-mail) Where: TBD

Course description:

This course aims to provide a comprehensive overview of the principles, technologies, and impacts of modern artificial intelligence (AI). Beginning with historical context, the course delves into the development of AI, emphasizing the transition from theoretical concepts to practical applications. Students will learn the fundamental concepts of AI, including machine learning, deep learning, and computational learning theory, with specific applications covering computer vision (CV), natural language processing (NLP), and computational biology. Through theoretical studies and hands-on projects, the course aims to equip students with the skills to understand and apply AI technologies across various fields, providing a solid foundation for their application of AI in various industries in the modern world.

Prerequisites:

Students are expected to have the following background:

Basic programming skills to write a reasonably non-trivial computer program in Python.
Basic understanding of statistics, linear algebra.
Understanding of data science. Knowing how to work with data and analyze.

Optional textbooks

“Probabilistic Machine Learning: An Introduction” by Kevin Murphy
“Pattern Recognition and Machine Learning” by Christopher Bishop
“Dive into Deep Learning” by Aston Zhang, Zach Lipton, Mu Li, Alex Smola

Grading

Homework (50%)
Final projects (50%)

Assignments

There will be NINE homework assignments including both FIVE theory problems and FOUR programming problems. Programming projects need to be written in Python. Deep learning final projects should be written in PyTorch.

Late policy

Assignments are to be submitted by the due date listed. Assignments will NOT BE accepted if they are submitted late. Additional extensions will be granted only due to serious and documented medical or family emergencies.

Final projects

Students are encouraged to work as teams of two or three. Each team can either choose the provided projects on the course website or a separate project related to Biology. If you choose to do your own project, please contact me beforehand to discuss it. Here are some possible directions:

An interesting mathematical problems around a paper.
Adopting a developed computational framework on a new dataset.
Following-up experiments of an existing work to understand its important properties.
Simple extension of an existing machine learning model, such as unsigned network to signed network, trees to DAGs, shallow neural networks to deep neural networks.
Using machine learning model to solve a problem in your own research area.

Syllabus (tentative)

Time	Topic	Contents
09/10	Introduction & Linear Regression	(1) Introduction to AI history; (2) Introduction to Machine Learning; (3) Probability; (4) Linear Regression Optional Reading: Linear Regression Made Simple: A Step-by-Step Tutorial [link]
09/17	LASSO	(1) Bias-variance decomposition; (2) Description length; (3) Ridge regression; (4) L0, L1 and LP penalty; (5) Stepwise regression
09/24	Gaussian Distribution	(1) Log-likelihood; (2) Revisiting linear regression; (3) Mixture Gaussian and EM algorithm
10/08	Logistic Regression	(1) Classification & Regression; (2) Logistic model; (3) Gradient ascent/descent; (4) Regularization
10/15	Neural Network	(1) Neural network structure (2) Backpropagation; (3) Optimization; (4) Initialization; (5) Dropout; (6) Batch normalization;
10/22	Support Vector Machine	(1) Objective and Optimization; (2) Lagrange duality; (3) Kernel trick
10/29	Deep Learning	(1) Convolutional Neural Network; (2) Recurrent Neural Network; (3) Graph Neural Network; (4) Transformer Optional reading: (1) Attention Is All You Need [pdf] (2) Transformer from scratch using pytorch [link] (3) TRANSFORMER EXPLAINER [link]
11/05	Reinforcement Learning	(1) Q-learning; (2) Policy Gradient; (3) Actor-Critic Algorithm
11/12	Graph Neural Network	(1) Graph Convolutional Network; (2) Graph Attention Model;
11/19	Boosting	(1) AdaBoost; (2) Additive regression
11/26	Non-parametric Methods	(1) Kernel density estimation; (2) KNN; (3) Principle Component Analysis; (4) K-means
12/03	Learning theory	(1) Occam’s Razor/No free lunch; (2) Basic error bounds; (3) Hoeffding’s Inequality; (4) Union bound; (5) VC theory; (6) Rademacher complexity; (7) PAC bound
12/10	Graphical Models (I)	(1) General concepts (2) Exact inference (3) Sum-product algorithm (4) Max-product algorithm (5) Conditional Random Fields Optional textbooks: (1) “Probabilistic Graphical Models” by By Daphne Koller and Nir Friedman (2) “Graphical Models, Exponential Families, and Variational Inference” by Martin J. Wainwright and Michael I. Jordan (3) Chapter 8 of “Pattern Recognition and Machine Learning” by Christopher M. Bishop
12/17	Graphical Models (II)	(1) Structure learning (2) Gaussian Graphical Model (3) Pseudo-likelihood approximation (4) Protein contact prediction (5) Deep-Learning-based structure learning Optional reading: (1) Sparse Inverse Covariance Estimation with the Graphical Lasso [pdf] (2) High-Dimensional Graphs and Variable Selection with the Lasso [pdf]
12/24	Project presentations	TBD