Syllabus

  • Course Title: Physics 198: (Deep) Learning Mechanics
  • Units: 2
  • Instructor: Mark Rhee
  • Email: mrkdh@berkeley.edu
  • Faculty Sponsor: Michael DeWeese
  • Location: Physics Building 251
  • Time: Wednesdays, 6–8pm

Course Description

Deep learning is in a peculiar situation at this moment in time. Empirically, the capabilities of AI leveraging deep learning (eg. LLMs) have surpassed the expectations of even the most radically optimistic researchers. Publicly available LLMs are disproving math conjectures left and right, AI is being injected into every stage of real drug development life cycles, entire business models have been destroyed by improving AI capabilities, and at this point, AI winning a Nobel Prize is old news. Yet, we lack a comprehensive scientific framework for understanding how exactly these models work and the development of frontier models is closer to alchemy than science.

Not long ago, engineers built working steam engines before anyone understood the science behind them. The effort to understand and build better engines ended up creating an entirely new branch of science: statistical mechanics. Deep learning may be our generation’s steam engine. Learning mechanics is the emerging discipline that aims to understand it from first principles, treating deep learning the way physics treats the natural world: seeking compact mathematical principles, tight connections between theory and experiment, and simple, intuitive explanations for complex phenomena.

This course is for those who want to be part of building that new science.

Readings draw heavily from the perspective paper There Will Be a Scientific Theory of Deep Learning (Simon et al., 2026) and the primary literature it synthesizes.

Prerequisites

  • Linear algebra and multivariate calculus at the level of Math 53, 54 / 110
  • Probability and statistics at the level of EECS 126 / Data 140 / Stat 134
  • Working knowledge of neural networks and deep learning (see ch. 1 from Nielson (2019))

It is highly recommended that you bring a physicist’s mindset to the table. This means you should be comfortable with the scientific method: formulating testable hypotheses, checking theoretical predictions against empirical measurements, solving simplified cases first, and using educated guesses (ansatze) to solve equations. We care more about simple, intuitive insights and tight connections between theory and experiment than we do about technically true mathematics.

Grading

Grading will be done on an absolute scale, with any grade above 70 points receiving a Pass.

  • Attendance & Participation 10 points
    • The liveliness of this course heavily depends on in-class participation. Please show up to class and ask questions! Two unexcused absences are allowed without penalty.
  • Reading Questions 20 points
    • For every paper we read, please generate 2-3 questions and submit them before class.
  • Final Project 30 points
    • The final project is the capstone of the course. Working individually or in pairs, you will choose between an experiment or an exposition:
      • Formulate a hypothesis, run an experiment
        • Identify a phenomenon related to topics covered in class (or for the especially ambitious, a related open problem from the whitepaper).
        • Formulate and present a testable hypothesis about that phenomenon.
        • Design and run an experiment to test the hypothesis (computational experiments on small models are perfectly appropriate).
        • Write up your findings, including whether the hypothesis was supported, what you learned, and what questions remain.
      • Write a blog post introducing a learning mechanics idea
        • Identify a phenomenon related to topics covered in class
        • Create and present a web widget about that phenomenon.
        • Write up a short blog post introducing a newcomer to the particular phenomenon.
  • Homework Assignments 40 points
    • Three to four problem sets will be distributed over the semester. Collaboration is encouraged but each student must write up their own solutions.

Learning Objectives

By the end of the course, students will be able to:

  • Articulate what a scientific theory of deep learning would look like and why it matters.
  • Derive and analyze learning dynamics in solvable settings (deep linear networks, kernel regression).
  • Explain the lazy/rich dichotomy and its implications for feature learning.
  • Explain how the tools of learning mechanics are being used to tackle empirical phenomena discovered by the mechanistic interpretability community.
  • Design and execute small-scale experiments that test theoretical predictions about neural network training.

Course Content

The learning mechanics perspective paper (Simon et al., 2026) is structured around five lines of evidence that a scientific theory of deep learning is within reach:

  1. Analytically solvable settings (toy models) that exhibit nontrivial learning phenomena exist.
  2. Insightful limits reveal fundamental behavior.
  3. Meaningful macroscopic statistics are captured by simple equations.
  4. Hyperparameters can be disentangled and understood.
  5. Universal phenomena appear across settings and tasks.

This course will focus on 1) analytically solvable settings and 2) insightful limits, and briefly touch on 4) disentangling hyperparameters and 5) universal phenomena.

The following is a non-exhaustive list of content we will cover:

  • Deep Linear Networks (DLNs)
  • The Neural Tangent Kernel (NTK)
  • Kernel Ridge Regression (KRR)
  • Eigenlearning
  • The Hermite Eigenstructure Ansatz (HEA)
  • The Lazy (NTK) and Rich (muP) Regimes
  • Grokking
  • Balancedness
  • Feature Learning
  • The Platonic Representation Hypothesis

The course is organized around 5 chapters:

  1. Toy Model I: Deep Linear Networks
  2. Toy Model II: Kernel Regression and the NTK
  3. The Lazy and Rich Regimes
  4. Case Study I: Grokking
  5. Case Study II: Representational Geometry

Weekly Schedule

Each session is 2 hours long. A typical class includes a 50-60 minute lecture on the reading, a 10-minute break, and a 40-50 minute deep-dive on derivations, experiments, or open questions in the format of a discussion.

Week 1

(First Week: No Class)

Week 2

Lecture 1. Introduction I: Learning Mechanics (potential guest lecture)

What’s the evidence for an emerging scientific theory of deep learning?

  • Readings: Simon et al. (2026)

Week 3

Lecture 2. Introduction II: Neural Networks

What exactly are neural networks? Why are they hard to study? How will we study them anyways?

  • Readings: ch. 1 from Nielson (2019)
  • Lecture Notes: link
  • Homework: MAYBE, optional math review

Week 4

Lecture 3. Toy Model I: Deep Linear Networks

What can we learn about deep learning from a highly mathematically tractable toy model in deep linear networks?

Week 5

Lecture 4. Toy Model I: Deep Linear Networks (continued)

How can we analytically solve for the training dynamics of deep linear networks?

  • Lecture Notes: link
  • Homework: link

Week 6

Lecture 5. Toy Model II: Kernel Regression and the NTK Analytically Solvable Settings III: Eigenlearning and the HEA (potential guest lecture)

Is there a limit in which neural networks become analytically solvable?

Week 7

Lecture 6. Toy Model II: Kernel Regression and the NTK (continued)

How can we develop a mathematical framework to study kernel regression? Can we predict how kernel regression will perform on real data?

Week 8

Lecture 7. The Lazy (NTK) and Rich (muP) Regimes

In the lazy (NTK) regime, neural networks don’t learn any structure. Is there a regime where they do?

Week 9

Lecture 8. The Lazy (NTK) and Rich (muP) Regimes (continued)

How can we disentangle hyperparameters to maximize feature learning?

Week 10

(Thanksgiving Break: No Class)

Week 11

Lecture 10. Case Study I: Grokking

How can we apply the tools of learning mechanics to understand grokking?

Week 12

Lecture 11. Case Study II: Representational Geometry

What kind of features are learned by language models? How might we characterize where such features come from and how they’re learned?

Week 13

(buffer)

Week 14

Lecture 13. Final Project Hypothesis Presentations

Week 15

Lecture 14. Final Project Office Hours

Week 16

(RRR Week: No Class)

This site uses Just the Docs, a documentation theme for Jekyll.