A tiny subset of the things which I think about most deeply.
🛠️ A Learning Mechanic’s Toolkit
Learning mechanics is an attempt to apply the scientific method to deep learning. It aims to develop a fundamental, mathematical, predictive, comprehensive, intuitive, and useful theory of deep learning. This set of notes is my attempt at developing a learning mechanic’s toolkit.
🔧 Deep Dives
Step-by-step derivations, refined expositions
- A deep dive into the ultimate toy model; deep linear networks
- exact solutions · training dynamics · deep linear networks
- Deep linear networks, while lacking expressivity, are a surprisingly good toy model of the weight-space dynamics of neural networks. With deep linear networks, we can concretely observe a connection between the features learned by the network and the optimization dynamics that it undergoes.
🔨 Notes
Summaries of important phenomena and models and some useful math
-
The lazy (NTK) and rich (muP) regimes
- infinite limits · lazy/rich
- By enforcing stable training criteria on a simple 3-layer linear network, we entirely determine all initialization hyperparameters with a single degree of freedom defined as the richness parameter.
-
When (wide) neural networks become linear
- infinite limits · neural tangent kernel
- As the widths of the layers in a neural network become large, the network becomes approximately equal to its first-order (linear) approximation.
-
Quadratic word embedding model (QWEM)
- exact solutions · feature learning · word embeddings
- The second-order approximation of the Word2Vec loss yields an equivalent supervised matrix factorization loss. This means we can study a minimal language model through a highly mathematically tractable model in matrix factorization.
-
Singular values under perturbation
🌱 Exploratory Notes
Mathematics
Computer Science
