Mark's Notes

A collection of notes from college courses, self-study and research. Domains span mathematics, physics, computer science and occasionally philosophy. I hope to rigorously work through ideas, establish connections across disciplines and build a deep understanding of how the world works.

Figure 1: Stars (Yosemite)

🛠️ A Learning Mechanic’s Toolkit

A learning mechanic studies learning mechanics—a dynamical and mechanistic perspective on traditional deep learning theory. This toolkit collects instruments for characterizing important properties and statistics of the training process, hidden representations, and final weights of neural networks.

🔧 Deep Dives

Step-by-step derivations, refined expositions

• Deep Linear Networks; A deep dive into Saxe et al. and the role of depth in learning
  • deep linear networks · dynamics · exact solutions
  • Deep linear networks are mathematically tractable yet retain some of the mysterious phenomena of deep learning. We derive the exact training dynamics of these toy models and prove that long plateaus and rapid transitions are inherent to depth.

🔨 Notes

Quick insights, useful math, summaries of important phenomena and models

• Maximal stable learning rate derivation
  • edge of stability · optimization
  • Given a well-behaved loss (constant Hessian), we analytically derive the maximal stable learning rate under gradient descent.
• Quadratic word embedding model (QWEM)
  • word embeddings · exact solutions
  • The second-order approximation of the Word2Vec loss yields an equivalent supervised matrix factorization loss.
• When (wide) neural networks become linear
  • infinite limits · neural tangent kernel
  • As the widths of the layers in a neural network become large, the network becomes approximately equal to its first-order (linear) approximation.
• Singular values under perturbation
  • ${\sigma }_{\text{max}}(A+B)\le {\sigma }_{max}(A)+{\sigma }_{max}(B)$

🧮 Math Proofs

Proving cool math theorems

• Weierstrass Approximation Theorem
  • Polynomials approximate continuous functions very well.
• Riesz Representation Theorem
  • A vector space and its dual are always in bijection.

🌱 Exploratory Notes

Raw notes, incomplete thoughts and ongoing learning

Mathematics

• Linear Algebra
• Applications

Computer Science

• Machine Learning

Philosophy

• AI, Values, and Alignment
• A reasonable argument against reason
• All actions are selfish

---

Figure 2: Sunset (Mt. Tam)

Margins

A small subset of thoughts that pass through my mind

Miscellaneous

• overwhelmed
• movies & tv shows