🧠 CS336: Language Modeling from Scratch Spring 2025

👥 Course Staff

• Tatsunori Hashimoto — Instructor
• Percy Liang — Instructor
• Marcel Rød — CA
• Neil Band — CA
• Rohith Kuditipudi — CA

---

📅 Logistics

• Lectures: Tuesday/Thursday 3:00–4:20pm, NVIDIA Auditorium
• Office Hours:
  • Tatsu Hashimoto (Gates 364): Fridays 3–4pm
  • Percy Liang (Gates 350): Fridays 11am–12pm
  • Marcel Rød (Gates 415): Mon/Wed 11am–12pm
  • Neil Band (Gates 358): Mon 4–5pm, Tues 5–6pm
  • Rohith Kuditipudi (Gates 358): Mon/Wed 10–11am
• Contact: Use public Slack channels for questions and announcements. For personal matters, email cs336-spr2425-staff@lists.stanford.edu.

---

📖 Course Overview

This course provides a comprehensive, hands-on introduction to language modeling, guiding students through building language models from scratch. Topics include data collection, transformer architectures, model training, evaluation, and deployment. The course is implementation-heavy and requires strong Python and deep learning skills.

---

✅ Prerequisites

• Proficiency in Python
• Experience with deep learning (PyTorch) and systems optimization
• College-level calculus and linear algebra
• Basic probability and statistics
• Prior coursework in machine learning (e.g., CS221, CS229, CS230, CS124, CS224N)

---

📝 Coursework

1. Basics: Implement and train a standard Transformer language model.
2. Systems: Profile, optimize, and distribute model training.
3. Scaling: Analyze and fit scaling laws for model growth.
4. Data: Process and filter large-scale pretraining data.
5. Alignment and Reasoning RL: Apply supervised finetuning and RL for reasoning tasks.