Language Modeling from First Principles

What you'll be able to do

✓Build a predictive-text engine from scratch — the next-word suggestions behind a phone keyboard
✓Count word n-grams and turn them into next-word probabilities with maximum likelihood
✓Generate text by sampling from your n-gram model
✓Fix the zero-probability problem with add-k (Laplace) smoothing
✓Measure a language model honestly with cross-entropy and perplexity

Build the autocomplete that finishes your sentences — the next-word predictor behind Gboard, Apple QuickType, and SwiftKey — from word counts up, with no neural network.

⌁ The original language model — n-gram predictors power phone-keyboard suggestions, search-query completion, and spelling correction, and they are the statistical foundation every neural LLM is built on.

Start this internship

Create an account to unlock the 8 sections, the workbench, and AskThili.

Begin