Multimodal
Images, audio, and documents beyond plain text.
9 tutorials, spanning 2 of the 17 learning layers — ordered so each builds on the last.
11Multimodal
- How Do Models See Images?An image becomes a sequence of patch tokens, and from there the machinery is the same one that reads text.
- How Image Generation WorksStart with noise and repeatedly remove a little of it. Diffusion is a different mechanism from text generation, and the difference explains the controls.
- Multimodal EmbeddingsTrain images and text into one shared vector space and you can search pictures with words. The idea behind CLIP and most cross-modal retrieval.
- Contrastive Learning and CLIPHow images and text end up in one shared space where a photo and its caption land near each other. The training idea that made multimodal models possible.
- How Speech Recognition WorksAudio becomes a spectrogram, the spectrogram becomes tokens, and from there it is sequence-to-sequence translation.
- How Text-to-Speech WorksTwo stages: text becomes a spectrogram, the spectrogram becomes audio. Plus why prosody is the hard part and latency is the practical one.
- Vision-Language ModelsPut image patches and text tokens in one sequence and attention does the rest. How a language model gains eyes, and what it still cannot do.
- Document and OCR UnderstandingExtracting text is solved. Extracting structure is not, and structure is usually what you actually need.