Deep Learning

Deep learning is Machine Learning with many stacked layers of artificial neurons, trained end to end. The big shift from classic ML is that you stop hand-crafting features. You don’t tell the network what an edge or a wheel looks like; it learns those features itself, layer by layer, from data.

How it learns: backpropagation

1. Feed an input forward through the layers and get a prediction.
2. Measure how wrong it was with a loss function.
3. Use the chain rule to work out how much each weight contributed to the error — the gradients flow backwards.
4. Nudge every weight a little in the direction that reduces the loss (gradient descent). Repeat millions of times.

Non-linear activation functions (sigmoid, ReLU) are what make depth worthwhile. Without them, a hundred layers collapse into the maths of one.

The architectures worth knowing

• Convolutional networks (CNNs) slide small learnt filters over an image. Early layers find edges, middle layers find textures, later layers find objects. It’s the same convolution maths as classic image processing — the filters are just learnt rather than designed. Encoder–decoder variants such as U-Net do segmentation.
• Recurrent networks (RNNs) process sequences one step at a time with a memory of what came before. They struggled with long sequences.
• Transformers replaced recurrence with attention: every token can look at every other token and weigh how relevant it is. That parallelises beautifully on GPUs, and it’s the architecture behind today’s large language models — and so behind AI agents.

In practice

At university I built small neural networks by hand — single neurons, Hebbian learning, the delta rule — and it felt academic. A few years later the same ideas, scaled up absurdly, are writing code next to me. What I keep from the fundamentals is a healthy model of what these systems are: very good pattern-fitters trained on a loss function. When an LLM is confidently wrong, that’s not a bug in the usual sense; it’s the training objective showing through.

Still a seed: I want to add notes on embeddings, fine-tuning versus prompting, and how evals work for generative models.