Natural Language Processing

Natural language processing (NLP) is the part of Machine Learning that deals with text. The core problem never changes: language is messy, ambiguous and context-dependent, and models need numbers. Every NLP technique is a different answer to “how do we turn words into numbers without losing what they mean?”

The pipeline, classic to modern

1. Tokenise and normalise. Split text into words or sub-words, lowercase, strip noise. Libraries: NLTK (thorough, slow), spaCy (fast, opinionated).
2. Represent. Bag of words and TF-IDF (counts, with no idea of order) → word embeddings (each word as a vector, so similar words sit near each other) → contextual embeddings from transformers (the same word gets different vectors in different sentences).
3. Model. Naive Bayes and SVMs for classification; LDA for topics (Topic Modelling with LDA); then CNNs, RNNs and attention-based networks; and now transformers for almost everything (Deep Learning).
4. Evaluate. Precision and recall against labelled data, or human judgement for generated text.

Two families of task

• Unsupervised: discover. Topic modelling and clustering find what a corpus is about, without labels.
• Supervised: classify. Once you know the categories, train a model to assign them: spam or not, topic tags, sentiment. Classic choices: multinomial naive Bayes (great with a few thousand examples) and SVMs (often more accurate, more compute). Deep models (CNNs, RNNs, hierarchical attention networks that read words and then sentences) keep improving with more data, where classic models plateau.

My dissertation used both: discover topics in forum posts first, then classify (Pathways into Cybercrime). Before that, a coursework project of mine evolved sentiment lexicons with genetic algorithms, which was NLP by natural selection.

What changed

Large language models do most of this zero-shot. Describe the categories in a prompt and they’ll classify, extract or summarise with no training data. That collapses weeks of feature engineering into an afternoon. But the old discipline still matters: label a test set, measure agreement, and don’t trust a model’s labels just because they read well.

Not to be confused with neuro-linguistic programming, the self-help kind of NLP.