Topic Modelling with LDA

Topic modelling is how you get a summary of a pile of text far too big to read. An algorithm scans the documents, notices which words tend to appear together, and groups them into topics: clusters of words that characterise what the collection is about. No labels needed. It’s unsupervised learning for text.

I learnt it for Pathways into Cybercrime, my dissertation, where the “pile” was years of forum posts.

How LDA thinks

Latent Dirichlet Allocation (Blei, Ng and Jordan, 2003) is the classic algorithm. It imagines that documents are generated like this:

1. Each document is a mixture of topics: 70% “anti-cheat”, 20% “game updates”, 10% “selling accounts”.
2. Each topic is a probability distribution over words. The “anti-cheat” topic favours ban, detected, bypass, kernel.
3. To write each word, pick a topic from the document’s mixture, then a word from that topic.

Training runs that story backwards: given only the words, infer the topic mixtures and the word distributions most likely to have produced them. Two smoothing parameters (alpha and beta) control how many topics a document tends to mix, and how many words a topic tends to use. A word can belong to several topics, which is more realistic than hard clustering.

Latent Semantic Analysis (LSA) rests on the same idea (words that appear in similar contexts mean similar things) but uses linear algebra rather than a probabilistic model.

The pipeline

1. Collect the corpus. For forum research that might come through an API (Mining Reddit for Research).
2. Clean: lowercase, strip non-letters, expand contractions, normalise whitespace, tokenise, remove stop words.
3. Explore: word frequencies and document lengths, so you know what you’re feeding in.
4. Vectorise into a bag of words, or TF-IDF.
5. Train LDA (Gensim in Python is the standard) for a chosen number of topics.
6. Interpret: read the top words per topic, name them, and check them against real documents. Coherence scores help pick the number of topics, but human judgement decides whether a topic means anything.

Where it shines in research

Researchers studying cybercrime have used LDA to map large, messy forums quickly:

• Tavabi et al. (2019) characterised activity across dozens of dark-web forums: vending, security, gaming.
• Kigerl (2018) clustered the comment histories of carding-forum members to profile them.
• Porter (2018) ran LDA month by month over a year of a darknet-market subreddit, tracking which tools and services rose and fell after major market takedowns.

Running the model per time slice turns topic modelling into a way of watching a community change, which is exactly what a “pathways” question needs.

Trade-offs

• Bag of words loses meaning. Word order, negation and sarcasm vanish.
• Topics need a human. The model hands you word lists; naming them is interpretation, and it’s easy to see what you expect to see.
• You choose the number of topics, and the results change with it.
• Slow on big corpora, though still far faster than reading.

LDA in the LLM era

Today I’d usually start with sentence embeddings, cluster those, and ask an LLM to name and summarise each cluster. It captures meaning better and needs less cleaning. But LDA is still a good explainable baseline: cheap, transparent and reproducible, which matters when results go into a paper. The language side of all this is covered in Natural Language Processing.

Sources

• Blei, D., Ng, A. and Jordan, M. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research.
• Tavabi, N. et al. (2019). Characterizing activity on the deep and dark web. Companion Proceedings of The Web Conference.
• Kigerl, A. (2018). Profiling cybercriminals: topic model clustering of carding forum member comment histories. Social Science Computer Review.
• Porter, K. (2018). Analyzing the DarkNetMarkets subreddit for evolutions of tools and trends using LDA topic modeling. Digital Investigation.