Mining Reddit for Research

Reddit is a favourite of social-media researchers for good reason. It’s organised into subreddits, focused communities with their own rules and moderators. Every post has a title, optional body text, comment threads, votes, sometimes a “flair” that categorises it, and a timestamp. That structure maps neatly onto research questions about communities and how they change. Datta and Adar (2019) chose it because its features resemble countless other discussion boards, at vast scale.

I mined it for Pathways into Cybercrime, my dissertation.

Collecting

• PRAW (the Python Reddit API Wrapper) is the friendly way in. Register an API application, then fetch a subreddit’s top or newest posts, walk comment trees, or stream new posts as they arrive. Brand-new posts have few comments, so you want a mix of streaming and back-filling.
• Historical data was the gap. The live API can’t search by date range. During my project the answer was the Pushshift archive (Baumgartner et al., 2020): find post IDs for a time window in the archive, then fetch full threads from the API.
• Store it relationally: post ID, subreddit, title, body, comments, flair, created time. Porter (2018) did exactly this before topic modelling a year of a subreddit, month by month.

Access has changed since. Reddit tightened API terms and archive access in 2023, so check the current research-access routes before planning a project like this.

Cleaning, before any modelling

Replace non-letters with spaces, lowercase, expand contractions (“they’d” → “they had”), collapse whitespace and line breaks, tokenise, remove stop words. Boring, and it decides the quality of everything downstream (Topic Modelling with LDA).

Pitfalls

• Survivorship. Deleted comments, banned accounts and removed subreddits vanish, often exactly the content you’re studying.
• Bots. Automated accounts can dominate some communities. Detect and filter them.
• Lurkers. Most members never post (often 80–90% of a community), so posting data describes the vocal minority.

Ethics: public isn’t the same as fair game

Posts are public, but the people who wrote them didn’t sign up to be research subjects. The principles I followed:

• Analyse in aggregate, with automated methods.
• Never publish verbatim posts. A quoted sentence can be pasted into a search engine and traced to its author.
• Pseudonymise any account you need to refer to.
• Store data securely, on password-protected machines, and only for as long as needed.
• Take extra care with sensitive topics (crime, health, identity), where being identified could harm someone.

Cardiff’s Social Data Science Lab publishes a thoughtful framework for its COSMOS software: aggregate representations such as topic clusters and word clouds, decontextualised examples instead of quotes, and consent where content is sensitive. It’s a good model to borrow.

In practice

The engineering lesson I took away: design the data model around the questions you’ll ask later. Timestamps on everything, stable IDs, and a clear line between raw and cleaned data. The research equivalent of good logging (Observability).

Sources

• Baumgartner, J. et al. (2020). The Pushshift Reddit Dataset. ICWSM.
• Datta, S. and Adar, E. (2019). Extracting inter-community conflicts in Reddit. ICWSM.
• Porter, K. (2018). Analyzing the DarkNetMarkets subreddit for evolutions of tools and trends using LDA topic modeling. Digital Investigation.
• Social Data Science Lab, Cardiff University. COSMOS research ethics statement.