Pose-Based Rep Counting

Counting reps is easy. Not counting the things that look like reps is the real work, and each rule should come from watching the footage where it went wrong. I learned this building a tool that turns raw training clips into short, counted clips (the reel maker on my ideas list).

The core method

• A pose model (MediaPipe Pose Landmarker) gives 33 body landmarks per frame. Convert them to pixels before computing angles; normalised coordinates distort angles in a 9:16 frame.
• Per frame, use the side the camera sees best (landmark visibility).
• A rep is the elbow angle dropping past a low threshold and coming back past a high one (hysteresis), with the thresholds adapted to each set’s range of motion using percentiles rather than fixed degrees.
• Smooth with a short rolling median sized in seconds, not frames, so it works at any sample rate.

What looked like reps but wasn’t

False rep	Fix that worked	Fix that didn’t
Standing, walking, reaching for the camera, adjusting a belt (the arms bend)	Hands planted: wrist drift per rep ≤ 0.35 torso lengths (real reps ~0.2, fakes ≥ 0.47)	Shoulder travel: the fakes moved more
Moving about before getting into position	Entry: counting starts at the first steady lock-out at the same shoulder height as the top of the reps	Straight arms alone (matches standing); ankle height (perspective when walking towards the camera)
The last dip with feet pushing on the bench	Legs vs body travel > 1.5 means the legs are helping (clean dips are 0.4–0.8); dips only	Applying it to wall or pike work, where the feet are planted on purpose
A half rep at failure	Must return within ~12° of the set’s lock-out angle and back to the top shoulder height
A rep at the edge of the detected window	Count 3 s beyond each detected segment

Sampling

Sped-up footage makes each rep a fraction of a second. At 10 fps a rep was two or three samples and read as a hold; 30 fps fixed it, at roughly half real time to process.

Classifying the exercise

Per frame, body geometry (upright or inverted, torso vertical, hips above shoulders, hands at hip level), smoothed over about 2 s, then grouped into sets: a pause over 6 s splits a set, and a set needs at least 3 clean reps. The same position can be two exercises: inverted, vertical and repping is wall handstand push-ups; no reps is a handstand hold.

The process lesson

Every rule came from looking at the frames where it was wrong, not from guessing. A debug plot of the angle with the detected reps overlaid, plus a strip of frames, found each problem in minutes. And ground truth from the person who did the set (“that one failed”, “that bit was sped up”) was the fastest feedback loop of all: keep a small ground-truth file per test session, the same way you’d keep fixtures for any classifier.

Related: Image Analysis Fundamentals, Dips and Ring Dips.