Skip to content
matthewcm.dev

    ↑ ↓ move · ↵ open · tab scopeopen graph →

    Eureka: Automating Reward Design with Coding LLMs

    NVIDIA's Eureka puts an LLM in a loop around reinforcement learning — it reads the environment code, writes reward functions, trains, reflects on the results and rewrites. A blueprint for closed-loop agents.

    Growingtended planted 2 min read
    On this page

    Eureka (Ma et al., NVIDIA, ICLR 2024) answers a question I find more interesting than the robotics it was built for: what happens when an LLM doesn’t just respond to a prompt, but drives a whole closed-loop design process?

    The task is reward design for Reinforcement Learning — notoriously fiddly work that human experts usually do by trial and error. Eureka hands it to an LLM acting as a “reward architect”.

    The loop

    1. Read the environment. The LLM gets the raw source code of the simulation as context, so it understands the task’s mechanics and objectives directly rather than through a description.
    2. Write rewards as code. It generates several candidate reward functions in Python, zero-shot, with no templates.
    3. Train. Each candidate trains a policy in a GPU-parallel physics simulator.
    4. Reflect. Eureka feeds back a reward reflection: statistics on how each component of the reward behaved during training. In effect, a structured critique.
    5. Revise. The LLM keeps the best candidate, works out which terms helped or hurt, and writes improved versions. Repeat for a fixed number of iterations.

    It beat expert human-written rewards on most of the benchmark tasks, including teaching a simulated hand to spin a pen.

    Why code, and why reflection

    • Code is inspectable. The LLM can reason term by term: “the distance penalty dominates; reduce its weight”. A natural-language reward couldn’t be executed or debugged like that.
    • Reflection is the key ingredient. Without feedback, it’s one-shot generation and a guess. With per-component training statistics, it’s guided search — closer to evolution than to autocomplete.
    • There’s a budget. Fixed iterations and samples per round. Explicit stopping criteria are what stop an agentic loop becoming an expensive infinite one.

    The general pattern

    Take the robotics away and Eureka is the shape of most useful agents I’ve seen: propose → execute in a real environment → observe metrics → reflect → revise, with a budget. Coding agents do it with tests (write code, run tests, read failures, fix). It’s the same agent → environment → feedback loop described in What Makes an AI Agent, with a TDD flavour: like a developer who starts writing tests just by reading the source code and the requirements.

    In practice

    The part I’ve stolen for my own tooling is reflection as a first-class artefact. When an automated step runs, I want a short structured record of what it tried, what happened and what it would change — not just a pass/fail. It makes the next iteration better, and it makes the loop auditable by a human. That’s as true for a CI pipeline as for an agent.

    • #ai-agents
    • #reinforcement-learning
    • #llm
    • #feedback-loops
    • #papers
    • Reinforcement LearningGrowing
      Which turns reward design itself into a loop: propose a reward → train → observe → reflect → revise. Eureka hands that loop to an LLM.
    • What Makes an AI AgentGrowing
      …t a language model does the reasoning and tools do the acting — APIs, a browser, a shell, a code interpreter. Eureka is a lovely worked example: an LLM writes code, runs an experiment, reads the results and rewrites the…

    ↪ Links to (2)