Module 3 · Reinforcement learning for agents · Lesson 7

Multi-turn RL environments

How to turn an agent's tools into a reinforcement learning environment: what the trainer runs per rollout, which tokens the policy is trained on, how one final reward is shared across several tool-calling turns, what a run costs, and how to choose a framework or hosted trainer.

80 min 2 videos 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Free GPUs on Kaggle Hugging Face account and CLI

By the end of this lesson you can

Wrap an agent's tools and stubs as an RL environment with a reset hook, tool methods and a trajectory reward, using TRL's environment_factory
Explain why tool-result tokens are masked from the loss, and compute the share of a rollout's tokens the policy is actually trained on
Compute episode-level and step-level (anchor-state) advantages for a group of multi-turn rollouts, and say what each one blames
Estimate the token and dollar cost of a multi-turn RL run on a free GPU, a rented GPU and a per-token hosted trainer
Choose between TRL, verifiers, OpenEnv, ART, SkyRL and Agent Lightning for a given agent stack

The GRPO lesson trained one completion per prompt: the model writes, a grader scores, done. The triage step is not like that. The model reads a ticket, calls lookup_account, reads the result, perhaps calls search_help_center, reads that, and only then writes the triage record. One reward arrives at the very end, after three or four model turns with text the model didn't write in between. Which tokens should that reward train, which should it leave alone, and who gets the blame when the final record is wrong but the tool calls were fine? This lesson is about the machinery that answers those questions, and about what it costs to run.

Multi-turn RL environments

Multi-turn RL environments

Multi-turn RL environments

Sign in to keep reading.

Sign in to keep reading.