Some terms from Reinforcment learning as I understand them

24 Aug, 2026

Last Updated: Aug 24, 2026

TLDR:

graph TD
A[Reward, What happened?] --> B[Return, How good was the future from here]
B --> C[Value/Q, What do I 'normally' expect from this state/action?]
C --> D[Advantage, Was this action better or worse than expected?]
D --> E[Policy Gradient, How should I alter the policy probabilitlies?]
E --> F[PPO, How do I make that update without changing the policy too violently?]
F --> G[GRPO, How can I estimate relative advantage from groups of LLM outputs without a seperate value model?]

1. RL, Reinforcement Learning: A machine-learning framework in which an agent interacts with an environment, takes actions, receives rewards, and learns a policy that tends to maximize its expected cumulative future reward.

2. Environment: RL comes with an environment in which the agent interacts with. Depending on what the agent does, the environment will also 'update' itself. eg, a chess environment or a game environment

3. State: Given environment, there is a specification of what state, at any given moment, the model is in. Precisely speaking, State is a representation of the environment at a particular time. (S_t) We want the State to be Markovian(>asterisk) because we want the State at any given time to capture all information relevant to predicting future dynamics and rewards, given the action.

4. Action: Given a state and environment, the model has available to it an action space, i.e the set of actions it can take. For example, in the chrome Dino game, the actions may be {jump, duck, do nothing}

5. Policy: Policy is a mathematical function that takes in the current state and provides an action to take, or more specifically a probability distribution of actions to take.

6. Reward Functions: Reward function encodes what counts as success for a given learning process. Where the model meets its maker lol. It can be represented as a function over (state, action, new_state). Keeping the same Dino example, we can have a very simple reward function as {if a collison occurs: -10, for each timestep survived: +0.02, +1 if an obstacle is successfully passed}.

7. Reward Signal, Rewards: The reward function computes the scalar quantiy of reward. For a given log of interactions by the agent, we will have the specific log of rewards. (r1 = +0.02, r2 = 1...r_n = -10) and so on.

8. Returns ($G_t$): Reward can be represented as R(s,a,s′), though other equivalent formulations are also common. The cumulative future reward from time t, typically discounted: $G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots$ where 0≤γ≤1 is the discount factor. Return lets the agent evaluate actions in terms of their long-term consequences, not merely their immediate reward.

9. Rollouts: Rollout is the actual sampled trajectory that the agent takes in a given run. For the Dino game, it can be something like: {s0​,a0​,r1​,s1​,a1​,r2​,…,sT​}.

10. Q: Q is defined over state and the action taken with a given policy function. It is the expected return if the agent takes action a in the state s and continuous following the policy $\pi$. $Q^{\pi }=E_{\pi }[G_t|S_t=s,A_t=a]$

11. Value: Value is defined over the state and is the expected return from being in state s and then following policy $\pi$. $V^{\pi }=E_{\pi }[G_t|S_t=s]$

12. Advantage: Now we have Q and V. V captures the 'ambient/normal' expected returns at a given state; Q captures the expected returns having done an action a. Advantage aims to capture the following intuition. "Given that I was already in this situation, how much better was the action/output I chose than I would normally expect." $A_{\pi }(s,a)=Q_{\pi }(s,a)-V_{\pi }(s)$

13. Policy Gradient: A family of RL methods that directly changes the parameters of the policy in a direction that increases expected return. Actions that have positive advantage become more likely; actions with negative advantage become less likely. ${\nabla }_{\theta }J(\theta )\propto E[A(s,a){\nabla }_{\theta }log({\pi }_{\theta }(a|s))]$

14. PPO: Short for Proximal Policy Optimization. A widely used policy-gradient algorithm that updates a policy while constraining the update so the new policy does not move too far from the old policy in a single training step.

15. GRPO: Short for Group Relative Policy Optimization, introduced in DeepSeekMath. For a given input, the policy generates a group of candidate outputs, each receives a scalar reward, and the outputs are evaluated relative to the other outputs in that group. Those relative scores are then used to update the policy toward outputs that performed better than their peers. Unlike PPO-style actor-critic methods, GRPO avoids requiring a separate learned value model.

16. Verifiable Rewards: A reward whose correctness can be determined automatically or objectively from the model's output, usually using a verifier such a a program, test suite, game engine, or some known-answer checker.