How is reinforcement learning different from a supervised trading model?

A supervised model predicts a predefined label, such as next-bar direction, and you still design the trading rules around it. An RL agent optimizes actions directly against a reward signal, so entry, exit, and sizing behavior emerge from training. The price is a much harder learning problem: rewards are sparse and noisy, and a poorly designed reward produces a confident policy optimizing the wrong thing.

Do reinforcement-learning trading agents actually work?

Published and hobbyist results are mixed and difficult to verify. Non-stationary markets, transaction costs, and the ease of overfitting a reward inside a simulator mean impressive backtests routinely fail when run forward. A learned policy is best treated as a hypothesis needing out-of-sample and live validation at small size, not a solved system; nothing about the framework guarantees an edge.

What is a multi-armed bandit in trading terms?

The simplest reinforcement learner: no state, just a menu of options (sub-strategies, parameter sets) whose payoffs are observed as they are played. The bandit balances exploiting the current best against exploring the rest, using rules like epsilon-greedy or upper confidence bounds. Its modesty is the appeal: fewer moving parts to overfit, which is why chart-level 'RL' tools are usually bandits.

What reward should a trading agent optimize?

Something risk-aware and cost-inclusive: raw profit rewards volatility-chasing, so practice favors risk-adjusted returns, drawdown-penalized profit, or cost-netted reward per unit exposure. Every choice has exploits, an agent penalized for drawdown may learn to stop trading, so reward design is iterative: inspect what behavior the reward actually bought, then patch the loopholes it found.

Why do RL trading results collapse from backtest to live?

The agent optimized its simulator, and the simulator was not the market. Missing costs and slippage, fills the real book would never give, information leaking through the state, and rewards exploitable inside the training loop all produce policies whose edge is an artifact. Add non-stationarity, the live market drifting away from the training distribution, and collapse is the default outcome that careful validation exists to prevent.

What features belong in the state?

The same scale-free, leakage-clean features any market model wants, plus the agent's own situation: position, unrealized profit, time in trade. That self-awareness is what lets a policy learn management behavior, not just direction. State bloat is the standing temptation; every added feature enlarges the space in which the agent can memorize training-era coincidences.

Concept

Reinforcement-learning Policies

Reinforcement-learning Policies are Machine Learning concepts.

What are Reinforcement-learning Policies?

Reinforcement learning (RL) frames trading as sequential decision-making. An agent observes a state (features describing the market and its own position), picks an action (go long, go flat, resize), receives a reward (profit, or a risk-adjusted variant), and updates its behavior to collect more reward over time. The policy is the product of that training: the learned mapping from states to actions. Where supervised models predict a label and leave the trading rules to you, RL optimizes the decision rule itself, so entries, exits, and sizing can in principle be learned inside one loop.

The machinery has a defined lineage: temporal-difference learning and Q-learning came out of the reinforcement-learning research of the 1980s, the Sutton-and-Barto textbook systematized the field, and deep RL's game-playing results in the 2010s drove the current wave of interest. Finance experimented early, with recurrent reinforcement approaches to trading published in the late 1990s, and the honest summary since is unchanged: the framework fits the problem beautifully and the data fights it viciously.

Implementations range from Q-learning and policy-gradient methods (often with neural networks inside) down to multi-armed bandits, the stateless simple case: repeatedly choose among a set of sub-strategies, watch the payoffs, and shift weight toward what has been working while still exploring alternatives. Chart-level implementations usually sit near the bandit end. The honest caveat is that markets are non-stationary and rewards are extremely noisy, so a policy can confidently learn patterns that were never real, and validating one is harder than validating a supervised model.

In trading applications the design choices are the system. The state is a feature-engineering exercise with everything that implies about leakage and stationarity; the action space decides whether the agent picks directions, sizes, or sub-strategies; and the reward is where projects quietly fail, since an agent optimizes exactly what the reward pays for, including simulator quirks, cost omissions, and drawdown loopholes. Evaluation has to be walk-forward and regime-spanning, because a policy is a fit to its training market first and a strategy second.

How to evaluate an RL trading policy

A policy cannot be judged by its equity curve alone; the design and the validation regime are the evidence.

1Inspect the three definitions first: what the state contains (and whether it is computable live), what actions the agent may take, and exactly what the reward pays for.
2Audit the simulator: transaction costs, slippage, latency, and realistic fills, since an agent will exploit any free lunch the simulator accidentally offers.
3Demand walk-forward results: train on one span, act on the next, rolling forward, with performance reported only from the unseen segments.
4Compare against dumb baselines: a static allocation or a simple bandit over the same sub-strategies; the RL machinery must beat what it complicates.
5Watch for policy drift in online versions: continuously updating agents change behavior over time, so yesterday's validation describes yesterday's policy.
6Deploy, if at all, at research size: live behavior under real frictions is part of the evaluation, not a victory lap.

How traders use it

As an adaptive allocator: a bandit-style policy scores a menu of sub-strategies (trend, mean-reversion, breakout) by recent reward and routes exposure toward the current winner, an automated form of strategy switching and rotation.
For trade management: with the reward shaped to penalize drawdown or holding time, the policy learns when to exit or resize rather than only which direction to pick.
As a continuously updating model: policies can adjust with every new bar in an online, incremental learning fashion, which suits regime-prone markets but also means behavior drifts and yesterday's backtest describes yesterday's policy.
At execution level: the best-documented industrial uses optimize order placement and scheduling, where feedback is fast, rewards are dense, and the environment is closer to stationary than directional prediction ever is.
As an arbitration layer: instead of replacing signals, a policy can learn how to weight an existing committee of them, the learned counterpart of fixed ensemble voting.

RL policies vs neighboring approaches

Neural Networks: Networks are function approximators; RL is a training objective. Deep RL puts one inside the other, but a network trained on supervised labels answers a prediction question while a policy answers a decision question, and the failure modes differ accordingly.

Online/incremental Learning: Online learning describes when a model updates (continuously, per observation); RL describes what it optimizes (cumulative reward through actions). An RL agent may learn online or from batches, and an online learner may be entirely supervised.

Ensemble Voting of Signals: A fixed ensemble weights its members by design; a bandit-style policy learns the weights from realized payoffs and keeps adjusting them. The learned version adapts to regime change and also chases noise, which is precisely the trade to evaluate.

Concept family

Machine Learning

32 concepts mapped · 32 in the Library