SEAR: Sample Efficient Action Chunking RL

Abstract

Action chunking improves exploration and accelerates value propagation in long-horizon reinforcement learning, but naively applying off-policy methods to the temporally extended action space at reduced decision frequency offsets these gains, leading to poor sample efficiency. Existing online action chunking methods address these issues through computationally expensive critic-only approaches or by relying on offline data. We introduce SEAR, a sample-efficient off-policy algorithm that enables online reinforcement learning with action chunks by addressing both challenges. To handle high-dimensional action sequences, SEAR employs a causal transformer critic trained with multi-horizon targets that provide a training signal for every prefix of an action chunk, effectively increasing the useful gradients per sample. To restore decision frequency, SEAR operates with a receding horizon and random replanning, ensuring uniform state coverage while combining the fast value propagation of large chunks with the reactivity of small ones. SEAR outperforms state-of-the-art online reinforcement learning methods including SimbaV2 on challenging Metaworld manipulation tasks. Beyond online RL, applying SEAR to the offline-to-online method QC improves its performance on the demanding OGBench cube-triple tasks.

Results

Metaworld-Hard: the 20 hardest Metaworld ML1 tasks (by SimbaV2 performance), with success measured at the last step of an episode. IQM with 95% bootstrap CIs.

Comparison to the state of the art

IQM success on Metaworld-Hard vs. samples for SEAR and baselines — **Online RL, Metaworld-Hard.** SEAR reaches over 80% IQM success. SimbaV2 and CQN-AS, the state of the art without and with action chunking, plateau around 60%. A naive chunked SimbaV2, SAC and T-SAC stay around 20%.

IQM success on OGBench cube-triple-play for SEAR-QC and QC — **Offline-to-online, OGBench cube-triple-play (1M).** Adding SEAR's multi-horizon targets and random replanning to QC, with everything else unchanged, beats QC, most clearly early in online training.

Method

An action-chunking policy $\pi(a_{t:t+N}\mid s_t)$ predicts $N$ actions at once and runs them open-loop. That gives structured exploration and unbiased $N$-step returns, but brings two problems: the critic must evaluate an action space that is $N$ times larger, and the actor is only queried in every $N$-th state. SEAR addresses each problem with one component.

Critic

Causal transformer + multi-horizon targets

One Q-value and one Bellman target for every prefix of a chunk, so the number of training targets grows with the chunk size.

Actor

Random replanning + receding horizon

During training, execute a random prefix of each chunk. The actor then sees every state, so short receding horizons at evaluation stay in-distribution.

Action-chunked maximum-entropy RL

SEAR builds on the maximum-entropy framework. The entropy bonus is computed and discounted per action, so the optimal policy does not depend on the chunk size, and $N=1$ recovers standard SAC-style MaxEnt RL. The action-chunk-conditioned Q-function scores a whole chunk executed open-loop and following $\pi$ afterwards:

$$Q^\pi(s_t, a_{t:t+N}) = r_t + \mathbb{E}_p\Big[\sum_{n=1}^{N-1}\gamma^n r_{t+n} + \gamma^N V^\pi(s_{t+N})\Big].$$

1 · The causal transformer critic

Summing a chunk's $N$ rewards into one target throws away information. Instead, SEAR computes a target for every prefix length $k = 1,\dots,N$:

$$G^{(k)}(s_t,a_{t:t+k}) = \underbrace{\sum_{i=0}^{k-1}\gamma^i r_{t+i}}_{k\text{-step return}} + \gamma^k\,\hat Q_{\bar\phi}\big(s_{t+k},\, a'\big),\qquad a'\sim\pi_\theta(\cdot\mid s_{t+k}).$$

$G^{(k)}$ only depends on the first $k$ actions, which is the same structure a causal transformer has. Each action token attends only to the state and earlier actions, so one forward pass gives all $N$ prefix Q-values. The causal mask also ensures that $Q^{(k)}$ only uses the $k$ actions that enter its $k$-step target $G^{(k)}$. Later actions of the chunk do not affect that return, so they are masked out. The critic regresses each Q-value onto its own target:

$$J(\phi) = \frac{1}{2N}\sum_{k=1}^{N}\Big(Q_\phi^{(k)}(s_t,a_{t:t+k}) - G^{(k)}(s_t,a_{t:t+k})\Big)^2.$$

Multi Horizon Critic Targets Naive N-Step Target

On Metaworld-Hard, both the intra-chunk targets and the transformer backbone matter. Only their combination clearly outperforms non-chunking baselines such as SimbaV2 (about 60%):

Critic ablations on Metaworld-Hard — **Critic design.** Multi-horizon targets are the most important component: fitting only $Q^{(N)}$ drops to about 30%. Bootstrapping with a single-step policy (“No Action Chunking”) and replacing the transformer with an MLP also lower success, though the MLP variant stays competitive.

2 · Random replanning restores state coverage

With a fixed chunk size, the policy is only evaluated in every $N$-th state of a trajectory. When initial states are narrowly distributed, as in many manipulation benchmarks, large parts of the state space are visited but never used to query the actor. A receding horizon at test time (replanning after $h < N$ steps) then queries the actor exactly in those out-of-distribution states, and that cancels most of the gain from the higher decision frequency. SEAR's fix is simple: during data collection, execute only a random prefix of each chunk before replanning.

actor queried (replanning state) visited open-loop only

chunk size N

Each row is one training episode from the same initial state. The bars show how often the actor was queried at each time step, accumulated over all episodes. Uniform chunking leaves the states between replanning points uncovered. Random replanning spreads the queries over every step, so any receding horizon $h$ lands in well-trained states.

In practice, random replanning determines how much SEAR benefits from a receding horizon:

Effect of random replanning across receding horizons — **Random replanning.** SEAR-10 with and without random replanning, evaluated across receding horizons. With random replanning, SEAR fully exploits the higher decision frequency of a shorter receding horizon. Without it, the actor is queried in the coverage gaps shown above, and most of that gain is lost.

Ablations

Chunk sizes and receding horizons

Sample efficiency for SEAR with chunk sizes 1, 5, 10, 20 — **Chunk size.** Every SEAR variant that uses action chunking beats SimbaV2. SEAR-10 balances value propagation and decision frequency best. SEAR-1 (no chunking) is slightly below SimbaV2, so the transformer alone does not explain the gains.

IQM success as a function of receding horizon for different chunk sizes — **Receding horizon.** Training on long chunks and deploying with a shorter receding horizon beats training on short chunks. The best horizon is 3–5 steps for every chunk size. SEAR trained with chunks and deployed at $h=1$ even beats SEAR-1 and SimbaV2, which makes it a good way to obtain a single-step policy.

BibTeX

@inproceedings{nagy2026sear,
  title     = {{SEAR}: Sample Efficient Action Chunking Reinforcement Learning},
  author    = {Nagy, C F Maximilian and Celik, Onur and Gospodinov, Emiliyan and
               Seligmann, Florian and Liao, Weiran and Kaushik, Aryan and Neumann, Gerhard},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=bTE6KrCVKn}
}