Regularised RL Improves Sim-to-Real Transfer

Preprint · under review

Danilo Vucetic^1,2,*, Mohamad H. Danesh^2,3,*, Florence Clerc⁴, Hsiu-Chin Lin^2,3, Jonathan Tremblay⁵

*Equal contribution

Download PDF·BibTeX

Top: Franka Lift with a shifted center-of-mass cube, PPO and FastTD3 fail to grasp it while SAC and SMM-AC succeed. Bottom: ANYmal-D climbing stairs with observation noise, PPO and FastTD3 fail while SAC and SMM-AC keep climbing. — Figure 1. Top: agents trained on the Franka Lift task are deployed on a cube with a different center-of-mass than seen in training; the first frame shows the start, the second the outcome. Bottom: agents trained on ANYmal-D Rough are deployed outdoors to climb stairs. With noise added to the observation, unregularised methods fail where SAC and SMM-AC succeed.

Abstract

Regularised reinforcement learning (RL) has provable robustness guarantees that should ease the transfer of policies from simulation to robots, known as sim-to-real. Yet, weakly regularised algorithms like Proximal Policy Optimisation (PPO) remain the de facto choice in sim-to-real pipelines, while the practical benefits of regularisation for transfer remain poorly characterised. In this work, we study the sim-to-real performance of commonly used RL algorithms, with the aim of characterising the effects of regularisation on robotic performance. To that end, we derive Soft MellowMax Actor-Critic (SMM-AC), a novel regularised RL algorithm based on the soft mellowmax value function, which carries strong robustness guarantees. On MuJoCo continuous control benchmarks, SMM-AC is competitive with baselines like Soft Actor-Critic (SAC). On real robots, SMM-AC achieves higher return, while exhibiting greater robustness to perturbations in observations, the environment, and dynamics than baselines. Together, these results demonstrate that regularised RL improves real-world policy deployment through better robustness to the sim-to-real gap, and changing real-world conditions.

We extend the soft mellowmax value function to continuous control as SMM-AC, an off-policy actor-critic built directly on SAC's machinery.
In MuJoCo, SMM-AC matches SAC and TD3 on seven of eight benchmark suites.
On a Franka arm and an ANYmal-D quadruped, every regularised policy transfers better from simulation than PPO or FastTD3, with SMM-AC transferring best.
Under every real-world perturbation tested — noise, mass shifts, friction, payload — SMM-AC keeps the largest share of its return and the highest absolute return.

1 Why regularisation should help sim-to-real

The sim-to-real gap is, precisely, an unmodelled change in dynamics that a policy must still handle at deployment. Regularised RL has a theoretical duality with robust RL: entropy- and KL-regularised policies carry provable guarantees against worst-case perturbations to reward or dynamics, and the size of that guaranteed "robust set" grows with the regulariser's strength. Soft MellowMax (SMM) value functions induce a larger robust set than entropy regularisation, but SMM has only ever been defined for discrete state-action spaces. We extend it to continuous control — where most robots actually live — and test whether its larger robust set turns into measurably better real-world transfer and perturbation resistance, not just a theoretical property.

2 Soft MellowMax Actor-Critic

SMM regularises the policy toward a reference distribution that is itself a softmax over Q-values, rather than the uniform distribution SAC's entropy term prefers. That reference is sharper and more performance-aware than SAC's, but it is also intractable in continuous action spaces: the integral defining it has no closed form. SMM-AC makes it tractable by learning three networks and training each with its own loss, built directly on top of SAC's machinery.

Critic — a regularised Q-target

V̂ = Qθ − (1/ω) log(πᵀ/πᵤ)

The critic bootstraps from a one-sample Monte Carlo estimate of the SMM value function, using the actor and reference network's own log-probabilities in place of the intractable reference integral. Everything else — replay buffer, target networks, critic architecture — stays identical to SAC, so the regulariser's effect is isolated.

Actor — anchored to the reference

log πᵀ − log πᵤ − ωQθ

Training the actor directly against SAC's loss is unstable here: without a grounding reference, the actor has nothing stable to track. Subtracting the reference network's log-probability anchors every actor update to a known, high-performing region, the same role a trust region plays in PPO/TRPO — but sharper, since the reference itself is Q-aware rather than just "close to the last iterate."

Reference network — a slow, stable anchor

J_ref = log πᵤ − αQθ

The reference network is trained with SAC's own actor loss, but at a much smaller learning rate, so it tracks the softmax of the Q-values as a slow-moving, low-variance target instead of chasing every critic update. This single addition is all that separates SMM-AC from SAC, making it easy to drop into FastSAC- or FlashSAC-style accelerated pipelines.

SMM-AC sits between SAC and PPO. SAC's policy is pulled toward being uniform over actions; PPO just keeps consecutive policy iterates close together. SMM-AC's reference is a softmax over Q-values, so it keeps the actor anchored to a known high-performing region while still exploring multiple good trajectories — which is exactly the property that should pay off once dynamics shift away from training.

3 Simulation: matching SAC and TD3 on MuJoCo

Before testing sim-to-real transfer, SMM-AC needs to be a competent continuous-control algorithm on its own. We implement it in CleanRL and sweep five seeds per configuration across eight MuJoCo benchmarks, comparing against SAC, TD3, and PPO.

Learning curves of SMM-AC, SAC, TD3, and PPO on eight MuJoCo continuous-control tasks — Figure 2. Smoothed episodic return across eight MuJoCo tasks. SMM-AC tracks SAC and TD3 closely on seven of eight tasks, at a comparable sample efficiency, with its one failure case on InvertedDoublePendulum-v4 attributed to reference-policy learning instability.

SMM-AC is competitive with the strongest baselines, reaching returns comparable to SAC at a similar rate everywhere except InvertedDoublePendulum-v4, where its performance collapses for reasons we could not fully diagnose. We don't expect SMM-AC to dominate SAC in simulation — the hypothesis under test is that its larger theoretical robust set pays off once a policy leaves the training distribution, which simulation alone can't probe.

4 Robotic experiments

We deploy policies trained in simulation on two robots and four tasks: a Franka Panda arm doing Reach and cube-Lift, and an ANYmal-D quadruped tracking commanded velocity on flat and rough terrain. PPO and FastTD3 use their best swept configuration per task; SAC and SMM-AC are each evaluated at three regularisation strengths (ω ∈ {400, 800, 1200}), with five seeds each.

The four real-world evaluation settings: Franka Reach, Franka Lift, ANYmal-D flat terrain, and ANYmal-D rough terrain — Figure 3. The four real-world evaluation settings. On ANYmal-D, the flat-terrain policy is blind to exteroceptive information, while the rough-terrain policy receives perceptual observations; arrows indicate the commanded direction of travel.

Sim-to-real transfer, before any perturbation

SMM-AC transfers best on Lift and on both locomotion tasks. In locomotion, every regularised configuration outperforms PPO and FastTD3 on both terrains, with the largest gap on rough terrain. On Lift, the two best configurations are both SMM-AC, and its best (ω = 800) is also far more consistent across seeds than PPO or SAC, whose large standard deviations show their transfer varying considerably from seed to seed.

Task	PPO	FastTD3	SAC (best ω)	SMM-AC (best ω)
Franka Reach	0.69±0.04	0.40±0.09	0.58±0.09	0.61±0.03
Franka Lift	133.1±54.8	123.1±4.1	125.8±69.9	152.9±2.5
ANYmal-D flat	18.9±5.6	23.6±0.7	25.3±0.3	25.9±0.4
ANYmal-D rough	8.6±3.7	13.7±4.1	19.3±4.9	21.9±1.6

Hardware return without perturbation, mean ± one standard deviation across seeds. Highlighted column is SMM-AC's best configuration per task.

Robustness to deployment-time perturbations

With each policy fixed, we perturb task parameters: observation noise on Franka, a 2× mass or 15mm center-of-mass offset on the cube, and on ANYmal lower friction, pushes, a 7kg payload, and proprioceptive noise. A configuration is only deployed under a perturbation if it fails fewer than 80% of simulated episodes under it — a safety rule applied identically to every method, which excludes PPO and FastTD3 from all ANYmal perturbation tests.

Return retained under real-world deployment-time perturbations across Franka Reach, Franka Lift, ANYmal flat, and ANYmal rough, for PPO, FastTD3, SAC, and SMM-AC at three regularisation strengths — Figure 4. Return retained (%) under each perturbation, relative to the same configuration's unperturbed return — 100% means no loss. At matched ω, SMM-AC keeps more of its return than SAC under every perturbation, and more than PPO/FastTD3 wherever they were deployable, with roughly half SAC's seed-to-seed variance on average.

Observation noise causes the largest losses, and is where methods differ most. On Franka, PPO loses most of its return under noise and FastTD3/SAC lose a substantial part of theirs, while SMM-AC keeps most of it. On ANYmal, proprioceptive noise separates SAC and SMM-AC more than any change to the physical system — since the dynamics are untouched there, the gap comes entirely from how each policy handles corrupted inputs.

Changes to the physical system cost less. A heavier or unbalanced cube costs SMM-AC nothing, while baselines lose part of their return. On ANYmal, SMM-AC loses little under lower friction, pushes, or payload on either terrain, while SAC loses noticeably more. Under every perturbation tested, the highest absolute return is achieved by SMM-AC.

Show full hardware results: absolute return under every perturbation

Perturbation	PPO	FastTD3	SAC ω=400	SAC ω=800	SAC ω=1200	SMM-AC ω=400	SMM-AC ω=800	SMM-AC ω=1200
Franka Reach
none	0.69±0.04	0.40±0.09	0.48±0.12	0.58±0.09	0.53±0.09	0.51±0.08	0.61±0.03	0.45±0.05
obs. noise (σ=0.005)	0.19±0.05	0.24±0.03	0.35±0.03	0.40±0.03	0.35±0.02	0.45±0.02	0.56±0.02	0.33±0.01
Franka Lift
none	133.1±54.8	123.1±4.1	92.9±83.8	125.8±69.9	94.9±86.0	148.5±8.6	152.9±2.5	123.9±69.4
cube mass 2×	97.5±11.0	102.9±9.5	81.9±5.2	99.1±4.4	87.1±5.4	152.5±9.1	151.6±3.1	128.3±8.2
CoM offset (15mm)	118.2±5.6	106.8±3.9	93.4±4.5	115.0±7.8	96.2±5.9	154.4±9.2	153.0±2.8	127.6±10.3
obs. noise (σ=0.001)	38.0±8.4	96.5±3.0	81.1±4.8	103.4±11.6	82.1±5.5	147.0±3.1	150.2±2.3	122.4±2.5
obs. noise (σ=0.005)	19.4±9.2	72.4±5.7	60.7±7.2	68.5±7.8	62.5±6.6	120.9±12.2	120.9±4.0	107.7±4.6
ANYmal-D flat
none	18.9±5.6	23.6±0.7	25.3±0.3	25.2±0.5	24.8±0.1	25.0±0.4	25.9±0.4	25.5±0.3
low friction	—	—	22.3±1.3	22.5±1.1	21.5±1.5	24.5±0.2	24.7±0.4	24.6±0.3
push	—	—	23.0±1.2	22.8±1.0	21.8±1.4	25.3±0.2	25.3±0.3	24.9±0.3
payload	—	—	24.1±0.7	22.6±1.1	22.8±0.9	25.1±0.2	25.2±0.3	25.0±0.2
prop. noise	—	—	20.7±2.2	20.5±2.0	20.5±1.9	24.7±0.2	24.7±0.4	24.7±0.3
ANYmal-D rough
none	8.6±3.7	13.7±4.1	19.1±1.6	19.3±4.9	18.8±3.4	21.0±1.2	20.8±1.3	21.9±1.6
low friction	—	—	16.4±1.5	17.0±0.8	17.4±1.0	18.5±0.6	21.2±0.6	21.9±1.1
payload	—	—	17.2±0.6	16.8±0.9	16.8±1.9	21.2±0.8	20.5±0.5	21.5±1.1
prop. noise	—	—	15.3±1.6	15.3±1.9	13.5±1.0	18.8±1.4	19.3±0.7	18.9±1.9

Mean ± one standard deviation of return across seeds. "—" marks configurations not deployed because they failed the simulation safety check. Highlighted entries are the highest return in each row.

5 Limitations & future work

SMM-AC's performance collapses on InvertedDoublePendulum-v4 for reasons we could not fully diagnose, likely tied to reference-policy learning instability. SAC's temperature autotuning has no analogue for SMM-AC yet — both α and ω are fixed and swept rather than learned, and we found naively autotuning either to be far too unstable. Because the optimal temperature varies across tasks and reward scales, adaptive temperature scheduling that preserves robustness guarantees is a natural next step. Limiting training-time domain randomisation also degrades every method, including SMM-AC, indicating that regularisation complements domain randomisation rather than replacing it.

6 BibTeX

@misc{vucetic2026regularised,
  title  = {Regularised RL Improves Sim-to-Real Transfer},
  author = {Vucetic, Danilo and Danesh, Mohamad H. and Clerc, Florence and
            Lin, Hsiu-Chin and Tremblay, Jonathan},
  year   = {2026},
  note   = {Preprint, under review}
}

PDF