BVER · under review as a conference paper at ICLR 2027
A curriculum for sparse-reward, long-horizon tasks that grows start states outward from the goal and goals outward from the start at the same time, and steers the two toward each other.
Policies trained with BVER in simulation, from sparse rewards and without demonstrations.
Give a reinforcement learning agent a long task and a reward only at the very end, and it will almost never see that reward. A quadruped that is rewarded for standing on top of a box, but not for anything on the way there, has no signal to learn from.
In legged locomotion and manipulation this is usually worked around with reference motions, hand-designed curricula, or shaped rewards. Each of these needs a demonstration or engineering that is specific to one task. Automatic curricula avoid that by generating easier intermediate tasks during training, but existing methods grow from one side only: they propose goals for a fixed start, or start states for a fixed goal. The frontier then has to cover the whole distance to the other end by itself.
BVER takes two ideas from sampling-based motion planning. RRT extends the tree node closest to a uniform random sample, which pushes growth into large unexplored regions (the Voronoi bias). RRT-Connect grows one tree from the start and one from the goal until they meet. BVER applies both to curriculum generation: it grows intermediate start states out of the goal and intermediate goals out of the initial state distribution, biases both toward unexplored parts of the task space, pulls them toward each other, and trains a single goal-conditioned policy on both kinds of episode.
We test it on thirteen simulated tasks: point-mass mazes, quadrupedal locomotion with ANYmal D, and ring-on-peg transfer with a Franka arm. The rewards are sparse on all of them except one terrain built to compare routes, which adds a shaping term. Without a curriculum, PPO rarely or never solves them. BVER learns faster than all the reference-free curricula we compare against, on every task. On box climbing it reaches 95% success on a 0.4 m box in about 65% fewer iterations than the best of them, is the only one of them that learns to climb a 0.7 m box, and yields a policy that keeps working when the start, the goal, or the initial heading changes. Without a demonstration, it comes close to the sample efficiency of the reference-based curricula on the 0.4 m box and on ring-on-peg transfer.
Two frontiers, grown by random walks and pulled together
BVER keeps, for each direction, a finite set of states from which the starts or goals of the next training iteration are drawn. The sets grow by random walks, and the policy's own returns decide which states stay.
Scroll sideways to see the whole diagram.
The interactive demo runs this loop on a small 2D grid world. You can draw your own maze, step through one iteration phase by phase, and race forward-only, backward-only and bidirectional expansion against each other. Its Pseudocode tab shows the subroutines RandomWalk, Threshold and Mix as typeset in the paper.
What BVER assumes. Like RC, it needs a simulator that can be reset to arbitrary states and stepped with random actions, samples from the initial distribution, one known state that achieves the goal, and a bounded task space that contains the target task and can be sampled uniformly.
Thirteen tasks in three domains
Everything runs in Isaac Lab and is trained with PPO. The reward is sparse: +1 in every step at which the goal is within a fixed tolerance, and 0 otherwise. The one exception is the three-route terrain, which adds a shaping term (see training coverage). The task space is a box in position space, (x, y) or (x, y, z), around the point mass, the robot base or the ring. Curves show the mean success rate over five seeds with the standard error shaded.
In the renders, cyan marks the initial distribution, green the target goal, and the white box the task space.
Faster than the other reference-free curricula, and more robust
On every task we compare against PPO without a curriculum (vanilla PPO), a random curriculum that samples starts and goals uniformly from the free task space, and reverse curriculum generation (RC). On ring-on-peg transfer and box climbing we also compare against two methods that need a demonstration, reference state initialisation (RSI) and Backplay, marked with ★. Their demonstrations are rollouts of policies trained with BVER. All methods use the same number of environments and simulation steps per iteration. Of BVER's hyperparameters, pconnect and Ns are shared by all tasks; the others are set per domain.
We evaluate the policies trained to climb the 0.4 m box under three variations: initial yaw perturbed by up to ±90°, start positions sampled uniformly from the task space (goal fixed at g★), and goal positions sampled uniformly from the task space (start from ρ0). BVER's policy keeps a high success rate under all three.
RSI and Backplay train on states near the reference trajectory and see few states outside it, which likely explains their lower robustness. RC varies only the start states and always trains on the target goal, so its lower goal robustness is expected. The fourth axis is sample efficiency, measured as time to 90% success relative to the fastest run over all methods and seeds (Backplay, 300 iterations).
Because BVER biases its expansion toward unexplored task space, it should train the policy on the whole terrain rather than only on the easiest way to the goal. This terrain puts three different obstacles between the start and the goal: stairs in the middle, pillars on one side and a ramp on the other. The initial distribution spans the full width, so each obstacle is the closest one for about a third of the starts.
Only on this task we add a dense shaping term that rewards progress toward the goal, so that vanilla PPO finds a solution at all. Both methods get the same reward, and BVER's reward band still uses only the sparse term. We roll out each policy in 128 episodes (one seed) and record which obstacle each successful episode crosses.
Vanilla PPO only learns the ramp, the easiest of the three: every successful episode detours to it, even from in front of the stairs or the pillars. BVER's policy crosses all three: the stairs in 64% of successful episodes, the pillars in 21% and the ramp in 15%.
Box climbing has a single, simple obstacle. To test whether its hyperparameters carry over to more complex geometry, we run BVER on four terrains scanned from real environments: two freestanding boulders, a steep rocky slope, and a field of small stones. All PPO and BVER hyperparameters are reused from box climbing without retuning. Only the task space, chosen to roughly cover each terrain, the episode length (10 to 12 s instead of 4 s) and the number of iterations (3000, or 5000 on the rocky ascent) change, and the number of environments is halved for training speed at the same exploit-to-explore ratio. Vanilla PPO does not succeed a single time on any of the four, even with a higher entropy coefficient. Numbers are final success rates, mean ± standard error over five seeds.
| Terrain | Vanilla PPO | BVER | |
|---|---|---|---|
| Freestanding boulder I | 0.0% | 91.0% ± 1.7 |
| Freestanding boulder II | 0.0% | 84.4% ± 2.1 |
| Rocky ascent | 0.0% | 77.2% ± 2.2 |
| Small stone field | 0.0% | 59.8% ± 4.4 |
Expansion direction. We run forwards and backwards expansion on their own, each with the full environment budget. The bidirectional version reaches 75% success first on every task, tied only with the backwards expansion when climbing up the 0.7 m box, while each single direction fails to reach 75% on two of the eight tasks. Which single direction works depends on the task. On box climbing, the expansion that starts on top of the box is clearly stronger, likely because gravity lets random walks cross the box edge downwards more easily than upwards: at 0.7 m, the expansion from the ground never reaches 75% when climbing up and needs seven times more iterations than BVER when climbing down. On ring-on-peg transfer neither direction alone reaches 75%; each spreads from its own peg into the space between but rarely reaches the other. In the mazes, BVER needs 1.4 to 2.9 times fewer iterations than the better single direction. Combining both directions removes the need to know in advance which one suits a task.
Connect ratio. On the three mazes, we compare the default pconnect = 0.5 with seeding only from uniform task-space samples (pconnect = 0, no connect) and only from the other direction's solved set (pconnect = 1, greedy). Both extremes are slower on all three mazes. Without connecting, the two frontiers spread evenly over the task space and likely take longer to meet, needing up to 2.1 times more iterations to 75% success. Greedy connecting loses the coverage bias and needs up to 1.9 times more, most visibly in the N-maze, whose walls block the direct way toward the other frontier.
| U-maze | N-maze | Large maze | Ring-on-peg | Up 0.4 m | Up 0.7 m | Down 0.4 m | Down 0.7 m | |
|---|---|---|---|---|---|---|---|---|
| BVER | 160 | 800 | 140 | 120 | 600 | 3800 | 300 | 400 |
| Forwards only | 380 | 1100 | 440 | ∞ | 800 | ∞ | 400 | 600 |
| Backwards only | 440 | ∞ | 400 | ∞ | 800 | 3800 | 1700 | 2800 |
| No connect (pconnect = 0) | 340 | 1000 | 200 | n/a | n/a | n/a | n/a | n/a |
| Greedy (pconnect = 1) | 260 | 1500 | 240 | n/a | n/a | n/a | n/a | n/a |
What BVER needs, what it costs, and what we have not tested
The random walks run in the same parallel simulator as the policy rollouts and add about 28% to the wall-clock training time in our experiments.
BVER needs a simulator that can be reset to arbitrary states, random-action rollouts, and a bounded task space with a meaningful distance metric. All task spaces here are two- or three-dimensional position boxes with a Euclidean metric. Higher-dimensional or non-Euclidean task spaces are untested; learned distances could help there.
Like RC, the backwards expansion also assumes roughly reversible dynamics. A random walk from the goal reaching a state does not mean the goal can be reached from that state, as gravity on the box edge shows. Such states are only dropped once a rollout has shown them to be unsolvable. Learned dynamics models or rollouts of the current policy could replace the random walks. Finally, all results are in simulation; running the policies on hardware is future work.
BibTeX
@misc{bver2026,
title = {Bidirectional {V}oronoi-biased Exploration Curriculum
for Reinforcement Learning},
author = {Anonymous},
year = {2026},
note = {Under review at ICLR 2027}
}