Hierarchical Reasoning Doesn't Beat Recurrence at 300K Parameter Scale

At 300K parameters on 10x10 ARC grids, restoring the HRM/TRM fast/slow hierarchy onto a flat recurrent loop bought me nothing over just making the flat model wider and training it longer, once I matched compute. This is the result I most wanted to come out the other way, because the two-timescale split is an elegant idea.

I want to be careful about the scope of that claim, because the papers this builds on have models roughly 30 times larger, tested on harder grids and on other tasks the hierarchy is better matched to. This is a replication report at small scale, not a verdict on the architecture. Here's what carried over from the papers and what didn't when I ran the fast/slow split on a single RTX 3090, and the open question it leaves me with.

ARC, recurrence, and why you vote

ARC-AGI is a benchmark of small grid puzzles where each task gives you a handful of input-output examples, you infer the transformation rule, and you apply it to a held-out input. The grids are small and discrete, the rules are compositional and varied, and each puzzle has exactly one correct output. This is the kind of problem where iterative refinement feels like the obvious tool, because you don't read off a transformation rule in one glance, you converge to it by looking back and forth between the examples. The empirical story backs that up. Small recurrent models trained from scratch beat much larger pretrained LLMs on these puzzles, and the careful version of the finding is that the gains come from the recurrent inductive bias itself rather than from elaborate architecture. URM (Gao et al.), a universal-transformer variant, achieves 53.8% pass@1 on ARC-AGI-1, crediting the performance to recurrence and nonlinearity rather than architectural novelty.

I started my own experiments with the simplest thing that carries that inductive bias, a flat inner loop that applies shared transformer weights repeatedly and re-injects the input at each step. The reason was iteration speed. A small model on one consumer GPU finishes a run in a few hours, making my own control experiments tractable. Flattening the architecture to a single loop meant fewer moving parts to isolate when searching for improvements.

To understand how the results are presented, it is also important to understand the evaluation harness. A single forward pass on one puzzle gives one candidate grid, a noisy signal. So the papers generate many augmented views of the same puzzle (rotations, reflections, color permutations, all of which preserve the underlying rule), run the model on each, invert the augmentation on each output, and take a majority vote. That gives you two very different things to measure. Single-shot accuracy is whether one clean forward pass produces the right grid. Pass@K is whether the correct grid shows up anywhere in K candidates, which is what the voting harness rewards.

These two metrics respond to different levers, and understanding the distinction was critical for the project. Single-shot accuracy responds to capacity, as you make the model wider it gets more puzzles right in one pass. Pass@K responds to candidate diversity, you get diversity from training longer, augmenting more, or sampling. Any of these approaches will lift pass@K while leaving single-shot essentially flat, because they aren't making the mode more correct, they're spreading the candidates out so the right answer lands somewhere in the search space. The cleanest demonstration came out of a width sweep earlier in the project, where training my best mid-size model 1.5 times longer moved single-shot by 0.17 points (noise) and moved pass@1000 by 11.69 points (significant). The hierarchy story turns out to be a case of the training recipe quietly improving pass@K through diversity while the capacity lever sat untouched.

How hierarchy works, and why I started flat

The HRM/TRM family splits the recurrence into two modules running at different frequencies. A fast, low-level module fL takes the current low-level state, the high-level state, and the input, then refines the low-level state across several inner steps. A slow high-level module fH then takes a single step using the refined low-level state, integrating what the fast loop worked out, and after the whole nested process, the high-level state is decoded to the answer (HRM, Wang et al.; TRM, Jolicoeur-Martineau, who strips HRM down to one tiny two-layer network and still reaches 45% on ARC-AGI-1 at 7M parameters). The intuition is that the slow carry holds a stable line of reasoning while the fast loop does the local work underneath it, a division of labor a flat loop doesn't express. Wrapped around that inner structure is deep supervision, where rather than backprop through the entire nested recursion the model runs a segment, computes a loss against the target, detaches the hidden states, runs another segment from the detached state, and repeats for several segments with a weight update on each, plus a learned Q-halt head that decides when to stop.

None of that was in my flat loop. I'd collapsed the two-frequency structure into pure inner-loop recurrence early on, deliberately, because this seemed like the simplest control experiment to start with. Restoring fL/fH and the deep-supervision wrapper felt like finally following the paper rather than indulging my own simplification.

The surprise, and the controls that took it apart

The first serious hierarchical run, call it R8a, beat every flat baseline I had. It hit 25.40% single-shot against 23.72% for the best flat model at the time, and posted a project-record 65.58% pass@1000. To move forward with this architecture, I wanted to see it beat the flat baseline on both metrics, and it cleared both comfortably. For about a day my notes said the slow/fast structure was doing real work.

The first control was matched optimizer steps, and it removed most of the result. My deep-supervision recipe ran one weight update per supervision step and four supervision steps per batch, so at the same nominal training length the hierarchy was taking four times as many gradient updates as the flat baselines. Running a flat single-module model out to the same 320K optimizer steps the hierarchy actually saw, the flat h=192 model reached 24.6 to 24.9% single-shot across seeds, tied with the hierarchy inside seed noise, and hit 74.9 to 78.3% pass@1000, beating the hierarchy's 65.58% decisively. The pass@K record that had looked like the strongest evidence for the architecture was the step budget, not the structure. I ran two seeds per side, since the comparison gap was so small.

The second control isolated the architecture from the training recipe, because single-shot had tied rather than flipped and maybe the split was still contributing something. Running the hierarchy with a single supervision step, no deep-supervision carry and no detach machinery, just the H/L split applied once, already matched the full run on single-shot at 25.46%, and sweeping supervision depth from one to two to four was a flat plateau, 25.46 to 25.74 to 25.40. This is the capacity-versus-diversity distinction landing directly in the data. The split is a single-shot lever and supervision depth is a pass@K lever, and my experiment had conflated them so the deep-supervision recipe collected credit for a single-shot number it had nothing to do with.

The third control was fixed parameters, and it's the one I can only half claim, so I want to be precise. A two-module hierarchy carries two attention stacks, and at this width those alone nearly equal a flat single-module model's entire parameter budget, so there was no clean way to param-match the two. The only know that got me to the budget was trimming the MLPs, which cost accuracy for reasons separate from the hierarchy, about 6 points on its own. The fixed-param hierarchy landed at 18.5 to 20.2%, a clean 3.5 to 5.2 points below the flat model at matched parameters and matched steps. However, the matched-step comparison had a clear result, at equal compute the hierarchy ties a flat model on single-shot and loses on pass@K. The param comparison points the same direction but is confounded, since crippling the MLPs depresses accuracy by itself.

The controlled verdict is narrow and I think solid. The slow/fast split provides no benefit over a plain transformer once you hold parameters and optimizer steps fixed. It ties on single-shot at matched steps, loses on pass@K at matched steps, and loses on single-shot at matched parameters. The apparent win was four times the gradient steps plus twice the parameters, not the fast/slow architecture.

What replicated, and what didn't

The sharper reading of where the hierarchy lost is worth diving into, because it's not where I first assumed. The hierarchy and the flat model tied on single-shot at matched steps. The flat model's entire advantage was on pass@K. So the thing the hierarchy gave up wasn't accuracy, it was candidate diversity for voting. The slow carry acts as a consistency regularizer across augmentations, integrating toward the same answer regardless of which augmented view it sees, which is exactly what collapses the trajectory spread that pass@K voting feeds on. The flat loop's wider spread under augmentation is the whole story, and this is the same "diversity helps voting, not the mode" axis that has now shown up in five separate places in this project. It's the most reliable pattern I've found, and it's the one I'd hand to others working in this domain.

However, three things differ between my setup and the papers, and only one of them is scale. On scale, the gap is real, I'm at around 300K parameters against TRM's 7M, HRM's 27M, and GRAM's 10M, so roughly 30 times smaller. The papers also run 30x30 grids and I'm running 10x10. That isn't a neutral axis, since TRM's own text notes its MLP-only variant suffers from overcapacity on large 30x30 grids and needs self-attention there, which means grid size interacts with architecture choice rather than just scaling difficulty. The papers' strongest hierarchy wins lean on Sudoku-Extreme and Maze in addition to ARC, which are global-constraint propagation tasks the two-timescale carry is almost purpose-built for, the slow carry enforcing global consistency while the fast loop does local propagation. That's a structural match to the task, not a scale effect, and it's entirely possible the hierarchy helps on Sudoku at my parameter count and never helps on 10x10 ARC at any parameter count.

So I have one supported axis and two confounds, which means the decisive next experiments aren't a parameter sweep. Scaling parameters on 10x10 ARC with my augmentation harness holds both confounds fixed and varies the one axis the papers suggest is least likely to be the sole driver, so I'd probably just reproduce the tie-on-single-shot, flat-wins-pass@K pattern at bigger numbers and spend cloud money confirming what I already have. The informative runs are moving to 30x30 ARC at current small scale to test the grid confound, where TRM says architecture starts behaving differently, and adding Sudoku-Extreme at current scale to test whether global-constraint structure is what the carry actually needs. If the hierarchy overtakes flat when I change the task or the grid at fixed tiny scale, then it was never about scale, which is a sharper and cheaper finding than any scale sweep.

The question worth more than the answer

The hierarchy doesn't earn its keep at this scale, on this grid, on this task. But sitting with the negative, the more interesting question isn't whether the two-timescale split beats a flat loop head to head. It's whether the split buys you a structure you can build on that a flat loop simply can't support. A slow high-level carry is a different kind of object than a single recurrent state. It's a stable, slowly-evolving trajectory, and that's exactly the kind of place you'd want to attach mechanisms that need somewhere persistent to live, like stochastic latent transitions that sample a distribution over reasoning paths, or learned halting that reasons over the high-level state rather than the local one. A flat loop has nowhere natural to put those, while the hierarchy does.

That reframes the whole project for me. The fast/slow split might not be a win as a standalone architecture at small scale, but it could be the substrate that makes other refinement mechanisms work, ones that are redundant or unstable when you try to bolt them onto a flat loop. That's the thread I'm pulling next, and it's where the GRAM line of work comes back in, since GRAM's stochastic transitions are designed to live in exactly this kind of high-level trajectory.

A negative result at small scale is a statement about scale, grid, and task, not a verdict on the mechanism, and the most useful thing a failed architecture gives you is a precise next experiment rather than a closed door. If you've run the fast/slow split at larger scale, or on a global-constraint task like Sudoku, and watched these same controls go the other way, I'd love to hear about it!