Thinking Fast and Slow Enables Stochastic Guidance

In the last post, I covered how a flat recurrent architecture outperformed the fast/slow hierarchy at the 300K parameter scale. However, a slow high-level carry is different than a single recurrent state, and the interesting question was whether that structure supports refinement mechanisms a flat loop simply can't. I now have a replicated answer, and it's yes! Restoring the two-timescale hierarchy converts GRAM-style hidden-space stochasticity from redundant with input augmentation into orthogonal to it. Combining the two diversity sources compounds pass@1000 by +5 to 9 points beyond the better of either alone, and that compounding held across two independently trained seeds and four sampling seeds. The same injection mechanism produced nothing on the flat loop.

The second finding takes most of it back. GRAM's training objective imposes a heavy accuracy tax on the deterministic backbone, and that tax generalizes from the flat loop to the hierarchy. The orthogonal stack tops out around 47% pass@1000, which is 28 to 31 points below what plain augmentation achieves on a clean flat model. The mechanism is real and replicated, but it isn't a practical win, and the useful part is understanding why both halves are true.

What GRAM is, and why the flat loop rejected it

GRAM (Generative Recursive Reasoning, Baek et al.) turns deterministic recursive reasoning into a probabilistic model by adding reparameterized Gaussian noise after each deterministic latent update, so instead of following one trajectory to one answer the model samples a distribution over reasoning trajectories. Training uses amortized variational inference, with a prior noise distribution used at inference and a target-conditioned posterior that gets to peek at the answer during training, and the KL between them regularizes how much the noise is allowed to encode. The motivating case is multi-solution reasoning, with N-Queens as the clearest example, where a single board admits many valid solutions, deterministic recurrent models collapse every run onto the same trajectory, and GRAM's sampling spreads across several of the valid answers.

I first tried this in my flattened setting, injecting the stochastic transition into the pure inner loop, and the result was a clean null. The noise produced a real majority-vote lift over a single deterministic trajectory, but in the controlled three-way comparison it added nothing beyond the augmentation diversity the evaluation harness already supplied. The candidates it generated were reshuffles of candidates augmentation would have found anyway. At the time I read this as a task mismatch, since ARC puzzles have one correct output and the posterior has no legitimate extra signal to encode.

Two things later complicated that reading. First, GRAM's own paper contains a quiet footnote saying they also tried injecting noise into the low-level state and found it didn't improve performance. I had reproduced their negative without realizing it, because their design puts the stochastic transition in the high-level trajectory, not the fast inner loop, and my flattened architecture had no high-level trajectory to put it in. Second, my own scope caveat on the flat null had flagged exactly this, that the missing slow carry might be the critical piece rather than the task. Once the hierarchy experiments from last post left me with a working H/L split and a clean negative on its standalone value, the obvious next experiment was to put the stochasticity where GRAM says it belongs and ask the redundancy question again.

The experiment, three arms and the controls around them

The design isolates two diversity sources and their interaction. Arm A is augmentation only, the standard harness from the whole project, many augmented views of each puzzle through a deterministic forward pass with votes across the inverted outputs. Arm B is perturbation only, a single fixed augmentation per puzzle with the candidate set generated purely by resampling GRAM's prior noise, so the only thing varying is the stochastic trajectory. Arm C crosses them, augmentation and prior resampling together. The question is whether arm C beats the better of A and B alone. If the two diversity sources are redundant, C matches max(A, B). If they're orthogonal, C compounds beyond it.

The whole claim rests on arm B not quietly smuggling in augmentation diversity, so the controls matter as much as the arms. I verified at the code level that arm B collects candidates only from the single fixed augmentation index, so its diversity comes from prior draws and nothing else. I then checked that the compounding isn't a sampling fluke by re-running the three-arm evaluation under four different sampling seeds on the same trained checkpoint, re-seeding only the GRAM draws while holding the augmentation pool fixed, and the compounding stayed positive at every seed. Single-seed claims are also underpowered at this resolution, since the deterministic backbone itself swung 7 points of pass@1 between my two training seeds (19.5% versus 26.6%), so I trained a second seed from scratch, seven hours on the 3090, and re-ran the full three-arm harness on it. The seed gap doesn't threaten the comparison, since the compounding is computed within each trained seed and the backbone difference cancels.

The flip

Here are the three arms on the first trained seed, pass@K at two candidate budgets.

The second trained seed showed the same structure, with the compounding banded at +4.6 to +6.5 points at N=100 and +5.2 to +7.1 at N=1000 across the four sampling seeds, overlapping the first seed's bands of +2.6 to +5.2 and +5.2 to +9.1. Every one of the sixteen measurements per budget came back positive.

The mechanism is what makes the numbers coherent rather than mysterious. Arm B alone is weak, diverse but mostly wrong, just as it was on the flat loop. What changed is the relationship between its errors and augmentation's errors. On the flat loop the two candidate pools overlapped, and their union caught nothing new. On the hierarchy, arm B's errors are decorrelated from arm A's, so the union covers puzzles neither source catches alone, and the compounding is the direct signature of that decorrelation. In last post's terms, the slow carry acts as a consistency regularizer across augmentations, collapsing the trajectory spread that voting feeds on. Prior resampling in the high-level trajectory is a diversity source that lives inside the thing doing the collapsing, which is plausibly why it can reach candidates augmentation can't. A flat loop has no equivalent place to put it, and when I put it there anyway, the noise just re-randomized a spread the loop already had.

That's the answer to last post's question. The two-timescale split supports at least one mechanism the flat loop demonstrably can't, and the effect is significant and repeatable across multiple training seeds.

The tax

The second half of the story is the cost. Everything above compares arms within a GRAM-trained model, and the GRAM-trained model's deterministic backbone is substantially worse than its non-GRAM counterpart. Arm A, the augmentation-only condition directly comparable to last post's numbers, lands around 39% pass@1000 on both trained seeds. The deterministic hierarchy from last post hit 65.58% on the same harness, and the corrected flat baseline at matched steps sits at 74.9 to 78.3%. So co-training the variational objective cost the backbone roughly 26 points of pass@1000 before the stochastic sampling gives any of it back, and the best orthogonal stack, arm C at 46.75%, remains 28 to 31 points below plain augmentation on a clean flat model. Orthogonal diversity stacked on a taxed backbone loses to plain augmentation on an untaxed one.

I want to be precise about what the tax result does and doesn't establish, because it's the number a careful reader should push on. What I can say is that the tax generalizes, since the flat GRAM runs paid it and the hierarchical GRAM runs pay it at similar magnitude, so the slow carry doesn't buffer it away. What I can't say is that the tax is intrinsic to GRAM rather than to my training setup. My runs sit at a fixed 320K-step budget with a particular KL warmup schedule and a bottlenecked noise dimensionality, and any of those could be the binding constraint rather than the variational objective itself. GRAM's published results at 10M parameters on 30x30 grids shows no sign of a tax like this, which is itself evidence that the effect I'm seeing is at least partly a property of this scale and this budget. The claim is that the tax is real, replicated, and unbuffered by the hierarchy in my setup, not that GRAM is inherently expensive.

The papers already knew half of this

The satisfying part of writing this up was discovering how much of the result the GRAM paper itself anticipates. Their low-level footnote predicted my flat null, as covered above. But they also ran a version of my orthogonality question directly, isolating sampling from augmentation on ARC-AGI-1, observing that the augmentation process alone already creates a wide variety of samples, and investigating the combination of augmentation with sampling rather than treating one as a replacement for the other. That's the augmentation-sampling interaction I've now measured at 300K parameters on 10x10 grids, characterized by them at 10M on 30x30.

I'm framing my contribution that way because it's a stronger claim posture than standing alone. A single-lab, two-seed result at tiny scale is easy to discount, but a small-scale reproduction of an interaction the original authors already flagged is consistent with established evidence rather than resting on my seeds alone. What my version adds is the flat-versus-hierarchy contrast, since the paper injects into the high-level trajectory by construction and never had a flattened variant to compare against beyond the footnote. The redundant-to-orthogonal flip when the slow carry is restored is, as far as I can tell, the new observation here.

Where this leaves the project

The practical lever for small-scale ARC remains exactly what it was two posts ago, augmentation plus width on a plain flat transformer. Nothing in this post changes that recommendation. What changed is the map. The hierarchy question now has a precise answer, the fast/slow split is not a standalone improvement but it is a substrate, and hidden-space stochastic guidance is the first mechanism I can show working on that substrate and failing without it.

There's one direction that could turn the mechanism practical. The tax comes from co-training the variational objective with the backbone, but the orthogonal diversity comes from sampling at inference. Those don't obviously have to be coupled. Decoupling the exploration sampler from the training objective, so that prior-style resampling runs against a clean deterministically-trained hierarchy, would ask whether you can collect the compounding without paying the tax. If the decorrelation survives, the orthogonal stack sits on a 65-point backbone instead of a 39-point one, and the arithmetic changes completely. If it doesn't survive, then the posterior training is what teaches the noise directions to be useful, and the tax is the price of the mechanism rather than an artifact beside it. Either answer is informative, and it's the next run.

I'd love to hear from anyone who has tried decoupled inference-time perturbation on recurrent reasoning models, or who has GRAM-style training running at scales where the tax behaves differently, because the tax attribution is the piece my single-GPU setup is least equipped to settle.