Future oracles

How accurately can you predict future tokens given an activation of an LLM? It seems interesting to know how far in advance an LLM is planning output tokens. There’s some work from 2023, Future Lens, that tries to answer with a few methods, most interestingly by training a soft prompt (i.e. learning soft tokens to put at the beginning to get the model to predict future tokens given an injected activation). But now we live in the glorious meta-modeling future: can I train something like an Activation Oracle to read out future tokens given a past activation? It turns out I can; I call this meta-model a future oracle.

I think this is kinda a weird thing to do. Given an activation, you can probably extract the context pretty well (since the previous tokens are very useful for predicting the next one), and if you have the context you can predict future tokens by just running the model itself. So a perfect reader should be able to do near perfect on the task of guessing future tokens given an activation. But here we’re not using a perfect reader, we’re using a LoRA fine-tune of the model itself.

I think with future oracles we should maybe get something like a way to kinda understand the extent LLMs are planning ahead, and seeing what they’re planning ahead. But I’m pretty unsure about this!

I did this research fairly quickly over a few days (mostly with Claude), and I probably made mistakes, please take everything I say with a grain of salt. I mostly did this research as a part of looking into making NLAs with explanations that are more causally linked to the activation; I don’t think future oracles are super useful for interpretability by themselves. Future oracles seem to be better than the future lens (given matched training compute), in large part because there are more learnable parameters, but I didn’t compare them thoroughly enough to be confident about that.

Training it

It was pretty straightforward to train a future oracle:

1. I took activations from the layer 67% of the way through Qwen3 8B/1.7B/0.6B on random FineWeb tokens.
2. I generated a continuation of 9 tokens given that FineWeb context (with greedy sampling).
3. I distilled a LoRA for ~2.5 epochs of 205k positions from 5.5k FineWeb documents where the activation was injected as a token (i.e. I replaced the output of the embed matrix with the activation¹) and the model wrote just the continuation (i.e. I trained the model to match the logits of the continuation given the activation)²

I used EasyNLA for most of the training code; you can see my (entirely Claude-written) code here if you want specifics.

Using it

You can try my online demo.

For example, here the future oracle reads out the other multiplicand correctly, but not the answer.

Here’s the fraction of positions where the future oracle is right³, across continuations⁴ from random points in held-out⁵ FineWeb documents (you can get ~5% by always guessing " the"):

Fin

I did everything on base models; it would be fun to do this for chat-tuned models too. Probably the most interesting would be to scale it up to larger models and longer continuations. It also might be interesting to try training it on other layers; I just chose 2/3 of the way through because that’s common in meta-models but it would probably be interesting to either train the future oracle to accept any layer as input, or train one for each layer.

Thanks to Alibaba for open-sourcing the Qwen3 series of models!