Four researchers at MIT CSAIL published Sparks of Weight-Space Programming on November 2, 2026. They report that a general-purpose LLM can read and write neural network weights directly (no optimizer, and in particular no gradient descent) around a natural-language specification, sometimes with no environment feedback. They call this Weight-Space Programming. Their headline run asks GPT-6 Astra to edit Alibaba's open-weight Qwen3.5-0.8B for math without training.
The authors are Rikiya Takehi* and Jayden Teoh* (equal first authors), Alex Zhang, and Omar Khattab. They describe the results as very early, not peer-reviewed, and release full experiment logs.
Key Facts
| Setting | MATH-500 | GSM8K | IFEval | Hit token limit |
|---|---|---|---|---|
| Qwen3.5-0.8B base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 forward passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 forward passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 forward passes | 48.1% | 67.1% | 62.3% | 33.1% |
- Editor model. GPT-6 Astra, used as shipped; the authors say they did not fine-tune Astra for these tasks.
- Target. Qwen3.5-0.8B with thinking on (24 layers). Astra gets a Python kernel over the weights (no GPU), practice items from the GSM8K and MATH training sets (7,469 + 7,275; not the test sets), and a budget of 10, 25, or 50 forward passes. Evaluation benchmarks are held out from Astra.
- Sampling. Temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5. Qwen's own model card notes the model is prone to thinking loops.
- Edit size. Astra attributed looping to one direction written by the MLPs of layers 15 and 19, then subtracted that direction from 64 output columns in each layer (131,072 weights, about 0.016% of ~0.8B).
- Author caveat (required reading). Fixing loops is not the same as injecting new math skill: it lets the model finish answers it could already work out. On Gemma 3 270M, which rarely loops (8.5% hit the token limit vs 93% for Qwen), MATH-500 moved only from 6.8% to about 7.0–7.7% across budgets; Astra itself reported those gains as not established. The authors' conclusion: weight edits can remove an obstacle that hid an ability; they could not add an ability the model lacked.
- Other experiments (brief). Astra named the game for small PPO policies (17,253 weights) from weights alone; wrote MinAtar policies from scratch that beat PPO trained for 100 million steps on Breakout and peers; with a forward-pass budget, wrote a ~1.48M-weight 9×9 Go net that beat equal-sized PPO; failed to spot a sycophantic Gemma from raw weights alone, then succeeded when given an attention tool.
Background
Weight-Space Programming, as defined in the post, sits next to prompt optimization, harness search, and training-code search: the LLM reasons about what weights compute, then changes them, instead of handing the job to Adam-style optimizers. Related lines (ROME/MEMIT-style editing, refusal-direction edits, hypernetworks, tiny hand-set transformers) are acknowledged; what the authors claim is new is an off-the-shelf frontier model reading and writing weights from tens of thousands of parameters as text up to a 0.8B language model with tools.
The edited Qwen checkpoint from the math experiment is on Hugging Face. The authors thank Hugging Face for rebuilding it from the logs, re-running evaluation, and hosting a demo. They say they are building a benchmark and harness for the task.
Reactions and Impact
Turing Post reported that within hours of the November 2 post, discussion of the announcement had passed a million views on X. That figure is Turing Post's reporting, not an independent count by AI Rank.
Turing Post also notes Lucas Beyer's observation that the prompt told Astra that if the model "repeats itself and never finishes, finding and fixing what drives that repetition counts." Coauthor Jayden Teoh said the short assignment was handed to Claude Code, which added that hint while expanding the prompt. The authors also tried a shorter prompt without the repetition example; at the 25-batch budget, MATH-500 still reached 14.1%, matching the hinted 25-batch row. Turing Post says the table has no 50-batch row for that shorter prompt.
What We Don't Know Yet
- How far the Qwen loop fix generalizes beyond the reported sampling settings and held-out suites
- Whether a 50-forward-pass run with the shorter, no-example prompt would match the 48.1% MATH-500 result (not published in the sources checked)
- Peer-reviewed confirmation; the authors frame the blog as very early results
- Whether Weight-Space Programming can add capabilities that are absent, not merely unblock ones that are already present (Gemma counter-example argues against that so far)