Astra edit reduces Qwen’s loops, lifting MATH-500 to 48.1%

MIT CSAIL researchers reported on November 2 that GPT-6 Astra directly edited a small Qwen model’s numerical weights, raising its MATH-500 score from 7.9% to 48.1% under the same sampling settings. The target was Qwen3.5-0.8B. The experiment prohibited gradient descent and training as ways to produce the edits.

Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab attribute the improvement to fixing repetitive reasoning that prevented the original model from finishing answers. They released the edited model and weight differences, allowing developers to inspect what changed against the public base model.

Astra received a Python workspace over the weights and practice problems separated from the final tests. Its opening prompt explicitly allowed fixing repetition that prevented completion. Its analysis workspace used CPUs and had no internet access. Three separate runs allowed 10, 25 or 50 forward passes: opportunities to run the target model and inspect its behavior. Each pass could process multiple problems or texts, so the budget was not a count of individual questions.

The 50-pass run involved 158 turns and 80 Python executions over 20.3 hours of elapsed time, including 4.7 hours waiting when the model could not be reached. Each budget produced its own final edit.

MATH-500 reached 12.3% with 10 passes and 14.1% with 25. In the 50-pass run, GSM8K rose from 13.8% to 67.1%, while instruction-following benchmark IFEval rose from 21.0% to 62.3%. The share of MATH-500 answers cut off at the token limit fell from 93.0% to 33.1%. The authors link the gains beyond mathematics to the same reduction in looping.

A repetition fix under specific test conditions

Evaluation enabled thinking and allowed up to 32,768 tokens per answer. The original and edited checkpoints used the same sampling and repetition-control settings. MATH-500 involved four responses to each of 500 problems, with accuracy averaged over the responses rather than selecting the best answer per problem.

A follow-up on Gemma 3 270M, which rarely looped, produced much smaller changes: 6.8% before editing and 7.0–7.7% afterward. That model used its own evaluation conditions, with thinking off and an 8,192-token limit. The researchers distinguish removing an obstacle to completing existing reasoning from adding a new mathematical ability.

The released checkpoint’s model card also reports a separate evaluation by Hugging Face’s Ahsen Khaliq. Across the full MATH-500 test, with 2,000 generated responses, it scored 51.0%; 31.6% of responses reached the length limit. The accompanying base-model smoke test covered only 25 problems, so this was not a new full-test comparison of both checkpoints.

The edit subtracted a looping direction in two layers. It targeted 131,072 entries, approximately 0.016% of the model’s weights. After rounding into the evaluation’s numerical format, the logs say 130,188 entries actually differ from the base. The distributed checkpoint includes the full multimodal model, whereas the original experiment used a text-only export; its language-model tensors were checked by hash.

The reconstruction script combines the base weights with the published differences and checks each rebuilt tensor’s SHA-256 against the evaluated checkpoint. A mismatch stops the script. The published code also checks the base tensors first, rejecting a starting artifact that differs from the expected original.

The repository supplies prompts, responses, executed code and outputs, and evaluation records. Its authors specify that the release covers the reported runs, excluding trials with earlier prompt versions and some intermediate artifacts.

The findings were released as a research blog post and accompanying experimental records. The team says it is building a benchmark and harness to compare models’ ability to read and edit weights systematically.