Qwen3.8-Flash-Next at IQ3_XXS on DeepSWE v1.1

One agent, one attempt per task, the model quantized to about 3.43 bits per weight and served on local GPUs. Run 2026-10-07 13:52:45 UTC to 2026-10-12 07:20:26 UTC.

The quantized model resolved 71 of 113 tasks (62.8%). Scored the way Datacurve scores its public board, that is 63.4% (71 of 112 graded attempts), with a 95% interval of 54.2–71.7%.

Labels in square brackets say where a figure comes from: counted from our run ledger and task evidence, computed from those counts or from Datacurve's public trial data, recorded from configs, logs and file records, and arithmetic from recorded figures. Nothing on this page is estimated.

Three ways to count, and the one that matches the board

The first three rows confirm the counts noted at the run's finish (71/113 (62.8%), 69/110 (62.7%), 69/113 (61.1%)). Datacurve keeps every trial that ran in its score, including those stopped at the agent time limit (116 such trials), and drops trials that hit an infrastructure error (155 dropped). The matching rule for our run is the last row: our 2 timeouts count as failures, the 4 attempts lost to harness failures are dropped, and the two retries that ran cleanly count. Restricted to first attempts, the same rule gives the second row. The rows differ by less than the width of any one interval.

Pass@1

The share of single attempts that resolve their task. Datacurve ran 4.0 scored attempts per task for each configuration on average and pools them; we ran one.

Resolved

Every hidden test passes on the agent's committed change, run in a fresh verifier container.

f2p and p2p

Fail-to-pass tests fail before the change and must pass after it; pass-to-pass tests already pass and must keep passing.

Quantization, bits per weight

Storing each weight in fewer bits than a standard 16-bit number. Fewer bits means smaller files and less memory traffic, at some cost in accuracy.

IQ3_XXS, IQ2_XS, IQ4_NL, Q2_0 …

GGUF weight formats, as used by llama.cpp. The number is roughly the bits per weight; "IQ" formats store groups of weights as entries of a small codebook, chosen with the help of an importance matrix. The file is labelled IQ3_XXS, but it mixes several formats (table below).

MTP draft

A small multi-token-prediction head that proposes the next few tokens; the full model checks every proposed token in one pass. It is there for speed.

Compaction

When the conversation nears the context window, the agent summarizes earlier turns to make room.

The setup

The model and the quant

Qwen3.8-Flash-Next GSQ-RCO, a mixture-of-experts model: 48 layers, 512 routed experts per layer with 10 used per token, a 262,144-token native context [recorded, file metadata]. The quantization is GSQ-RCO by ISTA DASLab, with an importance matrix from 1,000 calibration chunks [recorded].

Sizes are recorded file sizes; weight counts and bits per weight are arithmetic over the recorded tensor list (shape and bytes of every tensor). File hashes are not recorded.

The MTP draft. The engine loads its own multi-token-prediction draft layer (835 MiB of VRAM [recorded, engine log]); its format is not recorded.

Hardware and engine

The model was served from the GGUF files above plus the engine's own repacked copy of the experts. The agent and the task containers ran on a separate host, and the model server answered over the local network.

The harness

• Agent: the Pi coding agent (version 0.99.1), one attempt per task, a 204,800-token window.
• Limits, from each task's own definition: 10,800 s for the agent and 1,800 s for the verifier, no network for the agent, and grading in a separate fresh container [recorded, all 113 tasks].
• Tasks: DeepSWE v1.1, task images tagged v1.1, from Datacurve's task repository at revision 0b9fabbb63b9.
• Retries: an attempt that hit an infrastructure error was run once more; nothing was retried for score.
• Sampling: temperature 1.0, top-p 0.95, top-k 20 [recorded].
• Dates: 2026-10-07 13:52:45 UTC to 2026-10-12 07:20:26 UTC, paused between 2026-10-07 15:16:55 UTC and 2026-10-09 00:02:16 UTC.

Against the public board

Datacurve publishes every trial behind its leaderboard: 31,462 scored trials over 70 configurations of 28 models, all run with mini-swe-agent [public data, fetched 2026-10-09]. We scored each configuration as passed trials over scored trials, and took each model's best configuration. Method check: against the leaderboard page as read on 2026-10-13 (marked updated September 22, 2026), this reproduces the published Pass@1 of all 21 models the page lists, and for every one of them also the page's choice of best effort level, its average output tokens and its average steps. Average cost matches for 15; the page prices the other 6 differently from the cost field in the trial data. The one cost on this page, the public median in the effort table, comes from that field.

At 63.4%, 29 public configurations score higher, 1 the same and 40 lower; by each model's best configuration, 12 of 28 models score higher. Our 95% interval (54.2–71.7%) contains 31 of the 70 public scores, so the ranking within that band is not settled by one attempt per task. The board's own Qwen entry, qwen3.8-max at xhigh effort, computes to 57.5% under its harness.

Difficulty

For each task, Datacurve's data gives every model's pass rate; the task's average public pass rate is the mean over the 28 models [computed; it matches the published per-task difficulty file to within 0.0005]. The tasks we resolved average 62.1% on that scale; the ones we missed average 36.7%.

Hard tasks we solved

In the hardest fifth (Q1, average public pass rate 3–31%), our run resolved 4 of 23. No task under 26% was resolved.

Easy tasks we missed

In the easiest fifth (Q5, average public pass rate 73–92%), our run missed 2 of 22.

By language

Effort per task

Over our 112 graded attempts [counted from each task's summary]: 77.6 agent-hours in all. The public figures are medians over 31,462 trials run with mini-swe-agent on other hardware; they show the scale of a typical public attempt, not a like-for-like comparison.

Tool calls per attempt had a median of 148. 67 of our 112 attempts compacted at least once (at most 4 times); wall times ran from 14.7 to 180.0 minutes. Our output tokens are as the agent recorded them per turn; the public counts come from each provider and may count reasoning differently. Wall time depends on the serving hardware as much as on the model, so no speed comparison is made here.

Caveats and considerations

• One attempt per task. 112 graded attempts give a 95% interval of 54.2–71.7% (Wilson score interval) around 63.4%. Public configurations pool 4.0 attempts per task on average, so their scores are tighter.
• A different harness. We used the Pi coding agent with a 204,800-token window and compaction; Datacurve ran every public configuration with mini-swe-agent, whose context and compaction settings are not in the public data. Our attempts reached a median largest context of 171,923 tokens against 126,159 for the public trials. A harness can move a score on its own, so read the comparison as indicative.
• A quantized model, not the vendor's weights. 3.43 bits per weight on average, and the engine also stores its attention cache in int8. The engine computes cached experts on the GPU and the rest on the CPU, which rounds differently, so its outputs can differ slightly from a pure-GPU run.
• The vendor's 58.7% is Alibaba's reported figure for Flash-Next, repeated on model-listing sites. It is not an independent run, and its harness, attempts and benchmark version are not recorded here.
• Retries. Infrastructure errors were retried once. 3 tasks were retried: numba-stencil-boundary-modes (infrastructure error, infrastructure error); opa-template-string-reconstruction (infrastructure error, resolved); superjson-error-stack-serialization (infrastructure error, resolved).
• Task images v1.1. Our task images and Datacurve's public data are both v1.1; this page compares only against v1.1, not the earlier v1 board.
• Contamination is unknown either way. Datacurve says its tasks are written from scratch rather than adapted from existing commits; the model's training data is not public, so we cannot check for overlap.

Harness problems, and the tasks each touched

Reproduction

• Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf and Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf.
• Engine: Strata 0.1.40 at 82f46a8, run with --kv int8 --spec 4 --max-context 204800 --kv-resident 32768 --pcie-frac 0.20 --spec-min-p 0.70.
• Harness: revision ae9bc1e+dirty with Pi 0.99.1; DeepSWE task repository 0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea.
• Run ledger: 234 rows, SHA-256 647431d28916e4d287e659f801185cad5e72e9ec5d0d6bf4be42ece72c688157.
• Public data (Datacurve v1.1, fetched 2026-10-09): trials.json 310cb428fe8914cff21b8edcb0b17256884efca55666bd6d6646d223c77d2ce6, tasks.json bae967f6472943564c3fc5232fba3c8e0ac465c1be5ccf9dd4895d4ee9df6242, difficulty.json 00db63423282f56d9cbd321fedc55204e61cad4817b301d1ba6a4f9ec88c3a32. Source: deepswe.datacurve.ai.
• Leaderboard page used for the method check (read 2026-10-13): SHA-256 14436c31be1e50a0b62171e4aee4dd0ae0ce66b1e390af89c7e6e095ad59f1f1.
• Generator: python3 build_page.py writes data.json (every figure on this page) and this page; the same inputs give the same bytes.

Not recorded

• SHA-256 of the model files and of the MTP draft file (their sizes are recorded; their hashes are not).
• The format and source file of Strata's own MTP draft layer (only its 835 MiB of VRAM is logged).
• Whether Strata's repacked expert pack holds exactly the GGUF's expert bytes.
• The server's system RAM size and the RTX 4080's total VRAM (the log names the card, not its capacity).
• The server's decode speed during the run (it keeps no per-request log the harness reads).
• The harness, attempts, settings, date and benchmark version behind the vendor's 58.7%.
• The vendor's release format for the full-precision weights.
• mini-swe-agent's context limit and compaction settings for each public configuration.
• Whether Datacurve re-ran trials that hit infrastructure errors, or only dropped them.
• Why two runs ended with the agent killed (exit 137) mid-command.
• Energy or dollar cost of our run.
• Whether any DeepSWE task appears in the model's training data.

DeepSWE is Datacurve's benchmark; the public figures are computed from its published trial data. Our run used its public v1.1 tasks unchanged.