Benchmark report · glm-reap153-76-lsq · October 2026
All numbers on this page were measured on one MacBook Pro M5 Max with 128 GB, on the exact weights people download. Two kinds of test: fidelity, which compares every next-token prediction against the full-precision GLM-5.3-Flash, and tasks, which grade answers to coding, math and tool-calling problems.
Graded by running the code against the tests, matching the final answer, or checking the function call. Thinking on, temperature 1.0, top-p 0.95 (GLM's recommended sampling), one sample per problem. Subsets are small, so read the range, not the point.
| Benchmark | Problems | Flash-sauhard | 95% range | DWQ variant |
|---|---|---|---|---|
| HumanEval+ 164-problem set, first 40; extra EvalPlus tests | 40 | 95.0% | 83–99% | 92.5% |
| MBPP+ first 40; EvalPlus tests | 40 | 92.5% | 80–97% | 87.5% |
| MBPP (base tests) same 40, original tests | 40 | 100.0% | 91–100% | 97.5% |
| MATH-500 first 40, exact answer match | 40 | 97.5% | 87–100% | 100.0% |
| BFCL function calling 7 categories × 15 | 105 | 84.8% | 77–90% | 85.7% |
The DWQ variant is the same build after an extra distillation pass. It matched the original slightly better on fidelity but lost 3.75 points on code, so it was not shipped. Average answer length: 2,549 tokens on HumanEval+, 2,502 on MATH-500; no answer hit the thinking budget.
BFCL, 15 cases each, parsed with GLM's native tool format
Weakest: declining when no tool fits, and the "live" user-written cases, where 20% of answers made no call at all.
Read before quoting
These are subsets, not full leaderboard runs: 40 problems per dataset and 105 tool cases. A 95% score on 40 problems is anything from about 83–99%.
We could not run the original GLM-5.3-Flash on the same harness: it does not fit in 128 GB. That is the job of the fidelity tests below.
HumanEval and MATH are old and likely seen in training. They show the build still works, not how it ranks against frontier models.
48 held-out conversations (521,901 tokens) were run through full-precision GLM-5.3-Flash once, keeping its top-64 predictions at every position. Each build is scored against that. Lower KL divergence is closer; top-1 is how often both pick the same next token.
On the tokens the model writes itself. Lower is better.
Splitting by who wrote the text
On positions where the model writes its own answer, GLM 5.3 Flash-sauhard diverges by 0.108 and agrees on the top token 92.4% of the time.
On text a user or a tool wrote, KL is large (2.8 and 6.9) for every build, including near-lossless ones: neither model can predict a user's next word. It still moves with pruning, so we track it as a comprehension signal rather than a quality score.
| Build | GB | Assistant KL | Top-1 | Reasoning KL | Tool-text KL | User-text KL |
|---|---|---|---|---|---|---|
| REAP 153, plain rounding | 76 | 0.121 | 92.4% | 0.069 | 6.98 | 2.85 |
| Mass-scored 153, plain rounding | 76 | 0.123 | 92.3% | 0.086 | 6.73 | 2.78 |
| Mass-scored 162, plain rounding | 80 | 0.118 | 92.6% | 0.081 | 6.69 | 2.66 |
| Mass-scored 162, LSQ | 80 | 0.112 | 92.8% | 0.079 | 6.72 | 2.64 |
| REAP 153, LSQ (shipped) | 76 | 0.108 | 92.4% | 0.066 | 6.92 | 2.81 |
| Shipped + DWQ (KL loss) | 76 | 0.106 | 92.6% | 0.068 | 7.33 | 3.26 |
| Shipped + DWQ (MSE loss) | 76 | 0.107 | 92.9% | 0.060 | 7.09 | 3.07 |
LSQ rounding (refitting each quantization scale against calibration data) cut assistant KL by 11% at the same size. The two DWQ rows are slightly lower on assistant KL but higher on tool and user text, and worse on code.
All positions. Medians show the typical token; means are pulled up by user and tool text.
| Workload | Samples | Tokens | Median KL | Mean KL | Top-1 |
|---|---|---|---|---|---|
| Code | 12 | 27,950 | 0.400 | 3.227 | 60.1% |
| Agentic, multi-turn tools | 12 | 80,557 | 1.280 | 4.824 | 45.3% |
| Reasoning / math | 12 | 53,119 | 0.003 | 0.066 | 93.1% |
| Long documents, 16K–64K | 12 | 360,275 | 0.593 | 2.785 | 55.0% |
Measured on the MLX server people run (one request at a time, 2048-token prefill steps), and on the bare model at fixed prompt lengths. Speculative decoding is off: GLM's draft head ships in the weights but has not been verified yet.
Real requests to the OpenAI-compatible API
| Prompt tokens | Generated | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| 108 | 478 | 108 | 41.5 peak |
| 75 | 1200 | 52 | 24.4 |
| 880 | 1200 | 240 | 28.2 |
| 6,795 | 1200 | 449 | 38.4 |
| 27,130 | 1200 | 518 | 26.8 |
256 tokens decoded after each prompt
| Prompt | Prefill tok/s | First token | Decode tok/s | Peak |
|---|---|---|---|---|
| 1,024 | 626 | 1.6 s | 31.6 | 72.1 GB |
| 8,192 | 517 | 15.8 s | 29.3 | 74.3 GB |
| 32,768 | 413 | 79.2 s | 25.8 | 77.5 GB |
The server is capped at 84 GB; the rest stays with the system and your apps
3.476 bits per weight on average. Resident size while serving: 71.6 GB.
Public leaderboard scores. The original GLM-5.3-Flash scores 42 on the Artificial Analysis index at medium effort, one point above Claude Sonnet 5.5. GLM 5.3 Flash-sauhard has not been run on that suite; from its fidelity we expect 40–41, but that is an estimate.
Long context at 64K to 128K. Fidelity was measured on documents up to 64K tokens. Retrieval at 128K has not been tested.
Speculative decoding. The draft head is in the weights; until it is verified, all speeds above are plain decoding.
Software: MLX 0.32.4, mlx-lm 0.32.2, mlx-vlm 0.7.9, Python 3.12.15. Raw summaries: bench.json. Harness code: evals/.