kindlingai/glm-5.3-exlr8-k2-k3-k4

GLM-5.3 · exlr8 (K2/K3/K4)

Version 0.5 (git tag v0.5). See the changelog. Weight set ID: 2011c0cedde962d19fc43e74c1d409279e398d40d27a6fdb4bcb7b49ea2ea6fb (see Identifying the weights).

Also mirrored on ModelScope: kindlingai/glm-5.3-exlr8-k2-k3-k4.

These weights cannot be used without E4M3 rounding. A loader must round every value the trellis codebook produces to FP8 E4M3 (round to nearest even, saturating) before applying the side vectors. The encoder searched only over E4M3-rounded values, so the rounding is part of the format: a stock EXL3 decoder, or any decoder that skips the rounding, reads measurably worse weights from the same bytes.

A quantization of GLM-5.3 (744B MoE: 256 routed experts × 75 MoE layers plus the MTP layer 78, dense layers 0-2) in the exlr8 v2 format, built for NVIDIA GB10-class unified-memory devices served at tensor-parallel sizes 3-7 (TP=4, 5 and 6 are the main targets; see "Known results so far" for what has been tested).

Engine support

KindlingAI:

• kindlingai/gr8: the exlr8-native engine for GLM-5.3 on 3-6 DGX Sparks (GB10). It fetches each box's part of this repository and serves it at TP=3 and 4.
• kindlingai/exlr8: the format and a pure-PyTorch reference loader, for checking a loader or kernel against known-good weights.

Other engines:

• ashhart/TensorFold: a fast cross-platform inference engine.
• GordoAR/MLXLR8: an EXLR8 engine for Metal by Andrew Gordon.

How does this differ from other quantization formats?

EXLR8 is a new quantization format. It reuses EXL3's trellis encoding — QTIP-style codes where each 16×16 tile is a path through a 16-bit-state trellis, every weight contributing K new bits, found by Viterbi search over the mcg codebook — but re-engineers everything around it for MoE models served on GB10-class hardware (128 GB unified LPDDR5X, about 220 GB/s streaming), where decode speed is bound by how many bytes of expert weights you read per step. A K-bit tile is 256×K bits, packed exactly as exllamav3's pack_trellis packs it (16K little-endian int16 in tensor-core order) and decoded by exllamav3's reconstruct plus the rounding below.

On top of the stock codec, EXLR8 changes four things:

1. FP8-rounded codebook. Codebook values are rounded to E4M3 inside the Viterbi search, so decoded weights are exactly FP8 and prefill runs on FP8 tensor cores.

2. Deployment-oriented storage. EXL3 quantizes a whole tensor at one width. EXLR8 stores every routed expert at all three widths at once, and the width is chosen per expert-half at load time from a manifest — no re-encode to change the mix. Storage granularity is a 128-channel half (paired into 256-channel fragments).

3. One activation rotation per fan-out point. Each weight is decoded as w = R(diag(suh)·L(T))·diag(svh) — sign, scale, and a fixed 128-block Hadamard on each side (the Hadamard is inherited from exllamav3). The difference from per-tensor EXL3 is what's shared: on the hidden side, one sign vector per layer (signs.in/signs.out) and block-constant scales are used by every expert and every width, so the kernel rotates the token's activation once and shares a single A operand across all active experts instead of once per expert. Per-expert signs live on the intermediate side, folded into the stored weights.

4. MoE/TP-native layout. The sharding rule (16e + h + L) mod N covers any TP=3–7 rank count with zero re-layout; kernels index experts through pointer tables rather than contiguous ranges; each weight exists once (removing the duplicate dense and decode copies); and 64 KiB-aligned safetensors make remote halves HTTP range-fetchable (about 75 GB per node at TP=4 under the home manifest, instead of the whole checkpoint).

Why three widths?

Decode above concurrency 1 is bound by the bytes of expert weights read per step. Every routed expert is therefore stored at three trellis widths — K2, K3 and K4 (the shared expert at K3 and K4; layer 78's routed and shared experts at K3 only) — and the serving width is chosen at load time from a manifest, not at encode time. Codes do not nest between widths: each width is an independent, Hessian-aware trellis encode with an E4M3 codebook, so prefill runs on FP8 tensor cores with no quality cost.

What is in this repository

content	notes
experts/exlr8-layer-LLL.kK.safetensors	routed-expert trellis halves for layer LLL at width K, one file set per width
experts/exlr8-layer-LLL.index.safetensors	index, side data, checksums and per-width error tables for layer LLL
config.json, generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja	the base model's, unchanged, from zai-org/GLM-5.3-BF16 at 9d2398f
dense/exlr8-dense-000..078, dense/exlr8-vocab	dense weights: trellis K4 with int4 g32 exceptions where it wins on held-out data; q_a/kv_a/indexer MXFP8; kv_b block FP8; lm_head/embedding FP8 per-row
manifests/exlr8-manifest-home.json	about 3.25 bpw effective: per layer, the 64 experts with the worst 3-bit error at K4 and the other 192 at K3 (the split of the previous-generation 3.25 bpw checkpoint); shared K4; about 75 GB of routed experts per rank at TP=4
manifests/exlr8-manifest-down1.json	about 2.25 bpw effective: each expert one step below home; shared K3
manifests/exlr8-manifest-up1.json	4.0 bpw effective: every routed expert at K4; shared K4
manifests/exlr8-manifest-bit{225,243,260,325,338}.json	2.25 to 3.38 bpw: the width profiles below, which spend the bits where they lower KL against BF16 the most
index-slices/tp<N>/k<K>/rank<r>.safetensors	what rank r of TP=N needs at width K, for TP 3-7 and K2-K4 (see Index slices)
hashes/	sha256 and size of each safetensors file, the index slices included

Layer 78 (the MTP layer) stores its routed and shared experts at K3 only, so every manifest uses K3 there.

A loader reads the layer's index file and the manifest, takes one width per unit (gate/up per 128-channel half, down per expert), and fetches only the halves its TP rank owns — so one set of files serves TP=3 to 7 with no re-encode and no per-rank copies. Loader gives the byte layout.

Manifests

The three manifests above ship with the files:

manifest	routed experts
home	294.7 GB
down1	204.1 GB
up1	362.6 GB

These are the routed-expert weights of layers 3-77 that each manifest loads: trellis tiles plus side vectors. The MTP layer 78 (K3 in every manifest) adds 3.6 GB to each. Shared experts and the dense and vocabulary files are not included.

Width selection is deliberately a deployment decision: expert routing is workload-dependent (hot sets of different workloads barely overlap), so deployments with a routing histogram can write their own manifest to trade memory and bytes-per-step against measured per-unit error. Error tables in each index file carry the measured held-out error per (gate/up half, down) at every width in one comparable unit.

Width profiles

A width profile is a manifest: it says which width each routed expert loads at. Every profile reads the same files, so switching profiles is a load-time choice, with no new download beyond the widths it uses.

These KL numbers come from our own full-vocabulary test, which is tougher than the short panels most GLM-5.3 quantizations report: it scores long held-out documents (base64, hashes and IDs included) and wikitext. On the four-window panel that other GLM-5.3 cards use, the same profiles score far lower (bit325: 0.025 mean). The numbers are only useful for comparing quantizations measured on this same test.

profile	routed bits per weight	model size	expert weights per box, TP=4 / TP=3	KL vs BF16, held-out text: mean / median	KL vs BF16, wikitext: mean / median
down1	2.25	219 GB	52 / 70 GB	0.242 / 0.0327	0.313 / 0.0573
bit225	2.25	220 GB	52 / 70 GB	0.229 / 0.0308	0.284 / 0.0516
bit243	2.43	235 GB	56 / 75 GB	0.209 / 0.0266	0.260 / 0.0455
bit260	2.60	251 GB	60 / 80 GB	0.192 / 0.0222	0.231 / 0.0373
home	3.25	310 GB	75 / 100 GB	0.149 / 0.0125	0.104 / 0.0143
bit325	3.25	310 GB	75 / 100 GB	0.147 / 0.0120	0.097 / 0.0132
bit338	3.38	322 GB	78 / 104 GB	0.142 / 0.0114	0.092 / 0.0122
up1	4.00	378 GB	92 / 123 GB	0.127 / 0.0083	0.051 / 0.0070

How to read it:

• Model size is everything a full copy of the profile loads: all routed and shared experts at the profile's widths, plus the 10.7 GB of dense and vocabulary files. Expert weights per box is the busiest box's share of the experts at that tensor-parallel size, before the KV cache.
• KL is the full-vocabulary KL divergence of each next-token distribution from the BF16 model's, averaged over every position of teacher-forced text, on held-out text (64 rows of 2,048 tokens from documents the calibration never saw: 131,072 positions) and on wikitext-2 (65,536 positions). Lower is better. The median shows the typical token, the mean the tail.
• The means look high next to dense models' numbers, and that is the model, not the format. GLM-5.3's top-8 routing turns small numeric differences into different experts: two BF16 runs that differ only in their batching score mean 0.11 (median 0.004) against each other on 32,768 positions of the held-out text. Compare profiles with each other, and compare other quantizations only on the same harness: an EXL3 3.25 bpw quantization with home's per-expert split (davidsyoung/GLM-5.3-EXL3-TR3-3.25bpw) scores 0.161 on the held-out text here.

Which to use:

• The bit profiles spend the extra bits where they lower KL the most per byte. bit225 is down1's size with 5-10% lower KL. bit325 is home's size with lower KL (1.5% on held-out text, 7% on wikitext).
• Speed: at the same size the bit profiles decode slightly slower. In gr8 on four GB10s, bit325 decodes about 2.5% slower than home (three boots each), and bit225 about 2% slower than down1 on prose (two boots each). Prefill is unchanged.
• Memory: with gr8's default KV pool, bit243 leaves about 7 GiB free per box at TP=3 and bit260 about 3 GiB; at TP=4, bit338 leaves about 3.5 GiB. home, bit325, bit338 and up1 do not fit at TP=3.

Two more profiles were scored to fit the allocation and are not shipped: every routed expert at K2 (held-out 0.266, wikitext 0.368) and every routed expert at K3 (0.156 and 0.120).

Identifying the weights

The weight set ID names one exact set of weight files: the sha256 of the lines <path> <size> <sha256>, one per weight file (experts/*.safetensors and dense/*.safetensors, sorted by path, each line ending in a newline). This release's ID is 2011c0cedde962d19fc43e74c1d409279e398d40d27a6fdb4bcb7b49ea2ea6fb. Every manifest and every index slice carries it as set_id, so a copy of the files can be checked against a manifest or slice without reading the weights: recompute the ID from hashes/ (or from the files), and compare. Manifests are not part of the ID, since they choose widths and change more often than the weights.

Index slices

index-slices/tp<N>/k<K>/rank<r>.safetensors holds, for TP size N (3-7), width K (2-4) and rank r, every half that rank owns at that width under the sharding rule (16 * expert + half + layer) mod N, so a rank can fetch and load its halves without the layer index files. Per layer LLL (layer 78: K3 only):

tensor	dtype, shape	content
L<LLL>.units	I16 [M, 2]	(expert, half) of every unit the rank owns; expert 256 is the shared expert
L<LLL>.offset	I64 [M, 3]	byte offset of the unit's gate, up and down half in experts/exlr8-layer-LLL.k<K>.safetensors; each half is half_bytes = 98304 x K bytes
L<LLL>.checksum	U64 [M, 3]	each half's xxhash64, as in the index files (trellis bytes, its 128 per-channel values, then the 48 block scales)
L<LLL>.chan	F16 [M, 3, 128]	gate svh, up svh and down suh of the half's 128 channels
L<LLL>.block	F16 [E, 3, 48]	gate su_block, up su_block and down sv_block per expert (E = 257 with the shared expert)
L<LLL>.signs	I8 [2, 6144]	the layer's signs.in and signs.out

The metadata names the TP size, rank, width, layers, half_bytes, the rule, the set_id the slice was cut from and the sha256 of each source index file. In every file the offsets also follow 65536 + ((p * 256 + e) * 16 + h) * half_bytes (p = 0 gate, 1 up, 2 down), with the shared expert's halves after the routed ones.

Loader

The byte-level format, enough to write a loader. A pure-PyTorch reference loader is in kindlingai/exlr8.

Files are little-endian safetensors. Skip tensors named __pad.*. A matrix is handled as W (in, out), so rows are the input side and the HF tensor *.weight is Wᵀ.

Tile decode. A 16×16 tile at width K is 32K bytes (16K int16 words).

step	rule
bits	Swap each adjacent pair of int16 words, then read the words MSB first as one 256K-bit stream. Value vᵢ (i = 0..255) is bits [iK, iK + K).
state	sᵢ = (vᵢ + 2^K·vᵢ₋₁ + 2^2K·vᵢ₋₂ + …) mod 2^16, indices mod 256 (tail-biting).
codebook	x = ((sᵢ · 0xCBAC1FED) & 0x8FFF8FFF) ^ 0x3B603B60. Add the low and high 16 bits of x as two fp16 values, in fp16, and round the sum to E4M3.
placement	Stream index j = 8t + n (t = 0..31, n = 0..7) is tile element (row, col) = the n-th of (r₀, c₀), (r₀+1, c₀), (r₀+8, c₀), (r₀+9, c₀), (r₀, c₀+8), (r₀+1, c₀+8), (r₀+8, c₀+8), (r₀+9, c₀+8), with r₀ = 2(t mod 4) and c₀ = ⌊t/4⌋.

Reconstruction. W = diag(suh) · Hb · T · Hb · diag(svh), in fp32, with T the decoded tiles (in, out) and Hb block-diagonal with 128×128 blocks H/√128, H the Sylvester Hadamard matrix (H₁ = [1], H₂ₙ = [[H, H], [H, −H]]). In exllamav3 terms: preapply_had_r(preapply_had_l(T, 128) * suh[:, None], 128) * svh[None, :].

Experts. In experts/exlr8-layer-LLL.index.safetensors, kK.index[e, p, h] is the absolute byte offset and the length of half h (0..15) of projection p (0 gate, 1 up, 2 down) of expert e in experts/exlr8-layer-LLL.kK.safetensors. The manifest's layers[L] gives each width: gate[e][h], up[e][h], down[e]. The shared expert uses the tensors prefixed shared. with e = 0, at the widths in shared (gate[h], up[h], down). Each half decodes on its own, with its width's side data:

	gate, up half h = W[:, 128h : 128h+128] (6144 × 128)	down half h = W[128h : 128h+128, :] (128 × 6144)
suh	signs.in ⊙ repeat₁₂₈(kK.p.su_block[e])	kK.down.suh[e, 128h : 128h+128]
svh	kK.p.svh[e, 128h : 128h+128]	signs.out ⊙ repeat₁₂₈(kK.down.sv_block[e])
tiles	tile (k16, c), k16 = 0..383, c = 0..7, at byte (8·k16 + c)·32K: rows 16·k16, cols 16c	tile (b, r, c), b = 0..47, r, c = 0..7, at byte (64b + 8r + c)·32K: rows 16r, cols 128b + 16c

Here p is gate or up, repeat₁₂₈ repeats each of the 48 block scales over 128 channels, and tile rows and columns count from the start of the half.

Dense and vocabulary. dense/exlr8-dense-LLL.safetensors holds layer L's other weights, named without model.layers.L.. dense/exlr8-vocab.safetensors holds embed_tokens, lm_head and norm.weight (HF model.embed_tokens.weight, lm_head.weight, model.norm.weight). Norms and biases are stored as shipped under their names. A matrix M (HF M.weight) has M.record, int64 [8] = (format, out, in, K, side: 0 replicated, 1 output, 2 input, channels per shard, shared input rotation, solve method), and data tensors by format. Trellis decodes W (in, out) as above. The other formats give the HF weight (out, in) directly.

format	tensors	weight
1 trellis	M.trellis u8 [entries, bytes], M.suh f16 [in], M.svh f16 [out]	Side 2: entry g = input rows 128g.., tiles as a down half. Otherwise entry g = output cols 128g.., tiles as a gate/up half. Reconstruct the whole matrix.
2 int4 g32	M.q4 u8 [out, in/2], M.s4 f16 [out, in/32]. Side 2: [in/128, out, 64] and [in/128, out, 4]	(n − 8)·s, input 2j in the low nibble of byte j
3 MXFP8	M.fp8 u8 [out, in], M.e8m0 u8 [out, in/32]	E4M3 · 2^(e8m0 − 127) per 32 inputs
4 FP8 block	M.fp8 u8 [out, in], M.scale f32 [⌈out/128⌉, ⌈in/128⌉]	E4M3 · scale per 128×128 block
5 FP8 row	M.fp8 u8 [out, in], M.scale f32 [out]	E4M3 · scale per row
6, 7	M.bf16 or M.fp32 [out, in]	as stored

E4M3 is OCP FP8 E4M3 (float8_e4m3fn). The codebook rounds to it to nearest even, saturating.

Checksums (optional) are xxhash64 with seed 0:

• Expert half, kK.checksum[e, p, h]: its trellis bytes, then its 128 per-channel fp16 values (the svh or suh slice above), then the 48 fp16 block scales.
• Dense, M.checksum[b], per block b of 128 channels on the sharded side (128 output rows when replicated): for trellis, entry b, then the block's 128 values of the sharded side's vector (svh, or suh on side 2), then the whole other vector. For int4 on side 2, q4[b] then s4[b]. For FP8 block, rows 128b.. of fp8, then scale row b. Otherwise, rows 128b.. of each data tensor in the order listed.

Calibration

Encoded with Hessian-aware (LDLQ) rounding against a fixed calibration set of about 2M tokens (2048-token rows; 977 calibration rows and 64 held-out rows from whole unseen documents). The calibration text, activations and Hessians are not part of this repository. Quality is scored against the BF16 model on the held-out rows.

Loading

kindlingai/gr8 serves this release on GB10 boxes at TP=3 and 4, and fetches each box's part of it from this repository. A pure-PyTorch reference loader, for checking a loader or kernel against known-good weights, is in kindlingai/exlr8. This repository holds everything a loader needs: the quantized weights, the manifests, and the base model's config.json, tokenizer and chat template.

The format's three exllamav3-derived primitives (the mcg trellis codebook, the 128×128 Hadamard matrix, and the tile packing order) are taken from exllamav3 (MIT) unchanged.

Known results so far

• Full-vocabulary KL against BF16 for every shipped profile: see Width profiles.
• In gr8 with its default KV pool, home, bit325 and bit338 serve at TP=4, and down1, bit225, bit243 and bit260 at TP=3: smoke tests pass and an 87K-token needle is found. Speed and memory are in Width profiles.

Intended use and limitations

• Serving GLM-5.3 on GB10-class devices at TP=3-7. TP=5 needs engine support that is still landing.
• The DFlash2 drafter is not distributed with this checkpoint (its upstream licence does not permit redistribution); a deployment that wants it converts it at load time from the original files.
• Quantized weights: measure quality for your workload before relying on the lower-width manifests; down1 trades measurably more error for about 30% fewer bytes per step.
• Not for training. Router, norms and small projections are kept at their shipped precision (BF16/FP32) because routing is sensitive.

Licence

These weights are a quantization of Z.ai's GLM-5.3 and remain subject to the GLM-5.3 License, whose full text is in LICENSE. Its conditions apply to this quantization as a derivative work.

The parts this quantization adds (the exlr8 v2 packing, manifests and error tables) are also offered under CC BY 4.0: redistribution and modification with attribution are permitted. CC BY 4.0 does not remove or relax any condition of the GLM-5.3 License.

The exlr8 v2 format and the loader are separate works; the three exllamav3-derived primitives remain under their MIT licence.

Acknowledgements

exllamav3 made this possible through their original research on the EXL3 format.

Built on Z.ai's GLM-5.3; the encoder and the format's primitives are derived from exllamav3 0.0.43 (MIT).

brandonmusic's EXL3 quantizations of other models were an inspiration for some of this work.

Changelog

Each version is a git tag on this repository. The expert, dense and index files are unchanged since 0.1, so every version has the same weight set ID.

0.5 (2026-10-11)

Changes:

• Adds the bit225, bit243, bit260, bit325 and bit338 width profiles, with their sizes and full-vocabulary KL against BF16 in Width profiles. The weights and the set ID are unchanged.
• The card says where to load the release, now that gr8 is public.

0.4 (2026-10-10)

Changes:

• Adds the base model's config.json, generation_config.json, tokenizer and chat template, unchanged from zai-org/GLM-5.3-BF16 (9d2398f), so the repository is a complete model. The weights, widths and set ID are unchanged.
• Adds the byte-level Loader section and points to the reference loader.

0.3 (2026-10-08)

Changes:

• Adds the weight set ID (set_id) to the card and every manifest. The weights and widths are unchanged.
• Adds index slices for TP 3-7 at K2, K3 and K4, with their hashes.

0.2 (2026-10-08)

Fixes:

• home and up1 asked for K4 on layer 78's shared expert, which is stored at K3 only, so loaders failed on it. Both now use K3 there.
• Removed exlr8-manifest-calib-325.json. Its widths came from calibration routing counts, which the format spec rules out for shipped manifests.
• LICENSE now carries the full GLM-5.3 License text, which the licence requires in derivative works. The weights remain under it, and CC BY 4.0 covers only what this quantization adds.

Changes:

• The card states that loaders must apply E4M3 rounding.
• Manifests gain version and changelog fields, name layer 78's shared expert in mtp_layer, and no longer name an internal mount path in source.
• Card corrections: file paths, the layer 78 widths, an unsourced 181 GiB size, the loader (not public yet) and where to get config.json and the tokenizer.

0.1 (2026-10-08)

Initial release: routed experts of layers 3-77 at K2, K3 and K4 and of the MTP layer 78 at K3; dense and vocabulary files; home, down1, up1 and calib-325 manifests.