WeighingMachine 33,472 base weights · 240 shared experts
Follow a question through the actual weights, neuron activations, and next-byte decision.
Runs locally in this page. · Flow follows the computed activation trace; glowing hotspots mark edited weights for routed checkpoints.
Expert training. Each question is routed to a small patch on top of the shared Transformer. A patch changes selected MLP output weights; the shared base and router stay fixed while the patch is fitted to reproduce its assigned answers. At inference, the router picks one patch, which is added to a fresh copy of the base weights for that answer.
Collapsing experts. We compared patches in their 32-dimension int8 representation, tried nearby pairs first, and jointly refit each pair on both experts’ questions. A merge counted only if both groups kept their exact answers after reload. Successful pairs share one stored patch, while the router’s 300 outputs map to the resulting 240 patches.
Weight culling. For each eligible single-question patch, we simulated its assigned question and measured each changed weight’s contribution along the answer’s activation path. We tried removing low-impact contributions, kept a removal only when the full answer still matched exactly, and repeated until a round found no further safe removal. The shared patches were then jointly fine-tuned to emit NUL (byte 0) as the terminator. The int8 base and patches, plus a packed 1-bit router, were reloaded and checked on all 300 training answers.
The 300 presets are training examples from Three Men in a Boat, not a held-out benchmark. Custom questions always select an expert and may produce incorrect answers. Inference uses the stored model weights, not reference-answer lookup.
Scalability. The routed patches are independent units that can be placed across GPUs, so requests selecting different experts can run in parallel. This demo has 240 shared patches for 300 routes; each request selects one patch and still needs the shared base. Scaling to many GPUs would require efficient base-weight placement and routing, and the current one-patch-per-question design does not split prefill from token generation or distribute one request across all experts.