Qwen2.5-7B per-token computation: embedding, 28 layers of RMSNorm, Q K V projections, KV cache, grouped-query attention, O projection, residual, RMSNorm, SwiGLU MLP, residual, then final norm and LM head.

Weight matmul (shared by batch)Per-sequence (KV, attention)
Embedding lookup
[152064 × 3584], only B rows read
× 28 layers
RMSNorm
Q proj
3584 → 28 heads × 128
K proj
3584 → 4 heads × 128
V proj
3584 → 4 heads × 128
Attention
28 q heads, 7 share each kv head
KV cache (this sequence)
append new k, v · read all past
O proj
concat 28 heads → 3584
+ residual
RMSNorm
Gate
3584 → 18944
Up
3584 → 18944
SiLU(gate) ⊙ up
elementwise, 18944
Down
18944 → 3584
+ residual
Final RMSNorm + LM head
3584 → 152064 logits