LLM architecture from scratch

Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.

Implemented

Model	Model Size	Modality	Normalization	Positional Encoding	Attention	Mixture of Experts
GPT-2 XL	1.5B	Text	LayerNorm	Learned Absolute	Multi-Head Self-Attention (causal)	❌
Llama3-8B	8B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Llama 3.2-3B	3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
OlMo 2 7B	7B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Phi-4	14B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
DeepSeek v3	671B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Gemma 3	27B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
xLSTM	7B	Text	RMSNorm	RoPE	mLSTM	❌
Mistral 3.1 Small	24B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
Llama 4 Maverick	400B	Text	RMSNorm	RoPE	GQA	✅
Qwen3	32B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Qwen3 (235B-A22B)	235B A22B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
SmolLM3	3B	Text	RMSNorm	RoPE + NoPE	Grouped-Query Attention (causal)	❌