LAL — Logic-Assembly Language

A minimal CPU LLM inference engine in ~2000 lines of C. No dependencies, no abstractions, no GPU. Just hand-written SIMD kernels.

84% of llama.cpp's speed, 2% of the code.

---

30-second quickstart (Qwen2.5-7B-Instruct)

git clone https://github.com/samaidev/lal.git && cd lal
make qwen7b-server                    # build (auto-detects AVX2/AVX512)
./prebuilt/qwen7b_server --weights prebuilt/qwen7b_q4k_weights.bin \
  --prompt "Hello" --n 40 --threads 2  # run

Pre-built weights (7.5 GB) are not in the repo. Convert from HuggingFace:

python3 scripts/convert_qwen7b_q4k.py  # reads /root/qwen7b, outputs prebuilt/qwen7b_q4k_weights.bin

---

What LAL can do

• ✅ Run Qwen2.5-7B-Instruct at 1.4 tok/s on 2-vCPU Xeon (no GPU)
• ✅ 5 quantization formats: Q8, Q4_0, Q8_0, Q4_0A, Q4_K (llama.cpp-compatible)
• ✅ Hand-tuned AVX2 kernels with 256-bit maddubs, prefetch, 8-row parallelism
• ✅ Full BPE tokenizer, chat template, top-k sampling, repetition penalty
• ✅ Zero external dependencies (only libc + libm + libgomp)
• ✅ ~200KB binary, mmap-based memory, <100ms cold start

What LAL cannot do (be honest)

• ❌ Beat llama.cpp in speed (we're 84%, not 100%)
• ❌ Support GPU/Metal/NPU (CPU only, by design)
• ❌ Support 100+ models (only Qwen2.5-7B + GPT-2 out of the box)
• ❌ Continuous batching / concurrent requests (single-stream inference)
• ❌ KV cache quantization (448MB KV cache is F32)
• ❌ Importance matrix (imatrix) quantization (simple min-max only)

LAL is not a llama.cpp replacement. It's a minimal, readable, hackable reference.

---

Code structure (5 core files, each one line of explanation)

runtime/
  lal_q4k_kernel.h      — Q4_K matmul kernel (256-bit maddubs, 8-row parallel, prefetch)
  lal_q8_kernel.h       — Q8 matmul + LM head (sign-trick int8 dot product)
  lal_tokenizer.h       — BPE tokenizer (byte-level, GPT-2 style)
  lal_runtime.c         — weight loading (mmap), tensor lookup, utilities
  lal_weight_utils.h    — per-row quantization helpers

tools/server/
  qwen7b_server.c       — the server: forward pass, attention, RoPE, sampling
                         (this is where you add new models)

scripts/
  convert_qwen7b_q4k.py — BF16 safetensors → LAL Q4_K format
  q4k_unit_test.c       — correctness test for Q4_K kernel
  bench_q4k.c           — micro-benchmark for Q4_K kernel

That's it. No ggml graph, no backend abstraction, no plugin system. If you want to understand how LLM inference works, read qwen7b_server.c top to bottom — it's 900 lines and covers everything.

---

Adding a new model (template)

1. Copy qwen7b_server.c → yourmodel_server.c
2. Change these constants at the top:

   #define N_LAYER  28    // your model's layer count
   #define N_EMBD   3584  // hidden dim
   #define N_HEAD   28    // attention heads
   #define N_KV_HEAD 4    // GQA kv heads
   #define HEAD_DIM  128  // 3584/28
   #define MLP_DIM   18944 // intermediate size
   #define VOCAB_SIZE 152064

3. Adjust the chat template in encode_prompt() (each model has different special tokens)
4. Write a converter: scripts/convert_yourmodel_q4k.py (copy from convert_qwen7b_q4k.py)
5. Add a Makefile target:

   yourmodel-server: prebuilt/yourmodel_server
   prebuilt/yourmodel_server: tools/server/yourmodel_server.c runtime/*.h
       $(CC) $(CFLAGS) -fopenmp -o $@ tools/server/yourmodel_server.c runtime/lal_runtime.c -lm -lgomp

See docs/ADDING_MODELS.md for a detailed walkthrough.

---

Porting to new hardware (guide)

LAL is designed to be portable. The only platform-specific code is in the kernel headers.

Step 1: Identify your SIMD

// In each kernel header, check for platform:
#if defined(__AVX2__) && defined(__F16C__)
    // x86 with AVX2
#elif defined(__ARM_NEON)
    // ARM with NEON
#elif defined(__riscv_v)
    // RISC-V vector extension
#else
    // scalar fallback (always works, ~10x slower)
#endif

Step 2: Implement 3 core operations

You only need to implement these for your platform:

1. lal_matmul_q4_k in lal_q4k_kernel.h — the Q4_K matmul kernel
2. lal_lm_head_int8_range in lal_q8_kernel.h — the LM head dot product
3. unpack_scales_6bit — 12-byte packed scales → 16 uint8

Everything else (attention, RoPE, sampling, tokenizer) is platform-independent C.

Step 3: Test

gcc -O3 -march=yourarch -I. -o test scripts/q4k_unit_test.c -lm
./test  # should print PASS

Porting effort estimate

Platform	Effort	Notes
x86 AVX2	✅ Done	Current implementation
x86 AVX512	✅ Done	Kernel exists, slower due to downclocking
ARM NEON	~1 day	Similar to AVX2, use vmlal_s8
RISC-V RVV	~3 days	Vector extension, no maddubs equivalent
Custom NPU	1-2 weeks	Depends on instruction set

---

Documentation

• ARCHITECTURE.md — Why the code is written this way (optimization journey, design decisions)
• PITFALLS.md — Bugs we hit and how to avoid them (uint64 overflow, packing formats, AVX512 downclocking...)
• MINIMAL.md — How to add a new quantization format by changing 1 file
• HARDWARE_TEST_REPORT.md — Benchmark results vs llama.cpp
• ADDING_MODELS.md — Step-by-step guide for new models

---

Project status

⚠️ Experimental. Author does not guarantee continued maintenance.

This is a research/educational project. It works, it's tested, but it's not production-hardened.

• Forks welcome. If you want to maintain a fork, go ahead.
• PRs welcome. They'll be merged if they pass tests and don't break existing formats.
• Issues. Bug reports prioritized over feature requests.
• No co-maintainers actively sought, but if you've contributed meaningfully and want write access, ask.

If this project dies, that's okay. The code is here, the docs explain why, the git history tells the story. Future developers can fork it, learn from it, or ignore it. That's fine.

---

License

MIT. Do whatever you want with it. No warranty. If you build something useful, a mention is appreciated but not required.

Acknowledgments

• llama.cpp — for proving CPU LLM inference is viable, and for the Q4_K format design
• mistral.rs — for the Q8 sign-trick SIMD pattern
• Qwen Team — for the excellent Qwen2.5 model