SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

row-major matmul optimization

An innovative library for efficient LLM inference via low-bit quantization

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.

258 KB WASM runtime for Needle a 26M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.

🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-oriented durability with HNSW + DiskANN indexing for best-in-class performance.

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.

rust library to write integer types of any bit length into a buffer - from `i1` to `i64`.

Backprop-free learning study: spiking (LIF) neurons + Forward-Forward + JEPA + int4 QAT, with a full ablation notebook.

PyTorch implementation of TRIAD-PTQ (Trace-Router-Interaction-Aware Decomposition) — weight-only INT3/INT4 PTQ for compact LLMs and edge CNNs/ViTs, with real benchmarks on SmolLM/TinyLlama/MobileNetV2/EfficientNet-B0/MobileViT-S.

纯 Python 大模型高效推理参考工具包：INT8/INT4 训练后量化 + 分页 KV-cache + 投机解码，离线可跑

Quantize TinyLlama-1.1B-Chat from PyTorch to CoreML (float16, int8, int4) for efficient on-device inference on iOS 18+.

Utilities to rewrite ONNX convolution patterns into MatMul forms for optimal LLM-like int4 quantization (esp. Audio/Speech models).

Training-free INT3 KV cache quantization: 5.09× compression, ~10 lines of Python, <5% WikiText-2 ΔPPL on 8 of 8 open-weight Transformers (GPT-J 2021 → Gemma-4 2026). No calibration, no codebook, no rotation, no adapter. +2.4% decode overhead with torch.compile (no custom CUDA).

Block-scaled FP8 / FP4 / INT4 tensor primitive with Triton scaled-matmul at FP32 parity on H100. NumPy / PyTorch / MLX / JAX backends.

Analyzes per-layer quantization sensitivity in GPT-2 across 9 quantization configs (INT8/INT6/INT5/INT4/INT3/INT2 with group-wise and per-tensor). Key findings: INT8-g32 achieves 1.8x compression with +0.13 perplexity; group-wise quantization reduces INT4 degradation by 99%; mlp_proj layers dominate sensitivity.

Training-free fix for KV cache INT4 failures. Norm separation + per-channel quantization. Qwen2-7B: 744× improvement (ΔPPL +238 → +0.32). 12 models, 124M–40B. 4 lines of PyTorch.