Hot channels

tiny-vllm

AI Tools

Educational LLM inference engine built from scratch in C++ and CUDA

tiny-vllm is an educational sibling of vLLM, implementing a full LLM forward pass (Llama 3.2 1B) from scratch in C++ and CUDA with KV cache, continuous batching, GQA, RoPE, and custom CUDA kernels.

← Back to directory

tiny-vllm

AI Tools

Educational LLM inference engine built from scratch in C++ and CUDA

https://github.com/jmaczan/tiny-vllm

tiny-vllm is an educational sibling of vLLM, implementing a full LLM forward pass (Llama 3.2 1B) from scratch in C++ and CUDA with KV cache, continuous batching, GQA, RoPE, and custom CUDA kernels.