Hands-On: Deploying a Local Generative AI Pipeline on Raspberry Pi 5 with AI HAT+ 2

Hook: Stop explaining outages — run reproducible, offline demos

Developers and platform engineers: if your demos, workshops, or dev-lab exercises fail because of flaky cloud access, high latency, or unpredictable costs, you need a reliable local stack. In 2026 the Raspberry Pi 5 + AI HAT+ 2 combination makes small, usable generative AI models realistic at the edge. This hands-on guide shows how to set up, run, and benchmark a compact generative pipeline for offline demos, classroom labs, and proof-of-concept deployments.

What you'll get — at a glance

• Hardware checklist and setup tips for Raspberry Pi 5 + AI HAT+ 2
• Two inference paths: CPU-only (llama.cpp / ggml) and accelerated (ONNX + AI HAT+ 2 delegate)
• Model preparation and quantization steps for small generative models
• Benchmarking recipes and scripts (latency, tokens/sec, power)
• Optimization and production-ready tips for dev labs and offline demos

Why this matters in 2026

Late-2025 and early-2026 trends made this approach practical: 4-bit quantization (widespread), the GGUF model packaging standard, and improved ARM64 runtimes (llama.cpp, onnxruntime ARM delegates, and community-driven optimized kernels). Edge NPUs like those on the AI HAT+ 2 now offer real throughput for small to medium models, making low-latency, private inference feasible for demos and constrained deployments.

Prerequisites (hardware and software)

Hardware

• Raspberry Pi 5 (4–8 GB models recommended; 8 GB preferred if you plan larger models)
• AI HAT+ 2 module (vendor-supplied M.2/PCIe AI accelerator for Pi 5)
• Fast storage: NVMe SSD (USB4/PCIe adapter) or high-end microSD (UHS-II)
• Quality 5V/6A power supply (or factory-supplied Pi 5 PSU)
• Optional: active cooling (fan + heatsink) and USB power meter for power benchmarking

Software

• 64-bit OS: Raspberry Pi OS (64-bit) or Ubuntu Server 24.04/24.10 arm64 (2025/2026 builds)
• Developer tools: git, build-essential, python3, pip
• llama.cpp (or equivalent ggml runtime) for CPU path
• ONNX + onnxruntime (ARM64) for accelerated path — plus AI HAT+ 2 vendor SDK/delegate

Step 1 — Hardware setup and first-boot configuration

1. Assemble Pi 5 with AI HAT+ 2: securely seat the HAT into the Pi 5 expansion header or M.2 adapter per vendor instructions. Add an NVMe SSD to adapter if you plan to store models on disk for speed.
2. Attach an active cooling solution — Pi 5 under load benefits significantly from an airflow fan + heatsink. Monitor temps during first runs.
3. Flash your 64-bit OS (Raspberry Pi OS 64-bit or Ubuntu Server 24.04 LTS arm64). For Raspberry Pi OS, use Raspberry Pi Imager 2025+ and choose 64-bit release.
4. Initial boot: update packages and enable SSH for headless access.

   sudo apt update && sudo apt full-upgrade -y
   sudo reboot
   

5. Install common tools:

   sudo apt install -y git build-essential python3 python3-pip cmake libopenblas-dev liblapack-dev libomp-dev

Step 2 — Decide your inference path

Two practical approaches for developer labs:

• CPU-only (fastest to set up): Use llama.cpp / ggml backends and a 1.3B–3B quantized model. Ideal for single-user demos and offline chat bots.
• AI HAT+ 2 accelerated: Export model to ONNX or a vendor-supported runtime and use the HAT+ 2 delegate. Improved tokens/sec and lower latency for interactive demos with multiple sessions.

Path A — CPU-only with llama.cpp (quick and reliable)

Why choose this

llama.cpp (and forks) are lightweight, easy to compile on ARM64, and support multithreading and quantized ggml models. For many labs, a 1.3B or quantized 3B model gives acceptable latency without vendor drivers.

Install and build llama.cpp

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j4

If you have an 8-core Pi 5, use -j8. The build produces a fast, portable binary.

Get a small model and convert

Use a small open model (1.3B or community small models) in GGUF or convert an HF checkpoint to ggml/gguf. Many community models in 2025–2026 ship already in GGUF.

# Example: convert a Hugging Face checkpoint (high-level, follow model license)
# 1) download model to /models/my-model
# 2) use conversion utilities (see llama.cpp tools/convert or community scripts)
./quantize model-fp32.bin model-q4_0.bin q4_0

Store the quantized model on your NVMe SSD for best I/O performance.

Run a simple inference

./main -m models/my-model.gguf -p "Write a short workshop outline for local AI demos" -n 128

This should print streaming tokens. Measure latency with time if you want single-run numbers.

Path B — AI HAT+ 2 accelerated (higher throughput)

Why choose this

When you need faster interactive demos or multiple concurrent sessions, the AI HAT+ 2 accelerator reduces CPU load and increases tokens/sec. In 2026, vendor SDKs typically provide an ONNX Runtime delegate or an NPU runtime that plugs into existing pipelines.

Install the AI HAT+ 2 SDK and ONNX Runtime

Follow the vendor download instructions for AI HAT+ 2: install kernel driver, runtime, and ONNX delegate. The steps below are representative; always consult the vendor's docs for the exact package names and signatures.

# Example (vendor placeholder):
# 1) Install vendor kernel modules and runtime (requires reboot)
sudo dpkg -i ai-hat2-kernel-*.deb ai-hat2-runtime-*.deb
sudo reboot

# 2) Install onnxruntime for ARM64
python3 -m pip install onnxruntime==1.16.0 --extra-index-url https://download.vendor/onnx

After installation, verify the delegate is registered with onnxruntime and that the device shows up in vendor tools.

Export your model to ONNX

Use Hugging Face Transformers + Optimum (or the vendor’s conversion tool) to export a small model to ONNX. In many cases you’ll then apply quantization or compile to the vendor format.

python -m pip install transformers optimum[onnx]
python export_to_onnx.py --model my-small-model --output model.onnx
# Vendor-specific compilation (placeholder)
vendor-compile --input model.onnx --out model.hat2

Run onnxruntime with the HAT+ 2 delegate

python run_onnx_inference.py --model model.hat2 --prompt "Hello edge AI"

Expect lower wall-clock latency and higher tokens/sec, but check memory usage and thermal throttling under sustained load.

Model preparation and quantization best practices

• Prefer 4-bit (q4) quantization for best size/perf tradeoff. 3–4-bit quantization is the standard in 2025–2026.
• Use GGUF wherever possible — it’s become the de-facto packaging standard for ggml models and simplifies metadata handling.
• Validate model outputs after quantization — small numeric differences are expected but verify prompts used in your demos.
• Store hot models on NVMe to prevent I/O stalls when streaming, especially on demos that repeatedly load models.

Benchmarking: real, repeatable measurements

Benchmarks should measure three things: latency (time to first token), throughput (tokens/sec), and power draw. Use micro-benchmarks and representative prompts.

Simple latency/throughput script (bash)

# run-benchmark.sh
PROMPT="Summarize the history of local AI in two sentences."
MODEL_PATH=models/my-model.gguf
ITER=5
for i in $(seq 1 $ITER); do
  start=$(date +%s.%N)
  ./main -m $MODEL_PATH -p "$PROMPT" -n 128 > /tmp/out.txt
  end=$(date +%s.%N)
  elapsed=$(python3 - <