AI in the Browser - WebGPU & Local Models
With WebGPU, a web page can download a small quantized model once and run it on the visitor's own graphics chip, giving private, offline, zero-server-cost AI.
Learning Objectives
- Explain how models can run fully inside a web browser.
- Describe the role of WebGPU and WebAssembly in browser inference.
- Compare libraries such as Transformers.js, WebLLM and built-in browser AI.
- Decide when in-browser inference beats a cloud API.
- Plan for model size, download time and device limits.
Why This Matters
AI-075 covered on-device inference on phones and laptops through native apps. The browser is the most widely installed runtime on earth, and since WebGPU shipped in Chrome (2023), and later in Safari and Firefox, a plain web page can use the GPU for real model inference. That opens a new class of AI features: no API key, no per-token bill, and user data never leaves the device.
Everyday Analogy
Cloud AI is like phoning a translator for every sentence. Browser AI is like the translator handing you a pocket dictionary on your first visit. The first download takes a moment, then every lookup is instant, free and works with no signal.
How It Works
- The model is small and quantized. A 7-billion-parameter model at full precision is around 28 GB. Quantized to 4-bit (AI-065), a 1 to 3 billion parameter model fits in roughly 0.5 to 2 GB.
- The browser downloads and caches it. The first load fetches the weights; later visits load them from the browser cache.
- WebGPU runs the maths. WebGPU gives JavaScript direct access to the graphics chip for the matrix operations at the heart of a transformer (AI-012).
- WebAssembly is the fallback. On devices without WebGPU, WASM runs the model on the CPU, slower but still usable for small models.
The Main Options
| Option | What it is | Good for |
|---|---|---|
| Transformers.js (Hugging Face) | Runs thousands of Hugging Face models in JS, WebGPU or WASM | Embeddings, classification, speech-to-text, small LLMs |
| WebLLM (MLC) | Optimised chat LLM engine with an OpenAI-style API | Full chat assistants like Llama, Phi, Qwen, Gemma |
| ONNX Runtime Web | Microsoft's runtime for ONNX models | Custom models exported from PyTorch |
| Built-in browser AI (Chrome Gemini Nano) | The browser ships its own model and exposes APIs for summarise, translate, prompt | Lightweight features with no download for the site |
Example: Embeddings in the Browser
import { pipeline } from '@huggingface/transformers'
const embed = await pipeline(
'feature-extraction',
'Xenova/all-MiniLM-L6-v2',
{ device: 'webgpu' }
)
const vector = await embed('What is a vector database?', {
pooling: 'mean', normalize: true,
})
That model is about 23 MB and creates embeddings (AI-014) locally. You could build semantic search over a user's notes without sending a single note to a server.
When Browser AI Wins
- Privacy. Medical notes, journals, legal drafts never leave the device.
- Cost. No per-token bill, which matters for free products with many users.
- Offline. Works on a plane or in a poor-signal area once cached.
- Latency. No network round trip for small, frequent tasks like autocomplete or classification.
When the Cloud Still Wins
- Hard reasoning. Frontier models (AI-063) are far larger than anything a browser can hold.
- Low-end devices. Older phones may lack WebGPU or enough memory.
- First-visit experience. A 1 GB download before the feature works will lose many users.
- Consistency. Output speed varies a lot between a gaming laptop and a budget phone.
A common pattern is hybrid: run small tasks (embeddings, filters, quick drafts) in the browser and send hard questions to a cloud model.
Practical Planning
- Show download progress and let users opt in before fetching large weights.
- Pick the smallest model that passes your evals (AI-083).
- Detect WebGPU support (
navigator.gpu) and fall back gracefully. - Run inference in a Web Worker so the page stays responsive.
- Respect memory: release the model when the feature is closed.
Key Takeaways
- WebGPU lets web pages run AI models on the user's GPU, with WASM as a CPU fallback.
- Quantized small models (hundreds of MB to a few GB) make this practical.
- Transformers.js, WebLLM, ONNX Runtime Web and built-in browser AI are the main options.
- Browser AI wins on privacy, cost, offline use and latency for small tasks.
- Hybrid designs keep hard reasoning in the cloud and simple tasks on the device.
Glossary
- WebGPU
- A modern browser API that gives JavaScript direct access to the device's GPU for graphics and general computation, including neural network inference.
- WebAssembly (WASM)
- A fast, low-level code format that runs in the browser. Used as a CPU fallback for model inference when WebGPU is not available.
- Transformers.js
- Hugging Face's JavaScript library for running pre-trained models in the browser or Node.js, using WebGPU or WASM.
- WebLLM
- An open source engine from the MLC project that runs chat LLMs in the browser with WebGPU and offers an OpenAI-compatible API.
- Web Worker
- A background thread in the browser. Running inference in a worker keeps the main page responsive.
- Hybrid Inference
- A design that runs small, frequent or private tasks on the device and sends complex tasks to a cloud model.
References
- [Transformers.js documentation](https://huggingface.co/docs/transformers.js) - run Hugging Face models in the browser
- [WebLLM](https://webllm.mlc.ai/) - in-browser LLM inference engine
- [WebGPU on MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API) - API reference
- [Chrome built-in AI](https://developer.chrome.com/docs/ai/built-in) - Gemini Nano APIs in Chrome
Knowledge Check
8 questions