Skip to main content

AI in the Browser - WebGPU & Local Models

With WebGPU, a web page can download a small quantized model once and run it on the visitor's own graphics chip, giving private, offline, zero-server-cost AI.

Advanced8 min readv1.0Updated Oct 28, 2026
AI-assisted content — reviewed by the author, but verify important details independently

Visual SummaryClick to explore

Learning Objectives

  • Explain how models can run fully inside a web browser.
  • Describe the role of WebGPU and WebAssembly in browser inference.
  • Compare libraries such as Transformers.js, WebLLM and built-in browser AI.
  • Decide when in-browser inference beats a cloud API.
  • Plan for model size, download time and device limits.

Why This Matters

AI-075 covered on-device inference on phones and laptops through native apps. The browser is the most widely installed runtime on earth, and since WebGPU shipped in Chrome (2023), and later in Safari and Firefox, a plain web page can use the GPU for real model inference. That opens a new class of AI features: no API key, no per-token bill, and user data never leaves the device.

Everyday Analogy

Cloud AI is like phoning a translator for every sentence. Browser AI is like the translator handing you a pocket dictionary on your first visit. The first download takes a moment, then every lookup is instant, free and works with no signal.

How It Works

  1. The model is small and quantized. A 7-billion-parameter model at full precision is around 28 GB. Quantized to 4-bit (AI-065), a 1 to 3 billion parameter model fits in roughly 0.5 to 2 GB.
  2. The browser downloads and caches it. The first load fetches the weights; later visits load them from the browser cache.
  3. WebGPU runs the maths. WebGPU gives JavaScript direct access to the graphics chip for the matrix operations at the heart of a transformer (AI-012).
  4. WebAssembly is the fallback. On devices without WebGPU, WASM runs the model on the CPU, slower but still usable for small models.

The Main Options

Option What it is Good for
Transformers.js (Hugging Face) Runs thousands of Hugging Face models in JS, WebGPU or WASM Embeddings, classification, speech-to-text, small LLMs
WebLLM (MLC) Optimised chat LLM engine with an OpenAI-style API Full chat assistants like Llama, Phi, Qwen, Gemma
ONNX Runtime Web Microsoft's runtime for ONNX models Custom models exported from PyTorch
Built-in browser AI (Chrome Gemini Nano) The browser ships its own model and exposes APIs for summarise, translate, prompt Lightweight features with no download for the site

Example: Embeddings in the Browser

import { pipeline } from '@huggingface/transformers'

const embed = await pipeline(
  'feature-extraction',
  'Xenova/all-MiniLM-L6-v2',
  { device: 'webgpu' }
)

const vector = await embed('What is a vector database?', {
  pooling: 'mean', normalize: true,
})

That model is about 23 MB and creates embeddings (AI-014) locally. You could build semantic search over a user's notes without sending a single note to a server.

When Browser AI Wins

  • Privacy. Medical notes, journals, legal drafts never leave the device.
  • Cost. No per-token bill, which matters for free products with many users.
  • Offline. Works on a plane or in a poor-signal area once cached.
  • Latency. No network round trip for small, frequent tasks like autocomplete or classification.

When the Cloud Still Wins

  • Hard reasoning. Frontier models (AI-063) are far larger than anything a browser can hold.
  • Low-end devices. Older phones may lack WebGPU or enough memory.
  • First-visit experience. A 1 GB download before the feature works will lose many users.
  • Consistency. Output speed varies a lot between a gaming laptop and a budget phone.

A common pattern is hybrid: run small tasks (embeddings, filters, quick drafts) in the browser and send hard questions to a cloud model.

Practical Planning

  • Show download progress and let users opt in before fetching large weights.
  • Pick the smallest model that passes your evals (AI-083).
  • Detect WebGPU support (navigator.gpu) and fall back gracefully.
  • Run inference in a Web Worker so the page stays responsive.
  • Respect memory: release the model when the feature is closed.

Key Takeaways

  • WebGPU lets web pages run AI models on the user's GPU, with WASM as a CPU fallback.
  • Quantized small models (hundreds of MB to a few GB) make this practical.
  • Transformers.js, WebLLM, ONNX Runtime Web and built-in browser AI are the main options.
  • Browser AI wins on privacy, cost, offline use and latency for small tasks.
  • Hybrid designs keep hard reasoning in the cloud and simple tasks on the device.

Glossary

WebGPU
A modern browser API that gives JavaScript direct access to the device's GPU for graphics and general computation, including neural network inference.
WebAssembly (WASM)
A fast, low-level code format that runs in the browser. Used as a CPU fallback for model inference when WebGPU is not available.
Transformers.js
Hugging Face's JavaScript library for running pre-trained models in the browser or Node.js, using WebGPU or WASM.
WebLLM
An open source engine from the MLC project that runs chat LLMs in the browser with WebGPU and offers an OpenAI-compatible API.
Web Worker
A background thread in the browser. Running inference in a worker keeps the main page responsive.
Hybrid Inference
A design that runs small, frequent or private tasks on the device and sends complex tasks to a cloud model.

References

Knowledge Check

8 questions