Browser-Based AI

AI inference in the browser is no longer a demo-only curiosity. With WebGPU - the modern web API for GPU access - browsers can now run 1–7B parameter language models, image generation, and real-time vision models locally. No installation, no API key, no server. The model runs on the user's own hardware inside the browser tab.

Enabling Technologies

WebGPU

WebGPU (standardized in 2023) gives JavaScript direct access to the machine's GPU - the same hardware that runs ML inference in native applications. Unlike the older WebGL (designed for graphics, repurposed for ML), WebGPU supports compute shaders natively, making it properly suited for neural network inference:

• Available in Chrome 113+, Edge 113+, Firefox (experimental), Safari (experimental)
• Exposes GPU compute for matrix operations - the core operation of transformer inference
• Typical throughput for a 7B INT4 model on a modern laptop GPU: 10–30 tokens/second - comparable to a cloud API

WebAssembly (WASM)

WebAssembly is the fallback for devices without GPU access. CPU-based ML inference via WASM is significantly slower (typically 1–3 tokens/second for 7B models) but works on any device with a modern browser - including those without discrete GPUs. For small models (under 500M parameters), WASM speed is acceptable for many use cases.

WebLLM - LLMs in the Browser

WebLLM (from the MLC-AI team at Carnegie Mellon) is the primary framework for running full LLMs in the browser:

• Runs Llama 3, Gemma 2, Phi-3, Mistral, and many others via WebGPU
• Models are cached in IndexedDB after first download - subsequent loads are fast
• OpenAI-compatible API - you can swap WebLLM for the OpenAI SDK in your code with minimal changes
• Model sizes from 1B to 13B, with 4-bit quantization; 7B models need ~4GB VRAM (GPU memory)

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("Llama-3.2-3B-Instruct-q4f16_1-MLC");

const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Summarize this document..." }],
});

console.log(reply.choices[0].message.content);
// Runs entirely in the browser, zero API calls

Transformers.js - Hugging Face in the Browser

Hugging Face's Transformers.js brings the Python transformers library to JavaScript/TypeScript, running ONNX-format models in the browser:

• Supports: text generation, classification, NER, question answering, summarization, translation, embeddings, image classification, object detection, depth estimation, and more
• Models from the Hugging Face Hub in ONNX format (automatically converted)
• Works in browser (WebGPU/WASM) and Node.js
• Simpler API than WebLLM; better suited for inference tasks other than chat generation

import { pipeline } from "@xenova/transformers";

// Sentiment analysis - runs entirely in browser
const classifier = await pipeline("sentiment-analysis");
const result = await classifier("I love this product!");
// [{ label: "POSITIVE", score: 0.9998 }]

// Text embeddings for semantic search
const embedder = await pipeline("feature-extraction",
  "Xenova/all-MiniLM-L6-v2");
const embedding = await embedder("search query here");

MediaPipe Web - Vision in the Browser

Google's MediaPipe Web enables real-time computer vision directly in the browser via the device's camera:

• Pose estimation: 33 body keypoints at 30+ FPS
• Hand landmarks: 21 keypoints per hand, real-time
• Face detection and mesh: 478 face landmarks
• Object detection with classification
• Image segmentation (background removal, portrait mode)
• All running locally - no video stream sent to server

ONNX Runtime Web

Microsoft's ONNX Runtime Web runs any ONNX-format model in the browser. Used as the inference backend by Transformers.js and directly by enterprise applications that already have ONNX-format models:

• Backends: WebGPU (fastest), WebGL (compatible), WASM (fallback)
• Supports int4/int8 quantized models for size and speed
• Works with models exported from PyTorch, TensorFlow, scikit-learn, and others

Limitations

• Model size - browsers typically cap memory at 2–4GB. This limits models to roughly 7B INT4 (3.5GB) or smaller.
• First load - a 4GB model download is significant. Caching in IndexedDB helps for returning users, but first-run UX requires careful handling.
• WebGPU compatibility - Chrome and Edge ship with stable WebGPU. Firefox and Safari support is still maturing as of 2026.
• Performance variance - performance varies widely by device GPU. Users with integrated graphics get 3–5 tokens/second; discrete GPUs get 20–50 tokens/second.
• No multiprocessing - browser inference blocks the main thread unless run in a Web Worker; careful architecture required for responsive UIs.

Best Use Cases

• Privacy-sensitive applications (medical questionnaires, legal document review)
• Offline-capable tools that need AI (documentation assistants, code helpers)
• Demos and prototypes with zero backend cost
• Semantic search and embeddings for client-side search
• Real-time vision applications (pose detection, face filters, background removal)