Browsers can run real AI workloads.
Running AI at the Edge: Real Workloads Directly in the Browser — QCon London 2026 Recap
Browsers can run real AI workloads.
Browsers can run real AI workloads, and practical implementations now let engineers run inference on-device using WebGPU, WebNN, WebAssembly, and optimized runtimes so you can serve models without round trips to a server while keeping privacy and latency tight, which changes the architecture of interactive apps and how you think about user data.
How does this change work for teams?
Key Takeaways:
- On-device inference reduces latency and improves privacy.
- Use WebGPU and ONNX Runtime Web for best performance in many cases.
- Quantization and smaller models are essential for browser deployment.
- Single-page apps should use WebWorkers and cache models for robustness.
What is Running AI at the Edge in the Browser?
Browsers running ML means models execute in the client.
This approach executes models with WebGPU, WebNN, or WebAssembly backends—those APIs talk to the GPU or optimized CPU paths in the browser so you can do image classification, speech recognition, and some transformer inference without a server, which reduces round trips and allows offline scenarios for users with intermittent connectivity.
Why would you choose this?
Because latency and privacy matter.
When I first prototyped a browser-only classifier at a startup, the load times were brutal.
I rebuilt it by switching from a heavy Keras model to a 4MB quantized ONNX blob and loading it in a WebWorker, which cut inference time by two-thirds and kept photos on-device, and that change improved retention because users saw instant feedback.
Here's the kicker: small choices like threading and caching matter more than raw model size.
How do the pieces fit together?
Core Benefits/Importance
Fast results matter most.
On-device inference cuts network round trips and keeps sensitive data local, which helps with user trust and compliance while lowering cloud cost when you reduce server-side compute spikes, and it also gives predictable latency benefits for interactive features like camera apps, live captions, and real-time filters.
What are the other wins?
Privacy, offline capability, and cost savings.
Privacy becomes practical because photos and audio never leave the device unless the user opts in, and offline capabilities mean features are available on planes and in basements—this is how you design respectful products that honor user dignity and serve the common good.
When I worked on accessibility features, running models locally meant users on slow networks still got reliable captions, and that reliability translated to trust and usage growth.
What trade-offs should you expect?
Step-by-Step Implementation/Guide
Start small and measure.
Pick a tiny model, convert it to ONNX or TF.js format, quantize to 8-bit or 4-bit if the model tolerates it, run it in a WebWorker to keep the UI thread smooth, and benchmark with representative inputs to find where accuracy and speed cross acceptable thresholds for your users—those steps give you a repeatable rollout path.
Which runtimes should you try first?
Try ONNX Runtime Web, TensorFlow.js, or transformers.js.
Concrete steps I use when launching a browser ML feature: first, train and prune a model on server hardware; second, convert to ONNX and run quantization with tools like ONNX Runtime Web converters; third, bundle the model with a loader that checks the environment for WebGPU support and falls back to WebAssembly if necessary; fourth, load the model inside a WebWorker and use a WebGPU guide to schedule GPU workloads without blocking the main thread.
Is there a full-stack checklist?
Yes: model size, quantization, runtime, threading, caching.
Sample checklist I follow: (1) measure inference time on low-end devices, (2) benchmark warm vs cold loads, (3) add progressive loading (load a tiny model first, then swap to a larger one if the device supports it), (4) keep a server fallback for heavy tasks, (5) track metrics like first-inference time and compute use in user sessions, and (6) iterate.
If you need a compatibility fallback, use WASM guide strategies to ensure broad device coverage.
Comparison Table
Make the trade-offs visible.
The table below contrasts browser-based edge inference with the typical server-side inference approach so teams can weigh privacy, cost, and UX trade-offs before choosing a direction—this kind of analysis prevents architecture regrets later.
Which is better for you?
| Feature | Browser (Edge) Inference | Server-Side Inference |
|---|
| Latency | Low for single-user interactions | Low for batched requests |
| Privacy | High (data stays local) | Lower unless encrypted and isolated |
| Cost | Low per-request, uses client resources | Higher cloud compute costs |
| Scalability | Scales with clients | Scales with servers and infra |
| Model Size | Constrained by device memory | Supports larger models |
| Offline Support | Yes | No without caching |
Common Mistakes to Avoid
People copy server patterns.
The biggest misstep is shipping large, unquantized models to the browser without fallbacks, which kills load performance and frustrates users on modest devices, and another is using synchronous APIs that block the main thread—both are trivial to avoid but easy to miss during early prototypes.
What did I see in the field?
I've watched teams ship slow features.
At one company, a team deployed a 120MB model inside a single-page app and the app took twenty seconds to be usable on low-end phones, which tanked engagement, and fixing that required a rollback and a painful rewrite to progressive model loading and quantization.
Here's what nobody tells you: the user experience, not raw accuracy, decides success for client ML.
So what are practical rules?
Rules I follow: always provide a tiny fallback model, use WebWorkers, measure cold start times, and keep telemetry minimal and respectful of privacy—design choices should show stewardship of user data and resources.
Want a quick checklist to avoid mistakes?
Frequently Asked Questions
Q: Can browsers run large transformer models?
A: Not full-size ones usually, but trimmed and quantized versions can serve many tasks, and frameworks like transformers.js make it practical to run smaller language models client-side for autocomplete and on-device assistants.
Is offline chat possible?
Yes, with small or compressed models.
Q: Which browser APIs should I learn first?
A: Learn WebGPU for GPU workloads and WebNN for higher-level inference primitives, and then understand WebAssembly for fallback compute—MDN and the W3C spec are good references when you need details on supported features and performance considerations, and MDN WebGPU docs is a practical starting point.
Will my app run everywhere?
No, you need progressive enhancement.
Q: How do I handle updates and model versioning?
A: Use content-addressed model blobs with caching headers and compare checksums on load, fall back to server-side inference for older clients, and keep metrics that help you decide when to push new versions—this reduces wasted bandwidth and respects users' data caps.
Need more help on conversion and runtimes?
See ONNX Runtime Web and TensorFlow.js guides for concrete commands.
Final Thought
Browsers can run heavy workloads now.
Browsers can run heavy workloads now, and teams who do this well treat the capability with prudence—focusing on small models, clear fallbacks, and respectful telemetry so features serve users without wasting resources, because good engineering is also stewardship of talents and time.
Whoever ships the next useful on-device feature should remember that product quality and user dignity matter more than novelty.
— Ermenildo Valdez Castro, Jr.
Sources & Further Reading