Cheap models at the edge. When Workers AI is enough.

The small "Ask eStreetz" box on this website runs on Cloudflare Workers AI, not on Claude or GPT. That was a deliberate choice, and it made me think about where else we overpay.

Why we tried it

Workers AI runs open models (Llama, Gemma, Qwen, Mistral, gpt-oss and others) on Cloudflare's network. You call it from a Worker with one line, there's no API key to manage, and there's a daily free allowance. For a site widget that answers questions about our own services, that's a good fit: low stakes, short answers, and the full context is a few pages of our own text.

Where it was enough

Website Q&A. We pass the site's content in the prompt and ask the model to answer only from it. A 70B Llama model does this fine. If it doesn't know, it points people to our email.

Classifying and tagging. "Is this message a support request, a sales lead or spam?" Small models handle this well, especially with a few examples in the prompt.

Cleaning text. Fixing formatting, pulling a date out of a sentence, shortening a title. A small model is plenty.

Anything where latency matters more than brilliance. Running close to the user helps. The request doesn't cross an ocean to reach a model.

Where it wasn't

Agents with many steps. We tried a three-tool workflow on an open model. It got lost by step four more often than we were comfortable with. That moved back to Claude.

Long, careful writing. Client-facing drafts and anything with tone. The gap is still noticeable.

Hard code. Not close, for our work. We use Claude for that.

A simple way to decide

We ask two questions:

1. If the answer is a bit worse, does anyone get hurt or lose money?
2. Will this run thousands of times a day?

If the answers are "no" and "yes", we try a small open model first. If it passes our examples, it stays. If it doesn't, we move up. Starting cheap and moving up is easier than the other way round, because nobody ever volunteers to make the expensive thing cheaper once it works.

One practical note

Model catalogues change. A model that a tutorial uses today may be retired in six months. Keep the model name in one config value, run wrangler ai models now and then, and keep your test examples handy so switching is a ten-minute job.

Try the box at the bottom right of this page and see how it does. Then tell us where it got things wrong.

Written by Balakrishna at eStreetz. Questions or disagreements: naveen@estreetz.com