MedGemma and the Local Models Moment

I usually read “privacy-preserving” as a phrase someone had to put in a deck. Here it describes the deployment: run the model inside your infrastructure, give your agents access to it, and leave PHI where it already lives. That’s where we have been pushing portfolio companies anyway. The difference is that the model vendor is now shipping for that posture instead of treating it like an edge case.

Google also lists seven concrete use cases the models are optimized for: high-dimensional imaging from CT, MRI, and whole slide histopathology; longitudinal chest X-ray tracking; anatomical localization; structured extraction from medical lab reports; image classification across radiology, pathology, dermatology, and ophthalmology; report generation; and clinical question answering. That list is mostly the annoying work inside a healthcare operation: repetitive, privacy-sensitive, and awkward to send through a generic hosted API.

The selector is the work

One model doesn’t have to win everything. The useful thing in Google’s lineup is that the routing is legible. Labels and retrieval go to the image encoder. Prose, structured reasoning, and reports go to a MedGemma generator. Cases that mix images, text, and FHIR-shaped records go on the 27B multimodal shortlist.

I would build around this part. Not one medical model behind every button. A small router. Image search to MedSigLIP. Local generative image work to MedGemma 4B. Text-only reasoning to 27B text. FHIR-heavy multimodal work to the larger multimodal model. Much less grand than a giant clinical copilot. Also probably more useful.

Google is also clear about the limit. From the release page: “The outputs generated by these models are not intended to directly inform clinical diagnosis, patient management decisions, treatment recommendations, or any other direct clinical practice applications. All model outputs should be considered preliminary and require independent verification, clinical correlation, and further investigation.” Good. That’s how every medical AI tool should be treated right now.

The boundary

For the past three years, most healthcare AI deployments I have seen have the same shape. The EMR or workflow tool calls OpenAI, Anthropic, or Google. A BAA sits around the call. PHI leaves your network, runs through someone else’s GPUs, and comes back. Sometimes that is fine. Sometimes the security review turns into a long thread no one wants to be in.

With local inference, that middle hop is gone. No PHI leaves. No BAA with the model vendor, because the vendor is not in the path. The audit trail can live beside the EMR audit trail. That’s a smaller surface to explain.

A few practical things follow.

A week earlier

MedGemma got me to write this down because I had just seen the same pattern somewhere else.

On April 16, Simon Willison reported that Qwen3.6 35B A3B, a 20.9GB quantized model running locally on his M5 MacBook Pro through LM Studio, beat Claude Opus 4.7 on his long-running pelican-on-a-bicycle SVG test. Even with thinking level cranked to max, Opus could not render a correct bicycle frame. The flamingo-on-a-unicycle variant went the same way.

Yes, the test is silly. I still pay attention to it, because Simon has been running it long enough that the results tend to track real model utility. He is careful with his conclusions: he is not claiming the 21GB file is genuinely more useful than Anthropic’s newest release across the board. He is reporting that on one specific, non-trivial task, on his laptop, it was.

Two data points are not a law. But I have a hard time ignoring the pair: a 4B medical model writing a defensible X-ray impression on a laptop, and a 35B general model beating a hosted frontier API on a weird rendering test on consumer silicon. My old default was simple: serious healthcare AI lives behind a hosted API. That default is getting harder to keep.

And to be clear: I still love you, OpenAI and Anthropic. My Claude subscription is not going anywhere. Frontier hosted models still do plenty of things no 4B model can. I’m just less willing to send every task there by default.

Notes from the toolchain

If you have not run a local model in the last six months, the developer experience has changed more than the public scoring discourse has. A few concrete things we have learned putting this into portfolio company workflows.

Quantization has mostly stopped being scary. 4-bit via GGUF or MLX cuts memory by roughly 4x with accuracy loss that, for the kinds of structured tasks we run (extraction, summarization, classification), we cannot reliably tell from the full-precision version. A 27B model that would not fit on a workstation in its native form sits comfortably in 18GB of VRAM at Q4. MedGemma 4B quantized is about 3GB, which is why it runs on my old laptop and not just the current-gen one.

The runtime layer is the part that surprised me. llama.cpp handles most CPU and GPU targets cleanly. Apple Silicon gets first-class treatment through MLX, which is what makes the M-series laptop workflow feel fast instead of academic. Ryzen AI PCs now have AMD Lemonade doing the same job on the Windows / NPU side. Ollama and LM Studio wrap all of this with OpenAI-compatible HTTP endpoints, so your existing code that calls chat.completions.create drops in with one changed line.

Running a model locally in 2026 is weirdly normal. Not research-project normal. Normal-software normal. ollama pull medgemma:4b and you are talking to the thing. The integration cost I was budgeting for a year ago turned out to be almost nothing.

Tool calling works. Both MedGemma and Qwen support structured output and function calling via standard JSON schema. The same pattern we wrote about in Building the Bridges Your AI Needs. The MCP servers we originally built for hosted models dropped into the local setup with almost no changes.

The weak spots are real. Long-context reasoning beyond 32K tokens still favors hosted frontier models. Complex agentic workflows that stitch five or six tool calls into one session are still better on a Claude or GPT-class model. Anything that depends on a very recent training cutoff is frontier-only by definition. So I’m not thinking replacement. I’m thinking routing. Let the local model handle the boring private bulk. Spend hosted tokens when the request actually needs them.

What I would change now

We spend most of our time inside the AI infrastructure of portfolio companies: scribes, coding engines, prior auth agents, scheduling, patient messaging. Here is how I would change the build plan now.

Where I landed

Three years ago I assumed serious healthcare AI would live behind hosted APIs for a long time. I don’t anymore. A 4B medical model runs on a laptop. A 35B general model can beat a hosted frontier API on a concrete task on consumer hardware. That doesn’t make local the whole stack. It makes it part of the stack.

The practical upside is boring and good: less PHI moving around, lower variable cost, fewer hard dependencies on someone else’s uptime. That’s enough for me.

If you want to pressure-test which parts of your healthcare AI workload could move local, get in touch. We do this work inside portfolio companies.