---
title: Local AI Isn't Perfect
date: 2026-11-12
date_published: 2026-11-12
status: published
tags:
  - ai
  - mac
  - apple
summary: Local models from Ollama, LMStudio, and Apple are already very usable, but memory, context length, and quantization mean you'll need to pick your model carefully.
---

You probably know that I'm a fan of local language models for AI. Ollama, LMStudio, and Apple's on-device AI are all examples of local AI, and don't need big data centers serving up the power.

> Apple themselves appear to be developing an app called Harbor (via their Claris subsidiary) that allows a Thunderbolt 5 Mac to be used by other connected Macs as a local AI server.

But I do need to mention that local AI doesn't come without challenges. One reason why my Macs are all 64GB RAM or greater and maxed out on GPUs has to do with one problem: quantization.

The local problem with AI at the moment has to do with model size and context. It isn't just good enough that the model fits into your computer's GPU-accessible memory. A 70 billion parameter model may take 40+GBs of memory right at the get go. You still need memory for data, conversation memory, and context length. Indeed, the more lengthy and complex the task you ask the model to work on, the more likely that the local AI breaks when you attempt that task.

One way around the problem can be quantization. This reduces the bit size for weights. Think of describing a pixel value in your image using fewer bits. In AI, it's similar: the quantized value for a weight may still be recognizable after reducing the bits, but it's compromised in a way that's not recoverable.

This likely is how Apple fits an AI engine into 12GB RAM Apple Silicon devices for Siri. Fewer parameters, more quantization. Like pixel compression, this may still result in an outcome that is fine for your needs, but it certainly isn't the best that can be done.

If you're using Ollama or LMStudio, you'll probably have to experiment with different models to see what runs comfortably in your machine without hitting a choke point (you'll know, because you'll get a context limit message and the AI stops its response). Find a model that fits easily into the available memory, and consider picking a quantized version of that model.

It's pretty remarkable what you can do with a Gemma 4.0 12B or 12B QAT (quantized) model on most Macs. Both are less than 8GB in download size, so work pretty comfortably in a Mac with 32GB or more of memory.

Put a different way, we're in the early days with local AI. The data center models are, oh, at middle days at the moment. So yes, you can do more complex, deeper work, and faster, with ChatGPT and Claude and their brethren living in the big data centers. But don't discount the local choice; it's gotten very usable already, and will only get better.

Unfortunately we're still in the nerd days with local models. There's no "Local AI for Dummies" yet. But I'm pretty sure we'll get there. Apple certainly is pushing that direction.