What's the Best Local AI Model

As an information scientist and sci-fi geek, I find modern AI developments awesome to behold. But I'm frightened. Frightened that these tools will become forever locked behind corporate gates, generating profit for a bunch of already-rich sociopaths and making the rest of our lives miserable.

That's why local AI is so important.

Think about it: one of these models can cost more than $100 million dollars to train, and they're all trained on the public's data. Your data and mine. The lifeblood of our culture flows through these machine minds, so doesn't it make sense that our culture should benefit? That You and I should benefit? Who deserves it more, the CEO and shareholders of some ridiculous company, or we, the people who made the technology possible in the first place?

“Qwen 2.5 Coder Just Blew SOTA Closed LLMs Out Of The Water. The folks on the other side of the world in China are not just geniuses; they are big proponents of open-source as well.” (Source)

Open-source means that the inner workings of a piece of software are laid bare for anyone to see. In the case of AI, open source particularly refers to "open weights."

Imagine you have a super smart robot that can understand lots of different words and ideas, but it's still trying to learn how to be really good at understanding humans - the "weights" are like tiny switches in its brain that get flipped on or off depending on whether they help the robot make better decisions or not.

As with most things, closed-source development by major corporations leads us down a dark road of servitude

If the AI model's weights were open-source, it would mean that anyone could look at and modify them - kind of like how you can see the blueprints of a building and make changes to it.

In this case, making the weights open-source would allow:

• Researchers to study how the robot learned certain things and try to improve its understanding
• Developers to use the knowledge gained from studying the weights to create even better AI models
• Anyone to contribute their own modifications or improvements back into the system so that it gets even more powerful over time

It's like having a community-driven project where everyone can work together to make something really cool - but in this case, it's an AI brain.

Top open-source AI model under 13b

As of December 2024, an updated model sponsored by the United Arab Emirates (UAE) has topped the charts. The Falcon 3 model can beat out closed-source models in almost every category, and even do better than similarly sized open-source models.

The Falcon 3 7b and 10b models are simply incredible, standing head and shoulders above the competition. This means that they're better at interpreting human language and mathematics than any other similar models, and they're capable of better logical reasoning and mathematics than similar models.

Models of these sizes are especially exciting because they can run on a relatively weak laptop with comparably low RAM. For instance, I regularly run the Llama 3.1 8b model on my Framework 14 laptop, which doesn't have a dedicated GPU and only boasts 32gb of memory -- while on nothing but battery power.

Top open-source AI model for a home PC

The top open-source model for someone hoping to use an existing desktop computer has to be the new Qwen 2.5 32b model from China. The goal of this model was to match or exceed GPT 4 in most tasks, particularly coding support. It managed this, and more.

Impressively, Qwen actually outperformed leading closed-source models when run at its full size and capabilities. Running it on normal consumer-grade hardware will cut into that success margin, certainly, but you would still be looking at having completely private access to one of the most powerful AI models on the planet.

I run Qwen 2.5 on my home-built PC pretty smoothly, with token speeds slightly slower than my reading speed. While this is far slower than paid closed-source models, you have to remember that everything the model does is happening right on my own machine (and I'm not paying large monthly fees for the privilege).

My PC has a Ryzen 9 7950x processor, 64GB of RAM, and a 24G XTX graphics card - making it significantly powerful for a home PC, and reasonably capable of running models in the 32b range. For those of you with a Mac, you'll need the M2 chip and 64GB of RAM to properly use this model (and not close down all the other applications on your PC).

Closing up

Open-source models are the way of the future when it comes to AI development. As with most things, closed-source development by major corporations leads us down a dark road of servitude, while open-source puts the power of these models back in the hands of the people, where it belongs.

Give it a try for yourself! Download one of these models and tell me in the comments how well it worked for you!

👋 Hi there!
I’m Odin Halvorson, a librarian 📚, independent scholar 📖, film fanatic 🎬, fiction author 📝, and tech enthusiast 💻. If you like my work and want to support me, please consider becoming a paid subscriber to my newsletter for as little as $2.50 a month! 📣

Support me in other ways:
Use this affiliate link to sign up for Libro.fm, the best audiobook platform around! 🎧 Want your own Ghost website? Check out MagicPages for the cheapest rates via my affiliate link (they even offer lifetime hosting plans!) 📈

Thanks for your support! 🙏