How To Run Large Language Models Locally On Your Laptop (Using Ollama)

In the wake of ChatGPT’s debut, the AI landscape has undergone a seismic shift. Large Language Models (LLMs) are no longer just a niche; they’ve become the cornerstone of modern machine learning, representing the pinnacle of AI advancements. From tech giants to startups, everyone seems to be unveiling their own LLM and integrating into their products.

One of the biggest hurdles with Large Language Models has always been their demanding hardware requirements, making it seem nearly impossible to run them on our everyday laptops or desktops. Yet, here lies the breakthrough: we’ve found a way to harness the power of LLMs right on our local machines, sidestepping the need for those high-end GPUs. This is the guide on how to run LLMs on laptops.

Deploy LLMs Locally with Ollama

For this tutorial we will be using Ollama, a nifty tool that allows everyone to install and deploy LLMs very easily. So let’s begin.

Important notes:

1. For this tutorial we will be deploying Mistral 7B. This has a minimum requirement of 16GB memory.
2. 3B, 7B and 13B models require 8B, 16GB and 32GB memory respectively.
3. Inference is done locally meaning no internet connection is required besides the initial download of the model.
4. Mistral 7B is licensed under apache 2.0, allowing anyone to use and work with it.

Update July 2024: Meta released their latest and most powerful LLAMA 3.1 models. The process of running the Llama 3.1 models is the same, the article has been updated to reflect the required commands for Llama 3.1: 8B, 70B and 405B models. It is now recommended to download and run the Llama 3.1 models that can be run locally on your laptop.

That’s it! It is as simple as that.

Which LLM To Download? Mistral 7B vs Llama 2?

Which Large Language Model to deploy to might be a hard question. There are a lot of factors to consider and also your own requirements and what hardware you might have. There is a good rule of thumb; the higher the parameter count, the better the model performs. But you might have noticed something different in this tutorial and that is we are using Mistral 7B.

There are two major reasons for this. The first being Mistral 7B outperforms almost all open-source 7B models currently available. Mistral 7B outperforms Llama 2 13B in all benchmarks, which is quite impressive. In fact Mistral 7B outperforms Llama 1 34B on many benchmarks! The second reason being Mistral 7B requires 16GB memory which is more doable than a 32GB memory requirement for 13B models.

Why Run LLMs Locally?

Another important question that you should be asking is, why even bother running these massive LLMs on your laptop? Well, the short answer is you own everything.

Using a tool like Ollama will allow you to run LLMs on your own machine. That means no need to rely on a 3rd party APIs or using any cloud services. Running models locally means users’ data doesn’t need to leave their machine, offering a higher level of privacy and security. There is also a high degree of flexibility, allowing you to customise the model according to your needs. Ollama also offers REST API support.

It is going to be only a matter of time before we see even more powerful models being able to run locally on a laptop or desktop. Maybe even a phone in the not so far future.