Running your ChatGPT-like LLM locally on docker containers

Introduction:

In today’s data-driven world, the demand for advanced natural language processing (NLP) capabilities has surged. Large language models (LLMs) have emerged as powerful tools for tasks ranging from text generation to sentiment analysis. However, concerns regarding data privacy and security have led to a growing preference for running LLMs locally. In this article, we’ll explore how Docker containers and Docker Compose streamline the deployment of Ollama, a tool for running open-source LLMs locally, while enhancing data privacy and security.

Introducing Docker Containers

Docker containers have revolutionized software deployment by providing a lightweight, portable, and isolated environment for running applications. Containers encapsulate an application and its dependencies, ensuring consistent behavior across different environments. With Docker, developers can package their applications into containers, making deployment seamless and efficient.

A more comprehensive introduction on how to run applications on docker containers can be found here. A rather quicker-to-do version of is available here.

For the ones loving the visual (GUI) way of doing things, try portainer.io! (Link here)

Understanding the Shift to Local LLM Execution

In recent years, there has been a noticeable shift towards executing LLMs locally. This shift is driven by concerns surrounding data privacy and security. Organizations and individuals are increasingly wary of entrusting sensitive data to cloud-based services, where data may be vulnerable to breaches or unauthorized access. By running LLMs locally, users gain greater control over their data and can ensure compliance with privacy regulations.

Introducing Ollama: A Solution for Local LLM Execution

Ollama addresses the need for local LLM execution by providing a streamlined tool for running open-source LLMs locally. By bundling model weights, configurations, and datasets into a unified package managed by a Modelfile, Ollama simplifies the deployment process while preserving data privacy and security.

Leveraging Docker Compose for Ollama Deployment

Docker Compose offers a convenient way to deploy Ollama, enabling users to define and run multi-container Docker applications with ease. As from the hyperlinks in the earlier section you must have understood how to run applications on docker, we’ll directly jump into how Docker Compose can be used to deploy Ollama, ensuring that both the backend and web UI components are securely encapsulated within Docker containers. Here is the docker compose for you:

version: '3.8'

services:
  ollama-backend:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - /ollama:/root/.ollama
    restart: always
    container_name: ollama-backend

  ollama-openweb-ui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_API_BASE_URL=http://ollama-backend:11434/api
    depends_on:
      - ollama-backend
    restart: always
    container_name: ollama-openweb-ui

Docker Compose will build and start the containers based on the specifications in the docker-compose.yml file as we wrote above, exposing the webUI on host port 3000.

By running LLMs locally with Ollama, users can mitigate the risks associated with cloud-based services. Data remains within the user’s control, reducing the likelihood of unauthorized access or data breaches. Furthermore, running LLMs locally minimizes exposure to third-party services, enhancing overall data privacy and security.