Ollama + AnythingLLM RAG Stack
Chat with your own documents privately — Ollama and AnythingLLM pre-wired with vector storage and document ingestion.
Ask Questions of Your Own Documents — Privately
RAG (Retrieval-Augmented Generation) lets you drop in PDFs, wikis, code, or any text and then chat with it using an LLM. AnythingLLM is the polished front-end — drag in documents, create workspaces, chat with your content. Ollama runs the model locally. Getting them wired together with persistent vector storage and a working document ingestion pipeline takes an afternoon of config. This stack skips that.
What Is Included
- docker-compose.yml — Ollama + AnythingLLM pre-configured to talk to each other, with volume mounts for documents and vector DB persistence
- Vector storage — LanceDB configured and persistent across restarts (built into AnythingLLM, no separate service needed)
- Document inbox — a local folder mapped into the container; drop files in, AnythingLLM picks them up automatically
- model-pull.sh — pulls llama3.2 (chat) and nomic-embed-text (embeddings) on first start
- .env.example — Ollama endpoint, embedding model, auth token, volume paths
- README.md — setup guide, supported document types, workspace setup walkthrough, and CPU vs. GPU performance notes
How It Works
Run docker compose up -d. The model pull script downloads the chat and embedding models. Open AnythingLLM in your browser, create a workspace, drag in your documents, and start chatting. No OpenAI key needed — everything runs locally.
What You Get Immediately
- A working RAG setup that runs entirely on your hardware
- AnythingLLM UI for managing workspaces, documents, and users
- Persistent vector storage — embeddings survive container restarts
- Support for PDF, DOCX, TXT, Markdown, and more out of the box
- Multi-workspace support — separate knowledge bases for different projects or teams
- API access for integrating with other tools
Requirements
Linux server with Docker and Docker Compose. 8 GB RAM minimum (16 GB recommended for better model performance). GPU optional but improves response speed significantly. No internet connection required after initial model download.