Fluent Bit Log Aggregation Stack

Fluent Bit Log Aggregation Stack

Docker Compose stack that collects logs from all running containers using Fluent Bit and ships them to S3 with daily date partitioning. Includes an Athena table definition so you can query historical logs with SQL immediately after setup.

What Is Included

• Fluent Bit — Collects stdout and stderr logs from all containers via the Docker socket
• S3 Output — Ships logs to your S3 bucket with year/month/day prefix partitioning
• Athena Table — DDL statement to create an external table over the S3 prefix for SQL log queries
• Parsers — Pre-configured parsers for JSON, Docker, and NGINX log formats

How It Works

• Set your S3 bucket and AWS credentials in the .env file
• Run docker compose up -d
• Fluent Bit starts collecting all container logs and shipping batches to S3
• Run the Athena DDL once to create the external table; then query logs with standard SQL

What You Get

• docker-compose.yml — Fluent Bit with Docker socket mounted
• fluent-bit/fluent-bit.conf — input, filter, and S3 output configuration
• fluent-bit/parsers.conf — JSON, Docker, and NGINX log parsers
• athena/create-table.sql — external table DDL with correct S3 location and partition projection
• .env.example — S3 bucket name, AWS region, and credential variables
• README.md — setup guide, example Athena queries, and S3 cost notes

Requirements

Docker Engine with the Compose plugin. AWS account with S3 and Athena access. Standard S3 storage costs apply. Athena charges $5 per TB scanned.