Container logs and how not to lose them

When a service goes down, the log is the only source of truth. By default Docker writes it to a file next to the container and puts no cap on its size: a chatty application can fill the whole disk in a week.

How to read

docker logs -f --tail 100 <container>
docker compose logs -f app
docker compose logs --since 30m --timestamps app
docker logs <container> 2>&1 | grep -i error | tail -50

Worth remembering: Docker shows what the application writes to standard output and standard error. If it writes to its own file inside the container, docker logs will show nothing — you have to look for the log inside or redirect the output.

docker compose exec app ls -la /var/log

Cap the size right now

The global setting for all containers is /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "20m", "max-file": "3" }
}

sudo systemctl restart docker

The setting applies to containers created after a restart. Running ones must be recreated. For a single service, the same goes into Compose:

services:
  app:
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "3"

The result: no more than 60 MB per service instead of unbounded growth.

Check the current size before it surprises you: sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail. You cannot trim a running file with rm — the space is not freed while Docker holds the descriptor. The right way: recreate the container or sudo truncate -s 0 <file>.

Make logs survive the container

The log file is deleted along with the container. If the history matters, hand it over to the system journal:

logging:
  driver: journald
  options:
    tag: "myapp"

journalctl -t myapp -n 200 --since today
journalctl -t myapp -p err --since "2 days ago"

journald rotates records on its own and is capped by the SystemMaxUse setting in /etc/systemd/journald.conf. To check how much space it takes: journalctl --disk-usage.

For several services at once it is more convenient to collect everything in one place — for example, with a dedicated collector container that stores records in files by day. But start with journald: it is already there and costs nothing.

What to look for when debugging an outage

1. The last 200 lines before the crash: docker logs --tail 200 <container>.
2. Exit code: docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' <container>. Code 137 and true in OOMKilled mean the kernel killed the process because it ran out of memory — see resource limits.
3. The machine's system log: sudo dmesg -T | tail -50 will show out-of-memory kills and disk errors.

Common mistakes

• The disk filled up with logs and every service went down. Capping the size is a must right after installing Docker.
• Looking at the log of an already deleted container. After docker compose down it is gone. Collect records into journald in advance.
• Secrets in the log. Applications love printing connection strings with passwords. Anyone with access to the machine can read the log.
• Looking for the error in the text console. It is always 80×24 and long lines get cut off. To read logs, SSH or the “Screen” console is more convenient.