Centralizing Docker Logging Using Logspout and Graylog

Logspout, Docker and Graylog

I’m a big fan of Docker and of centralizing logs. Recently, as I’ve moved more and more of my applications to containers, I’ve realized I’m not centralizing as many of my logs to Graylog. I’m used to applications logging to syslog and then aggregating the syslogs of each machine.

However, Docker doesn’t work this way. Docker uses logging drivers to send their logs in various formats to various applications. After looking into this I decided a combination of Logspout and Graylog was the best way to get the kind of logging I want with the least downsides.

Docker logging drivers overview

Docker logging drivers send container logs to remote locations and files. The default driver, “json-log”, stores logs in JSON format on a local disk. In addition to JSON, Docker has the following logging drivers:

  • none – no logging
  • local – logs stored in custom format for minimal overhead
  • journald – storing container logs in the system journal on the host machine
  • syslog – supporting UDP, TCP, TLS logging to a syslog daemon
  • fluentd – writing logs to the fluentd daemon on the host machine
  • splunk – HTTP/HTTPS forwarding to Splunk server
  • gelf – writing logs in Graylog Extended Log Format (GELF) for Graylog endpoints
  • awslogs – writes log messages to Amazon CloudWatch Logs
  • etwlogs – writes Writes log messages as Event Tracing for Windows (ETW) events (Windows platforms only)
  • gcplogs – writes log messages to Google Cloud Platform (GCP) Logging
  • logentries – writes log messages to Rapid7 Logentries
See also  Living In A Smart Home: An Overview

You can also access Docker logs using the “docker logs” command. The only logging drivers compatible with this commend when using Docker Community Edition are:

  • local
  • json-file
  • journald

Specifying a logging driver is as simple as using specifying the –log-driver option:

docker run \
      --log-driver json-file \
      alpine echo hello world

Logspout Overview

Logspout is a Docker container that routes logs for Docker containers using the Docker API instead of a driver. It attaches to all containers running on the host and routes their logs as configured. Logspout can route logs from different containers to different locations. It can also route logs from the same container to multiple locations.

Why Logspout instead of syslog driver?

With such rich, built in logging capabilities why would you use Logspout instead of a logging driver? Well, there are a few reasons:

  • If you like routing to syslog as I do, you won’t be able to use the docker logs command because it isn’t compatible with the syslog driver.
  • Logspout allows you to manage all your Docker logging outside of each container, which is more in line with the microservices approach.
  • Many of the logging drivers have dependencies on the host system. Logspout eliminates these dependencies. This makes moving containers to different hosts easier.
  • Docker doesn’t support multiple logging drivers at the same time. With Logspout you can send logs to multiple locations.

How to configure Logspout with Docker and Graylog

First, make sure you have a working instance of Graylog. I went through how to set up a Graylog Docker container in a previous article. After that setting up Logspout is easy. Let’s start with a docker run command and then break it down:

docker run -d --name="logspout" \
        --volume=/var/run/docker.sock:/var/run/docker.sock \
        --publish=127.0.0.1:8000:80 \
        gliderlabs/logspout \
        syslog://192.168.10.20:8514

The publish options allow us to access all the logs Logspout is shipping using the curl command:

curl http://127.0.0.1:8000/logs

This allows you to watch the logs being shipped in realtime. I did this at first to debug Logspout receiving and shipping my logs.

See also  My ZoneMinder Surveillance Setup

Here we are sending logs to a syslog server. I have Graylog setup to receive Syslog messages. To send Logspout logs to my Graylog instance I would need to specify the IP address and port corresponding to my Graylog input. I had to restart my containers after creating the Logspout container to see my Docker logs in Graylog.

Below you can see that Plex, which I run in a docker container, now logs to Graylog.

Graylog logspout output from Plex
Graylog Plex Logging via Logspout

As I mentioned earlier, you can specify multiple destinations. You can also specify different formats, multiline logging, JSON parsing and more. The Logspout Github page has more information on advanced options. https://github.com/gliderlabs/logspout. Also, as you can see, we are using the gliderlabs/logspout image.

Some of you may be wondering why I am not using the GELF format if I am using Graylog. These may not be good reasons but I do this because:

  • I’ve got processing rules already set up for the syslog format, and everything else on my network logs to syslog
  • At the time of the writing of this article, the official Gliderlabs Logspout image doesn’t support the GELF format. The Vincit/logspout-gelf image does, however, I prefer to use the official image

Final Thoughts

Application and server logging are important for security and troubleshooting. When you use Docker containers you need to take a couple of extra steps to aggregate your logs. I log using Logspout because it is simple to set up and it’s a better solution than logging directly from the Docker drivers for me.

Interested in supporting HomeTechHacker?

Have you found the content on this site useful? If so, are you interested in supporting me and this site? There’s no obligation of course, but I would really appreciate any support you can give. Below are a few ways you can show support:

See also  Home Assistant: A Fingerprint Scanner for Your Garage

Thank you! I really appreciate it!
Share this:

Centralizing Docker Logging Using Logspout and Graylog

by HomeTechHacker time to read: 3 min