Centralizing Docker Logging Using Logspout and Graylog

I’m a big fan of Docker and of centralizing logs. Recently, as I’ve moved more and more of my applications to containers, I’ve realized I’m not centralizing as many of my logs to Graylog. I’m used to applications logging to syslog and then aggregating the syslogs of each machine.

However, Docker doesn’t work this way. Docker uses logging drivers to send their logs in various formats to various applications. After looking into this I decided a combination of Logspout and Graylog was the best way to get the kind of logging I want with the least downsides.

Docker logging drivers overview

Docker logging drivers send container logs to remote locations and files. The default driver, “json-log”, stores logs in JSON format on a local disk. In addition to JSON, Docker has the following logging drivers:

• none – no logging
• local – logs stored in custom format for minimal overhead
• journald – storing container logs in the system journal on the host machine
• syslog – supporting UDP, TCP, TLS logging to a syslog daemon
• fluentd – writing logs to the fluentd daemon on the host machine
• splunk – HTTP/HTTPS forwarding to Splunk server
• gelf – writing logs in Graylog Extended Log Format (GELF) for Graylog endpoints
• awslogs – writes log messages to Amazon CloudWatch Logs
• etwlogs – writes Writes log messages as Event Tracing for Windows (ETW) events (Windows platforms only)
• gcplogs – writes log messages to Google Cloud Platform (GCP) Logging
• logentries – writes log messages to Rapid7 Logentries

You can also access Docker logs using the “docker logs” command. The only logging drivers compatible with this commend when using Docker Community Edition are:

• local
• json-file
• journald

Specifying a logging driver is as simple as using specifying the –log-driver option:

docker run \
      --log-driver json-file \
      alpine echo hello world

Logspout Overview

Logspout is a Docker container that routes logs for Docker containers using the Docker API instead of a driver. It attaches to all containers running on the host and routes their logs as configured. Logspout can route logs from different containers to different locations. It can also route logs from the same container to multiple locations.

Why Logspout instead of syslog driver?

With such rich, built in logging capabilities why would you use Logspout instead of a logging driver? Well, there are a few reasons:

• If you like routing to syslog as I do, you won’t be able to use the docker logs command because it isn’t compatible with the syslog driver.
• Logspout allows you to manage all your Docker logging outside of each container, which is more in line with the microservices approach.
• Many of the logging drivers have dependencies on the host system. Logspout eliminates these dependencies. This makes moving containers to different hosts easier.
• Docker doesn’t support multiple logging drivers at the same time. With Logspout you can send logs to multiple locations.

How to configure Logspout with Docker and Graylog

First, make sure you have a working instance of Graylog. I went through how to set up a Graylog Docker container in a previous article. After that setting up Logspout is easy. Let’s start with a docker run command and then break it down:

docker run -d --name="logspout" \
        --volume=/var/run/docker.sock:/var/run/docker.sock \
        --publish=127.0.0.1:8000:80 \
        gliderlabs/logspout \
        syslog://192.168.10.20:8514

The publish options allow us to access all the logs Logspout is shipping using the curl command:

curl http://127.0.0.1:8000/logs

This allows you to watch the logs being shipped in realtime. I did this at first to debug Logspout receiving and shipping my logs.

Here we are sending logs to a syslog server. I have Graylog setup to receive Syslog messages. To send Logspout logs to my Graylog instance I would need to specify the IP address and port corresponding to my Graylog input. I had to restart my containers after creating the Logspout container to see my Docker logs in Graylog.

Below you can see that Plex, which I run in a docker container, now logs to Graylog.

As I mentioned earlier, you can specify multiple destinations. You can also specify different formats, multiline logging, JSON parsing and more. The Logspout Github page has more information on advanced options. https://github.com/gliderlabs/logspout. Also, as you can see, we are using the gliderlabs/logspout image.

Some of you may be wondering why I am not using the GELF format if I am using Graylog. These may not be good reasons but I do this because:

• I’ve got processing rules already set up for the syslog format, and everything else on my network logs to syslog
• At the time of the writing of this article, the official Gliderlabs Logspout image doesn’t support the GELF format. The Vincit/logspout-gelf image does, however, I prefer to use the official image

Final Thoughts

Application and server logging are important for security and troubleshooting. When you use Docker containers you need to take a couple of extra steps to aggregate your logs. I log using Logspout because it is simple to set up and it’s a better solution than logging directly from the Docker drivers for me.

Interested in supporting HomeTechHacker?

Have you found the content on this site useful? If so, are you interested in supporting me and this site? There’s no obligation of course, but I would really appreciate any support you can give. Below are a few ways you can show support:

• Share this site with your friends and on social media (use the sharing links at the end of this page for your convenience)
• Subscribe to this site
• Purchase one of my books, The Personal Cybersecurity Manual, The Home Network Manual or The Smart Home Manual, for yourself or as a gift
• Put a link to HomeTechHacker on a site you have access to. Be sure to let me know about it!
• Reach out to me via my contact page or Twitter and let me know something I should write about
• Shop at Amazon through my affiliate links and ads on these pages. See my disclosures for more details about affiliate links. You can also just shop from one of the links below:
  1. HomeTechHacker Shop: This is a listing of products that I use, have reviewed, and that I recommend
  2. HomeTechHacker Technology Advisor: This suite of tools will give you customized home technology product recommendations based on your needs
  3. My Amazon affiliate link: Just click on this link to go to Amazon and shop

Thank you! I really appreciate it!