Hans Hasselberg

Purpose

I wanted to extend the 6Wunderkinder infrastructure with service discovery and improve the health checks. For these goals Consul seemed like a very good fit. Getting the key/value store was just the icing. In the following, I will explain the setup, things I’ve run into and share the configuration. I was looking for such a description myself and couldn’t find one.

First steps with Consul

The first feature of Consul I used was the KV store. During provisioning of the instances I used Consul to set up credentials for database access. It was running for quite some time before I started integrating services and health checks. When I did that, I noticed two problems:

1. Foreign nodes in the cluster which turned out to be my fault
2. Unrelieable joining and leaving of nodes

1. Foreign nodes

This was a really strange and scary thing. I only discovered it by going through the list of nodes manually with consul members. The Consul setup is in public AWS EC2. Not every port was properly protected by AWS Security groups. That was the first thing I changed. I created a dedicated AWS Security Group, in which I opened every port mentioned in the docs at the very bottom to itself. Now every server and client must have this group in order to participate. I also enabled encryption and TLS with this help. I should’ve done that in the first place and you should too.

2. Unrelieable joining and leaving of nodes

Whenever I inspected consul monitor pages of output were flying by and I thought that is normal. I learned it is not. I was running into a problem and after giving it some thought, Armon from Hashicorp was able to identify and fix it. Turns out the Serf event bus was saturated. The lesson here is that you should always be monitoring because then the problem would’ve been easy to spot. I should’ve done that in the first place and you should too.

Configuration

As promised I’ve uploaded the configuration.

Conclusion

The cluster is stable and everything works as expected. I am looking forward to put Consul to its use: orchestration. I want to thank Torsten for his great feedback on this post! Thanks for reading.