It’s been a few weeks since my last update, and for anyone who follows me on Twitter you would have discovered that my progress is going, albeit slow. Life tends to get in the way of things, and getting through the learning curve of this project, without getting horribly sidetracked, has been a huge challenge, but, I knew this when I started, and I actually love every second of it! I know so much more about linux, networking, ansible, and docker at this point, then I though I could cram into just a few months at this.
So whats been going on? The primary update is that the Intel NUC servers are up and running. I have DNS, Docker, Java, Consul Server, and Nomad Server currently running on them. This has a huge benefit of allowing me to already start to play around inside my cluster, because every other node (alpha, berries, and sandbox) all are consul/nomad clients and docker running. Spinning up an ELK cluster is just a quick command line from my windows box!
The current state didn’t come without challenges. The first challenge was, getting docker installed. Let me guess, there is like… 500 tutorials online in getting Docker running on Raspberry Pi right? Well, until you hit the newest “Docker CE” that launched around the first of March, which doesn’t install the same way, making most tutorials pretty outdated, then when looking through the new documentation for Docker CE, I hit a few brick walls getting the correct distribution. Turns out I needed to do some additional work and mappings for the Pi’s armv7l and use the armhf packages. Which additionally, I had to do similar things for Consul/Nomad to get the arm downloads.
Docker in total, took me about a day or so in getting it fully working on the Pi’s, only about 20 minutes on CentOS, so that was a good thing :)
Consul, was pretty simple, I am pretty familiar with this, and standing up a new cluster is really pretty trivial. Nomad is similar to how Consul is configured, except… it can use Consul to self-configure, so it really only took a few settings, and it bootstrapped and discovered its self. Something to think about in the future when building my own software.
DNS was minimal effort, remember that part I talked about previously in using Ansible? Well, I just added the new servers to the DNS group in my hosts file, and ran Ansible… Done. I have switched over my router’s DHCP settings to use two of the three DNS servers for my network, and BAM, everything is accessable with DNS now! I did do a bit of extra effort to configure the google DNS forwarding and Consul DNS forwarding, creating a nice public/private/dynamic setup for all of my systems.
The next phases of this project is to start monitoring everything. I have the hashi-ui running in a container to monitory my Nomad nodes, however, that isn’t enough for me, since this project is about doing it “right”, or finding a “right for me” solution. I have a quick and dirty prototype up and running right now (btw, thats why its awesome to spin shit up/down so easily) and will be using the ELK stack + Beats to do the first level of monitoring. Metricbeat should be able to give me everything I need initially, however I might delve into creating a few custom beats just for my own purposes and open source them (temp-beat, nomad-beat). One of the hardest parts about getting Metricbeat deployed is compiling it for arm, while Elastic has builds/tests for arm/arm64, they don’t actually distribute it, meaning I will need to compile it myself. The good news is…. its golang, I can compile it on anything, I already have the builds ready for push through Ansible.
Final thoughts being, Core systems and software are up, now the Infrastructure Services are being built out, and scale should start later this week by adding the first 6 module Pi cluster into the system to test out my automation scripts.