Sunday, September 3, 2017

Project Alphaberry - Part 6 - The hard parts

Can you believe that yet another few months have passed? A lot of what I planned on working on in Part 5 has been working out really well, but in this post I will be discussing what went well, what isn’t going well, and where the project is at today.

First, let’s get the bad out of the way, picking the operating systems. This turned out to be a complete nightmare that I didn’t intend on. I started simple, Rasbian is just Debian Jessie at its core, so why not use Debian Jessie on the Alpha? Well, Debian is like the operating system for power users I find out, its all kinds of minimal, and if you want any kinds of third party support for hardware, you gotta put it in the install media. That was fine for the networking devices, but when it came to the video card, my nightmares started. The Alienware Alpha R1 has an internal embedded Nvidia chip-set, this wouldn’t have been a huge issue, but it is the only video output on the system (it also has a video input, which was interesting). I could get Debian installed, but on the first boot into the OS, it would just hang on trying to load the display drivers. Research proved that Nvidia and Debian are not super friendly, and the built in drivers are pretty much horrible for the card, but I couldn’t get to a bootable point to actually install the Nvidia drivers. That is when I decided that running Debian on the Alpha was a non-option, I don’t want it to be a huge headache to get the OS built up from scratch, in the case where I want to rebuild the cluster.

That’s about the time I decided to go onto trying out Ubuntu. That worked flawlessly and quickly on the Alpha, I started with just Ubuntu server rather than Core, because I didn’t want to have to deal with creating a Canonical account just to install the OS. With Ubuntu on the Alpha, I figured hey, why not put Ubuntu Server on the PI’s? Dear lord, that was yet another rabbit hole that took me weeks to just give up on. There is a stable Ubuntu Server build for RPI2, but not RPI3, the RPI3 distro is community driven, and if you apt-get upgrade, it pretty much bricks the PI requiring that you rebuild the OS again. However, there was Ubuntu Mate for RPI3, and that worked without issues. I got all clever, and decided to take that Ubuntu Mate image, then strip everything “UI” out of the OS. That worked, and when completed, I had a working 32GB image that I could flash and repeat. However, it is 32GB image. I have yet to work with shrinking images, but I really just wanted something easier.

At this point, I just gave up, I don’t need the same OS on the Alpha and the PI, it isn’t necessary, and honestly, is a huge pain in the ass. I ended up deciding on using Ubuntu Server for the Alpha and HypriotOS for the PIs. HypriotOS was perfect, it is a super simple drop in image already pre-configured with Docker. They keep it pretty up to date with somewhat up to date Docker version (at this time they have images for 17.05). What I really appreciate is the tooling surrounding the OS. They have a really nice tool for burning images as well as a device-init system with some basics (hostname, wireless setup) that is kind of a bit like cloud-init, which I am somewhat familiar with. A simple yaml file in the /boot drive and you can easily set the hostname. Better than that, its open source and written in golang, and I am actually going to start doing some contributions to it, to add some additional features, such as setting some more settings on the OS and adding a way to join an existing docker swarm (via a swarm key/address combination).

While figuring out all of this, I have been dealing with the design for how all of these little credit card computers are going to be mounted and powered without being a huge mess of wires everywhere. I have a notebook where I have been doing scratch drawing of mounting plates and an enclosure for the system since the beginning of the project. Turns out, mounting and powering 24 Raspberry Pi’s, 5 network switches, a firewall, and essentially a fully fledged PC inside a little case, is kind of hard.

Starting with power. I didn’t want a powerstrip, and crap loads of plugs, I wanted a single power cable, that meant that I needed a power supply that was capable of driving this entire project. After a LOT of research, I found the Corsair RM650 The primary part of this power supply that sticks out above the rest, is the 25 amps on the 5v power. At load, with just the CPU/Networking/Memory in use on the PI’s, they draw about 700-800mA. The network switches (which are 5v) pull between 300-400mA under load, the gigabit network switch pulls about 300-400mA under heavy load, and the pfSense firewall pulls under 300mA under load. Doing some poor person math here, that’s less that 25A overall. It’s not perfect, but it will work. It also frees me up with about 54A on the 12V and another 25A on the 3.3V. That’s plenty of power for the project. One of the next issues was, the Alpha uses a 19V laptop style power supply, so I needed to get a power converter for 12V to 19V, that wasn’t super hard to find, but took about two weeks to ship from China.

Because the power is coming from the ATX power supply, I could utilize the GPIO pins instead of the Micro USB due to the clean power coming through. But now, I needed a way to break out the power from the 4 pin Molex connectors to the 2 pins on the PIs, but I didn’t want a single Molex for each Pi. I was able to find these little boards that take in a 4 pin Molex and have six 3 pin connectors, meant to be a Fan Splitter. One core flaw with the splitter though, is that it is intended to split the 12v power. With some minor soldering, I will just turn the Molex around on the board and have it split out the 5v power instead. That way I never accidentally try to send 12v into the board. I was able to find some 3 pin jumper cables at Fry’s that fit perfectly into the PCB and plugged into the Rasberry PI pins correctly. Powering the switches and firewall comes down to just getting some barrel connectors and putting a Molex connector on the other side. I still haven’t wired these up, but I may just use the 4 pin 5v connector out of the Powersupply for these. Finally, to turn the thing on and off, because ATX Power Supplies have a jumper to power on, I picked up an ATX On/Off 24 pin adapter.

While the power was a huge time sink, another part of the project is still an issue. Where to mount all this stuff, and how… This is where things seemed easy, but it became a horrible game of Tetris, especially with the way the PI’s are laid out on the PCB, still wanting to get access to the SD cards, and finally thinking about some sort of cooling for this cluster of servers (4 140mm Noctual PWM will be used in addition to heatsinks on all Pis). One of the biggest reasons I am not farther along on this project, is I don’t have anything to mount this stuff to yet to actually get the software portions working.

Let’s talk about my almost success, before I move on to what is currently going on with the case design. I got super excited about the Cooler Master HAF XB II EVO case, on paper it looked to be the perfect size for the project. It had the clearance for all the PI’s in the top part where the motherboard tray is, and there was a nice spot where the drive cages are underneath for the required 200x200mm Alpha, it seemed so perfect. But, after purchasing it, ripping the guts out, and starting some test fits, it did fit perfectly, if I didn’t need to actually use the ports on the servers. My core problem was, I forgot to take into account the insertion/removal of the motherboard tray, as well as where all my power and networking cables would go, this didn’t leave a lot of room, and it would have to be a bit more permanent than I wanted them to be. This case isn’t out of the question just yet, but I need to do some more real fittings before using it again.

The final, and most important part of this case, is how everything mounts internally. The nightmare of my life, and probably the first time in my life I have done any sort of mechanical design. I have some test designs (current in Photoshop, but going to convert to Illustrator soon) for a reversible tray for 3 PIs. They mount pretty nicely, and with some foam board prototyping, they seem to do the trick. Getting it to the non-prototype phase, to something I don’t mind mounting electronics to and powering, has been… frustrating. Early on I knew that the PIs were going to get mounted to 3mm acrylic, as that is a cheap, non-conductive surface, and it looks kind of cool. However, getting Acrylic cut, is a lot harder than it sounds, especially since the design isn’t super simple. After trying a few shops around my area, I was just about to give up, when I remembered, this is a DIY project, why not… DIY.

Watching TONS of YouTube videos on cutting acrylic, bending it, making boxes, etc… really just let me down the same path over and over, I need a controlled way to cut the damn stuff, and I just don’t have a way to do that cleanly. So as of this morning, I ordered the Inventables X-Carve CNC Machine. It has super easy to use web based software to control the machine, as well as really high reviews on the quality of the system. While it is a bit expensive ($1800 fully loaded), I think I will get my money back with side projects and even doing some cutting for some co-workers. I opted for the 1000mm version (the big one) because I knew I would want to make large things, they do have smaller versions as well. The CNC machine will arrive sometime next week, and I will be setting it up next weekend with the hopes of cutting out my first sets of trays for the PIs and start at least mounting them up for software testing.

And, that’s where we are, so many decisions made, so many plans complete, few designs left, and then the awesome part, the assembly and software installations can start. I’m heading out to Hashiconf in the middle of the month, and have been quite busy with a project at work that is wrapping up after a year, and then I want to spend some serious time finishing this project towards the end of the month, and into the early parts of next month.

There is another topic to discuss about this project, the Alienware Alpha (The Alpha in Alphaberry), but I am going to save that for another blog post.

23 comments:

Anonymous said...

Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this.AngularJS Training in Chennai | Best AngularJS Training Institute in Chennai| AngularJS Training in Velachery |
Angular4 Training Institute in Chennai

Robotic Process Automation Tutorial said...

Thank you so much for your information,its very useful and helpful to me.Keep updating and sharing. Thank you.
RPA training in chennai | UiPath training in chennai | rpa course in chennai | Best UiPath Training in chennai


DJ PRASATH said...

Thanks for your post. This is excellent information. The list of your blogs is very helpful for those who want to learn, It is amazing!!! You have been helping many application.
best selenium training in chennai | best selenium training institute in chennai selenium training in chennai | best selenium training in chennai | selenium training in Velachery

Rithi Rawat said...

Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

best training insitute for machine learning
best machine learning institutes in chennai
top institutes for machine learning in chennai
Android training in chennai
PMP training in chennai

kavinilavu G said...

Such a Great Article!! I learned something new from your blog. Amazing stuff. I would like to follow your blog frequently. Keep Rocking!!
Blue Prism training in chennai | Best Blue Prism Training Institute in Chennai

Roja Priya said...

Hi, Thanks a lot for your explanation which is really nice. I have read all your posts here. It is amazing!!!
Keeps the users interest in the website, and keep on sharing more, To know more about our service:
Please free to call us @ +91 9884412301 / 9600112302

Openstack course training in Chennai | best Openstack course in Chennai | best Openstack certification training in Chennai | Openstack certification course in Chennai | openstack training in chennai omr | openstack training in chennai velachery | openstack training in Chennai | openstack course fees in Chennai | openstack certification training in Chennai | best openstack training in Chennai | openstack certification in Chennai

Anonymous said...

Wow!! Really a nice Article. Thank you so much for your efforts. Definitely, it will be helpful for others. I would like to follow your blog. Share more like this. Thanks Again.
iot training in Chennai | Best iot Training Institute in Chennai

Priya Rajesh said...

Thanks for sharing this pretty post, it was good and helpful. Share more like this.
Spring Training in Chennai
Spring Hibernate Training in Chennai
Struts Training in Chennai
RPA Training in Chennai
AngularJS Training in Chennai
AWS course in Chennai
DevOps Certification Chennai
R Training in Chennai

vicky said...

An interesting discussion is worth comment. I think that you should write more on this topic, it might not be a taboo subject but generally people are not enough to speak on such topics. To the next. Cheers
super smash flash 2 download

Anonymous said...

Thanks for such a great article here. I was searching for something like this for quite a long time and at last, I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays.iot certification chennai | iot training courses in chennai | iot training institutes in chennai | industrial iot training chennai

Anbarasan14 said...

Happy to read your blog and gained a lot of new information in detail. Thanks for sharing.
IELTS Coaching in Chennai
IELTS Coaching Centre in Chennai
IELTS Training in Chennai
Best IELTS Coaching in Chennai
IELTS Classes in Mumbai
IELTS Coaching in Mumbai
Best IELTS Coaching in Mumbai
IELTS Center in Mumbai

Riya Raj said...

Wonderful Blog!!! Thanks for sharing with us...
German Classes in Bangalore
German Language Classes in Bangalore
German Classes in Coimbatore
German language classes in Coimbatore
German Classes in Madurai
PHP Course in Madurai
Spoken English Class in Madurai
Selenium Training in Coimbatore
SEO Training in Coimbatore
Web Designing Course in Madurai

Anonymous said...

Thanks for such a great article here. I was searching for something like this for quite a long time and at last, I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays.angularjs best training center in chennai | angularjs training in velachery | angularjs training in chennai | angularjs training in omr

divyapriya said...

Excellent blog for the people who needs information about this technology.
French Classes in Chennai
french classes
German Language Course in Chennai
IELTS Coaching centre in Chennai
Japanese Language Classes in Chennai
spanish classes in chennai
French Classes in Tambaram
French Classes in OMR

cynthiawilliams said...

I feel happy to see your webpage and looking forward for more updates.
Machine Learning course in Chennai
Machine Learning Training in Chennai
Data Science Course in Chennai
Data Science Classes in Chennai
R Programming Training in Chennai
Data Science Training in Velachery
Data Science Course in Anna Nagar

Asad said...

I was very pleased to find this web-site.I wanted to thanks for your time for this wonderful read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you blog post.
SetupabudhabiBusiness

IT Tutorials said...



Get the most advanced Hadoop Course by Professional expert. Just attend a FREE Demo session.
call us @ 9884412301 | 9600112302
Hadoop training in chennai | Hadoop training in velachery

infantroshow said...

very useful blog thanks for sharing this information
Selenium training in chennai | Selenium training in velachery

kishore said...


It is very beneficial information for me, I really impressed. This is very nice article and continues sharing with us...
Unix Training in Chennai
Unix Training
Excel Training in Chennai
Oracle DBA Training in Chennai
Tableau Training in Chennai
Job Openings in Chennai
Oracle Training in Chennai
Power BI Training in Chennai
Pega Training in Chennai
Advanced Excel Training in Chennai

jose said...

Really nice post. Thank you for sharing amazing information.
Java Training in Credo Systemz/Java Training in Chennai Credo Systemz/Java Training in Chennai/Java Training in Chennai with Placements/Java Training in Velachery/Java Training in OMR/Java Training Institute in Chennai/Java Training Center in Chennai/Java Training in Chennai fees/Best Java Training in Chennai/Best Java Training in Chennai with Placements/Best Java Training Institute in Chennai/Best Java Training Institute near me/Best Java Training in Velachery/Best Java Training in OMR/Best Java Training in India/Best Online Java Training in India/Best Java Training with Placement in Chennai

Adhuntt said...

Great blog thanks for sharing Looking for the best creative agency to fuel new brand ideas? Adhuntt Media is not just a digital marketing company in chennai. We specialize in revamping your brand identity to drive in best traffic that converts. Buckle up for a ride that is going to be filled with SEO, Social media marketing, unique Graphic & Logo design and efficient ads strike the perfect chord! Join the Adhuntt Media adventure right at Adhuntt Media
digital marketing company in chennai

Karuna said...

Nice blog thanks for sharing Tidy up your ambience by decorating them with amazing landscape plants in Chennai. Karuna Nursery Gardens is your portal to all the green that the world has to offer. Revolutionize your indoors, garden, terrace or office right now with us.
plant nursery in chennai

Pixies said...

Excellent blog thanks for sharing Run your salon business successfully by tying up with the best beauty shop in Chennai - The Pixies Beauty Shop. With tons of prestigious brands to choose from, and amazing offers we’ll have you amazed.
beauty Shop in Chennai