Juju Ubuntu

Finale: OpenStack Deployed on MAAS with Juju!

This is the final part 6 of my “Deploying OpenStack on MAAS 1.9+ with Juju” series, which is finally, after almost 1 year! \o/ In the last post – Advanced Networking: Deploying OpenStack on MAAS 1.9+ with Juju – I described the main concepts in Juju, and especially the recent networking enhancements, you could use today, with most recent stable release of Juju: 2.0.1.

Here the pieces finally come together, and at the end you should have your own, working private cloud, running the latest OpenStack (Newton), on 4 bare metal machines deployed by Juju via MAAS!

First, I’d like to thank all of my readers for all their support and patience! I know it took a long time, but I hope it will be all worth the wait 🙂

I know from experience a lot can go wrong installing a complicated software platform like OpenStack on such non-trivial networking setup. And when it does, it’s easy to start from scratch with Juju, but it still wastes time and bears frustration. I’d like to spare you that last bit by first going through a “pre-flight check” deployment. Using a simple Pinger charm I wrote for that purpose (although it’s generally useful as well). The charm does not follow the best practices about writing charms for Juju, but that’s intentionally so – to keep it simple. Pinger (source) contains the mandatory metadata, less than 150 lines of bash script, and a single config option. What it does is simple – runs ping(8) on a number of targets – configurable list (extra-targets setting), along with the private address and any explicitly bound extra-bindings (ep0..ep9)  specified at deployment time.

Those targets are checked on each unit, and over a peer relation units of the same pinger charm can exchange and check each other’s targets. The result of the check is conveniently indicated with “active / OK” or “blocked / FAIL” Juju status values. 

Test Deployment with Pinger(s) 

We’ll use a couple of scripts and YAML config files and follow roughly the same steps for the test deployment and the real OpenStack deployment.

Setting up Juju 2.0.1

You’ll need to add ppa:juju/stable to get 2.0.1 on trusty (or xenial):

Now we should add our MAAS as a cloud in Juju (more details here). Create a maas-hw-cloud.yaml (e.g. in $HOME) with the following content:

The endpoint must match the one you used for logging into the MAAS CLI, but without the /api/2.0/ suffix! (Juju adds that automatically). Get it by running maas list:

If should be using separate MAAS users for each Juju controller you bootstrap.

Now we need the a maas-hw-creds.yaml as well. Pick the “juju” user’s MAAS key, we previously created (you could use the “root” user, but it’s a bad practice) and change the highlighted maas-oauth with it:

Finally, run these two to configure Juju to use your MAAS:

We’re now ready to bootstrap and deploy, but first a few details about it. 

Bundle, Script and Configuration

In the beginning I mentioned we’ll be using a modified openstack-base bundle (its revision 48 in particular I’ve tested). However, we’ll also use a small bash script to drive the deployment. Because we have exactly 4 machines to work with, we cannot deploy the original bundle directly (we’ll need 5 machines – one for the Juju Controller). Therefore, the script runs juju bootstrap then adds more units to the Juju Controller machine after deploying a scaled-down bundle with 3 machines only. The same approach will be used for the test deployment, and we’ll also need a small YAML config (which can be the same for the real deployment later).

Here are the script and config files I used:

And the bundle itself is quite simple:

Since we have 3 storage/compute nodes and 1 network node, we can bootstrap to one of the storage/compute nodes (I’ve used --to node-22 in the script to do that), deploy the 3-node bundle, and add more units to the Juju controller node. We use 2 copies of the Pinger charm, because the network and storage/compute nodes have different endpoint bindings requirements (compute-external space vs storage-cluster, respectively). Also, some units are placed inside LXD containers to ensure they will meet the network connectivity requirements for the OpenStack bundle. Notice how we’re using spaces constraints on the machines to pick the correct ones (only node-11 has access to the compute-external space, while all the storage/compute nodes have access to the storage-cluster space). Pinger’s endpoints (ep0..ep5) are bound to the spaces OpenStack charms expects to be present. To verify end-to-end connectivity on the nodes once deployed by MAAS, we add the IPs configured on MAAS interfaces (ending in .1) to the extra-targets setting of each of the pingers. Finally, to ensure DNS resolution works and deployed nodes can reach the internet, google.com (or any similar public hostname) is also present in extra-targets.

In the script could omit the -m pinger-maas-hw:controller arguments, if we have no other Juju controller / model running (the switch command is sufficient to ensure we deploy the remaining 3 nodes on the controller model). The initial (and final) kill-controller commands are not strictly necessary, just make it easier to run the script multiple times and clean up before/after each run.

Deploying (Successfully)

Now for the easy part, run $ ./deploy-4-nodes-pinger.sh  and sit back and wait for about 15m. If all goes well, you’ll eventually get the juju status (we’re watching) output like this: 

Congrats if you got that far – you’re ready to deploy OpenStack with a reasonable expectation for it to succeed 🙂 When satisfied, hit Ctrl+C to interrupt the watch command, and answer “y” to kill the Juju controller and prepare for the OpenStack deployment.  

Troubleshooting (Failures)

If you see “FAIL (x of y targets unreachable)” on one or more units, something’s not correctly configured (i.e. YMMV – in any case OpenStack won’t work as well until the issues are resolved).

Like described on the Pinger charm page, check the unit, status, or ping logs of the failed unit(s):

A few tips for probable causes:

  • If you cannot resolve google.com, the MAAS DNS is not working properly (check /var/log/maas/maas.log and/or /var/log/syslog)
  • If google.com is resolvable, but unreachable your nodes cannot reach the internet (check if you have SNAT/MASQUERADE enabled in iptables for traffic coming from, but not destined to the range)
  • If one or more endpoints of the units (or the MAAS .1 IPs) are unreachable, you likely have misconfigured VLANs/ports on either MAAS or the switches.
  • If nodes in the same zone are reachable but across zones there are not, definitely check the switches’ ports/VLANs.

Deploying OpenStack Newton

OK, assuming all went fine so far, we only need to change the script slightly (see below) and use the modified openstack-base bundle with 3 nodes only.

Only notable change is the remove-unit / add-unit “dance” happening there. We want to use the bundle for declaring all the relations (as they are a lot), but we still want to end up with the same distribution of units and placement like with the original 4-node openstack-base bundle. We need to host the dashboard and keystone on the network-node (alongside the Juju controller), but they need to be also in the bundle to declare their relations, config, bindings, etc. I found it least painful to temporarily deploy openstack-dashboard/0 and keystone/0 on one of the storage/compute nodes, and then “move them” (remove and add a new unit of their application) to machine 0. The other few add-unit commands ensure we have the expected number of the units, matching the original bundle.

The bundle itself is quite a lot of lines (less than the original 4-node one, but still) to dump here I think. So I’ve added a link to it instead: http://paste.ubuntu.com/23424163/ It should be straightforward to compare it to the original and see the changes are mostly cosmetic, except the interesting ones describing endpoint bindings.

Show Time!

Like before, just run $ ./deploy-4-nodes.sh, sit back (a while longer this time – 30-45m) and watch OpenStack being stood up!

Once done, you should see output similar to the one below, and if you want to keep it do not answer “y” to the kill-controller question after hitting Ctrl+C! 🙂


Awesome! Now What?

You can follow the instructions in the openstack-base bundle README to get to the dashboard (Horizon), scale the nodes out, and verify it all works. There are a ton of resources online about OpenStack itself, but then again… You can also use Juju on top of your newly installed OpenStack to deploy workloads (you’d need to import some images first)!

Last but not least: have fun! 🙂 Experiment with your own in-house cloud, tweak it, tear it all or partially down and redeploy it (the process is the same). I might get to write some follow ups about some of these steps in the near future.

In Conclusion

Whew 🙂 we made it to the end! \o/ I hope it was useful and worth waiting for this final part (it almost didn’t happen TBH).

I’ll appreciate your comments and experiences, also try to help with issues you might encounter. Juju is more a free-time, side project for me now though (not a job), so it might take a while to respond.

All the best, thanks again, and good luck! 

Here are some convenient links to all articles in the series:

  1. Introduction
  2. Hardware Setup
  3. MAAS Setup
  4. Nodes Networking
  5. Advanced Networking
  6. Finale (this one)




Leave a Reply to Mathias Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by: Wordpress