Advanced Networking: Deploying OpenStack on MAAS 1.9+ with Juju
This is part 5 of my ongoing “Deploying OpenStack on MAAS 1.9+ with Juju” series. In the last post – Nodes Networking: Deploying OpenStack on MAAS 1.9+ with Juju – I described how to configure all 4 nodes in MAAS 1.9/2.0, so they have multiple network interfaces with addresses on all the subnets and VLANs OpenStack needs.
This post will be a little different, as we’ll focus on Juju, and how to take full advantage of the enhanced networking features Juju supports on MAAS 1.9+/2.0+. Those features, however exciting, are quite new and still being properly documented, with few examples available. I’d like to think about this and the following post as (hopefully useful), tour of Juju’s enhanced networking with examples. We’ll start with an overview, then see how Juju 2.0 improves networking support for charmers and users. Returning to the original goal, deploying OpenStack, I can tell you from experience: a lot can go wrong due to subtly broken network connectivity. Therefore the user experience can be improved dramatically, if we have an easy way to verify whether our thus-far configured MAAS nodes, once deployed satisfy the connectivity requirements of OpenStack.
We’ll do that with an example charm I wrote, called ‘pinger‘. Pinger can verify the connectivity across all machines where its units are deployed to. In other words, by deploying pinger to all 4 MAAS nodes, letting it automatically verify the connectivity across all of them, and seeing a successful result, you can be reasonably certain the OpenStack deployment on the same MAAS will also work. Since the pinger charm is quite simple, it can also serve as an example for charmers on how to use Juju 2.0’s new networking features.
However, since the post got quite long I decided to describe the features first, and leave the actual deployment of OpenStack, and the pinger “pre-flight” check for the next (and final) post in the series.
Juju 2.0 Concepts (in less than 600 words)
I’ll try to explain the core concepts of Juju briefly, highlighting 2.0 specific bits in green. Feel free to skip or skim through if already familiar with those concepts. If you have no experience writing charms, I’d recommend reading the Developer Guide for all the details and hands-on tutorials.
Models, Charms, Applications, Units, and Machines
Juju models applications and their interactions. A Juju model (used to be called “environment” before Juju 2.0) contains everything managed by Juju, including machines, containers, applications, units, etc. Multiple models are can be created (and used like whiteboards) in 2.0, once a Juju controller machine is bootstrapped (controllers used to be called Juju “state servers” or “API servers”).
Charms are about deploying applications (used to be called “services” before Juju 2.0). Applications are what you care about – the actual workloads Juju can help you model, orchestrate, scale, and manage (e.g. Apache Hadoop cluster or OpenStack private cloud). Applications are comprised of one or more units, each one deployed from the same charm. Units are assigned to machines, which can be physical (like on MAAS), or virtual (in public clouds like AWS, GCE, Azure, Joyent, Rackspace, CloudSigma; or private clouds like OpenStack, vSphere, even locally with LXD). All of those machines can host containers (LXC, replaced by LXD in Juju 2.0), and KVM instances (on real hardware or cloud instances with HVM support).
Applications rarely work on their own, even if they appear self-contained, like a PostgreSQL database cluster. Juju models reusable, well defined “protocols” (or interfaces) for applications to interoperate using named “channels”, called relations. Charms can define 3 types of relations in their metadata: “provides”, “requires”, or “peers”. Any number of those relations can be defined by a single charm. “Provided” relations model the “server-to-client” protocol, while “required” relations model the reverse – “client-to-server”. Peer relations are a combination of both, but within the same application, a “peer-to-peer” protocol. For example, the mysql application can be related – at the same time – to both mediawiki and wordpress, because mysql provides a relation endpoint called “db:mysql”, matching the relation endpoint “db:mysql” required by both mediawiki and wordpress (“db” is the relation name, “mysql” is the supported interface, or “protocol”, the combination of both is a relation endpoint, which represents one side(end) of an established provider-requirer relation or the only side of a peer relation).
Hooks are executables (scripts or binaries) in the “hooks” directory of a charm, following the naming scheme defined by Juju (e.g. “install”, “config-changed”, “peer-relation-joined”, “db-relation-changed”). Hooks are called by Juju as events relevant to the charm happen, and allow the charm to respond to those events by performing application-specific tasks. Juju executes hooks with a few useful environment variables set (e.g. $JUJU_UNIT_NAME) and provides a number of hook tools the charm can use (e.g. unit-get, relation-set, open-port, etc.). Hooks are “transactional” by design: nothing a hook does is actually “committed” unless the hook succeeds (returns zero exit code). Hooks also need to be idempotent, as failed hooks are automatically retried by Juju, or can be re-run manually by the user when a unit is error state due to a failed hook.
In the earlier example, when mediawiki is related to mysql, “db-relation-joined”, then “db-relation-changed” are triggered on all units of both applications. MySQL responds by creating a database, username, and password and setting those as relation settings using
relation-set. This triggers a “db-relation-changed” hook where mediawiki reads the relation settings with
relation-get, and configures its database connection to MySQL accordingly.
Charms and Networking: until Juju 1.25
Charms can ask Juju to open TCP/UDP ports (or port ranges) on machines hosting units of the application. Since different units can be co-located on the same machine, Juju tracks all open ports on each machine to detect port conflicts before they happen, and notifies the charm to give it a chance to try a different port.
Each machine, and by inference any units hosted on that machine, always have a “private-address” local setting (accessible with
unit-get private-address) provided by Juju. Any remote unit in a relation also has a “private-address” relation setting (accessible with
relation-get private-address [remote-unit]; remote unit name is optional when relation-get is run within a relation hook, like db-relation-changed), also always available and set by Juju.
Public addresses are used to advertise publicly accessible application endpoints, comprised of the public address and an open port, to external clients. For the application to be accessible from outside the Juju model, it needs to be “exposed“. Exposing an application (e.g.
juju expose wordpress) instructs Juju to do what’s needed to allow external access to the application. Unlike “private-address”, there is no “public-address” setting populated by Juju. Usually, a charm uses e.g.
open-port 8080/tcp and leaves it to the user to expose the application, then (somehow) notify external clients of the public address and port(s) (visible in the
juju status output for the unit) the application is accessible at.
Charms and Networking: since Juju 2.0
Historically, every machine (and the units it hosts) is assumed to be have a single IPv4 address (or hostname) considered “private”, and the same or different “public” address (likewise, a single IPv4 address or hostname).
A single private/public address might be a simple and convenient concept, but also quite inadequate for applications that need anything more sophisticated Also, IPv6 is coming sooner than you might think.
Juju 2.0 introduces a number of new concepts related to networking: spaces, subnets, spaces constraints, endpoint bindings, and a new hook tool: network-get.
We have covered the important aspects of spaces and subnets in previous posts in the series: spaces contain one or more subnets with the same security and traffic segmentation/isolation concerns.
Spaces constraints, like all Juju constraints, can be used to select a suitable machine for hosting a unit of an application with specific connectivity requirements. Like all constraints, spaces constraints can be changed at any time by the user (using
juju set-constraints for applications or
juju set-model-constraints for the whole model). Changing them does not affect existing machines or units, only newly deployed ones (see below why this is significant). Currently, spaces constraints are supported fully on MAAS 1.9/2.0, and partially on AWS.
juju deploy cs:xenial/percona-cluster mysql --constraints 'arch=amd64 spaces=internal,admin,^storage'
On MAAS this will select amd64 machines connected to the “internal” and “admin” MAAS spaces, but not connected to the “storage” space. On AWS, machines connected to the “internal” space will be selected, ignoring the remaining entries in the list.
Endpoint Bindings and network-get
By far, the one of the most important new networking features in 2.0 is the introduction of endpoint bindings – distinct application endpoints, which can be “bound” (connected) to user-specified spaces, at deployment time.
What is an (application) endpoint? NOTE: Not to be confused with a relation endpoint – the pair of relation name and interface (“db:mysql”), representing one side of a relation (with all participating units).
Deploying each unit of an application with endpoint “foo” bound to space “bar” guarantees all machines hosting those units will have a dedicated address (not necessarily on a dedicated network interface though) from one of the subnets in space “bar”. The address is accessible to the unit under the name “foo” (using network-get, see below), at any and all times: soon after the host machine is started, for the entire lifetime of the unit, and likely until the machine is removed.
How can a charm define named endpoints? Two ways:
- All existing charm relation names (in “requires”, “provides”, “peers”) can be used as named endpoints to –bind (see below).
- Using the new extra-bindings section in the charm metadata.yaml (more on this further down).
juju deploy cs:xenial/percona-cluster mysql --bind 'shared-db=internal-api public-api'
On MAAS this will select a machine with at least 2 network interfaces, first one on the “internal-api” space, the other on the “public-api” space. It also means, the charm can internally use the new network-get hook tool to get the “internal-api” address it has on for the “shared-db” binding, and any other binding will return the “public-api” address (not specifying ‘<binding>=’ before the space name means default space, used for any not explicitly bound endpoints).
Example (in a charm hook):
network-get shared-db --primary-address
The above will return e.g. 10.100.0.111 (assuming, like described in previous posts, subnet 10.100.0.0/20 is in space “internal-api”). NOTE:
--primary-address is currently required.
Example (in a charm hook):
network-get cluster --primary-address
Since the “cluster” endpoint was not explicitly bound, but there is a default space specified (“public-api”), 10.50.0.111 will be returned (again, assuming subnet 10.50.0.0/20 is in the “public-api” space).
What happens if we used
--bind 'shared-db=internal-api' only, and try the above? When no default space is specified, any unbound endpoint will use the legacy “private-address” value. In other words,
network-get cluster --primary-address and
unit-get private-address will return the same address. This is convenient, works on all clouds, not just MAAS (private-address is supported everywhere), but most importantly: new charms should use
network-get <binding> --primary-address instead of
unit-get private-address (<binding> is an existing relation name or extra-binding name).
Why bother with the longer syntax? Network-get will evolve further in the upcoming releases, while staying backwards-compatible (e.g. currently requiring –primary-address offers a drop-in replacement for
unit-get private-address). The no-arguments behavior will return all relevant addresses, with arguments to support filtering (including by type – IPv4/IPv6), and more details (like network interfaces info).
Like with all hook tools, you can run
juju help-tool network-get to get information about the hook tool command, including its arguments and examples. The usual
-o|--output=file arguments are also accepted by network-get.
Imagine you’re writing a charm for an application, which needs two separate network connections (one for admin traffic, another for client traffic). Those connections are only used by the clients of that application, and not by the application itself.
One way to model that is to define 2 peers relations (e.g. “admin” and “client”), and bind them as needed. Why peer relations? Because they created automatically by Juju, at deployment time, unlike the provided/required relations, which are only created when a user adds the relation between them (
juju add-relation app1:rel1 app2:rel2). Peer relations are useful for modeling internal application workflows, like clustering, load-balancing, replication.Simply adding a named entry to the “peers:” section of the charm’s metadata.yaml is enough to be able to bind that entry by name (no need to implement any hooks). An important limitation of peer relations is that once defined, they cannot be removed by upgrading to newer versions of the same charm (using
juju upgrade-charm). See LP bug #1386926 for details. In a future Juju release this will likely change, but until it does – bear in mind each peer relation you add is there to stay for the lifetime of your charm. So 2 peer relations are excellent if you really need 2 separate peer-to-peer “protocols” each going over a dedicated connection with a known address.
There is another way to model those 2 connections, especially in cases where we just need to be reachable on 2 separate addresses (by our clients). In Juju 2.0, a new top-level section can be added to the charm’s metadata.yaml: extra-bindings. It contains a simple list of names, which can be used as bindings with
juju deploy .. --bind ‘<binding>=<space>… ‘, in addition to existing charm relation names.
Example (excerpt from cs:xenial/keystone’s metadata.yaml):
name: keystone summary: OpenStack identity service # ... more lines below omitted ... tags: - openstack - identity - misc extra-bindings: public: admin: internal: # ... yet more lines ...
That’s all! Adding that extra-bindings section now allows us to deploy this charm like this:
juju deploy cs:xenial/keystone --bind 'public=public-api admin=admin-api internal=internal-api'
In case you’re wondering why not use a simple list like:
extra-bindings: ["public", "admin", "internal"], because in future releases of Juju, additional properties will be added to each binding entry (e.g. the ability to request a dedicated Layer-2 network interface, so you can deploy Neutron and run a DHCP server on it).
Endpoint Bindings and Bundles
When using a bundle to deploy multiple related applications, you can add a “bindings:” subsection for each application (a.k.a. service – NOTE, the naming transition from “service” to “application” in Juju 2.0 is still in progress, and for compatibility with existing bundles, both “services:” and “applications:” sections are accepted, but treated the same).
Example (modified mediawiki bundle – added bindings are highlighted):
services: haproxy: charm: cs:trusty/haproxy-13 num_units: 1 bindings: website: public reverseproxy: internal peer: internal expose: true mediawiki: charm: cs:trusty/mediawiki-3 num_units: 1 bindings: cache: internal website: internal db: db options: name: MAAS + Juju Network Spaces Wiki mysql: charm: cs:trusty/mysql-29 num_units: 1 bindings: db: db cluster: internal series: trusty relations: - - haproxy:reverseproxy - mediawiki:website - - mediawiki:db - mysql:db
Deploying this bundle is as simple as:
juju deploy ~/bundles/mediawiki-single.yaml
Roughly equivalent to the following individual commands:
juju deploy cs:trusty/haproxy-13 --bind 'internal website=public' juju deploy cs:trusty/mediawiki-3 --bind 'db=db internal' --config name='MAAS...' juju deploy cs:trusty/mysql-29 --bind 'db=db cluster=internal' juju add-relation haproxy:reverseproxy mediawiki:website juju add-relation mysql mediawiki:db juju expose haproxy
When deploying a bundle, Juju will rollback all changes, if the constrains, endpoint bindings, or storage requirements for any unit cannot be fulfilled.
Unfortunately this is one of the few examples, as the feature is quite new. More information, albeit now a bit outdated, can be found on our Juju Sapphire (a.k.a. “the networking guys” 🙂 ) Team’s site here.
Fortunately, the next post will demonstrate how easy it is to deploy OpenStack from a bundle, using all the nice new features we learned today! 🙂
Finale: OpenStack Deployed on MAAS with Juju!
Thanks for your support, and I hope you’ll like the last chapter even more!
Convenient links to all articles in the series:
- Hardware Setup
- MAAS Setup
- Nodes Networking
- Advanced Networking (this one)
How are things going? I’m awaiting impatiently to see your charms. 🙂
I’m sorry, as it turned out I’m travelling most of July, and couldn’t find time to finish the final post in the series. I’ll try to do in before July, 25 – if I can, if not it will be during the week of July, 25 when I’m back home.
Thanks for following so far, please don’t despair 🙂
Great. Fingers crossed and thanks for the great posts so far!
I just want to join in as well – I really appreciate the posts thus far! And with special emphasis on the additional notes about MAAS 2.0. So much of the advice and screenshots about MAAS just don’t *quite* work or make sense. To a newcomer like me, it’s hard to make sense of the otherwise minor jargon shifts and the like. This blog series is a sort of Rosetta Stone in helping me get past that.
And I’m also eagerly awaiting the Juju post. I hadn’t realized how useful it would be to have all those VPNs, and from the bit I’ve been able to get running, I’m excited to be jumping on this bandwagon!
Regarding networking setup. I got MAAS 1.9.3 running and have deployed Openestack. Got VM’s running and all that seems to be working. However i have decided that i would like to now route another IP range to the servers. Then added a network and a router. So here is a question or two, how do i add additional VLAN to the setup and doing it without having to rebuild a whole environment? Second question is, after i added a VLAN i realised that the ports for a public network where nodes are plugged in are not trunk ports. I changed that and added multiple VLAN’s but obviously since i have VLAN tagging now i cannot reach any of the perviously reachable VM instance’s. Is there a way to setup VLAN tagging in MAAS to the nodes? And going back to a previous question what if in 6 months i want to add another VLAN with another different IP range? Fogot to mentation i used autopilot to add Openstack services.
Thank you. Looking forward to the last one. Any ETA on Part 6?
has anybody here had a problem while generating ssh key? The BUI gives me the following error output:
“Invalid SSH public key: ‘PosixPath’ object has no attribute ‘path'”
It may be worth updating MAAS. I remember there was a bug related to this (I think it’s this: https://bugs.launchpad.net/maas/+bug/1604128 ), so you should be able to update to it. I think I applied the patch by hand, just to get going, though.
I am using maas 2.0 and ubuntu server 16.04. All the packages seem to be updated and upgraded, but i am still unable to generate the ssh key. How did you applied patch by hand?
It seem that MAAS is updated and upgraded (i’m currently on version 2.0), but i am still unable to copy the generated key into the web dashboard. How did you applied the patch by hand? You had the same problem?
You might not be using ` ppa:maas/next ` – I’d recommend it for bleeding edge updates (and MAAS 2.0 is a bit new).
In this case, the bug is conveniently run from a Python script. So you can just open the file in `nano` or the like and apply the diff by hand. You can find the diff by going to the bug tracker link, scrolling down until you see “Related branches” and chosing a diff that has been merged into the most recent release (here it’s r5186). Click on the diff link and apply the changes needed in the non-test files (or all, if you’re industrious). You’ll see the change is very minor: change anywhere `keypath.path` is written to `str(keypath)`. Enjoy!
Thanks, changing to maas/next and rebooting the system worked for me. Thanks a lot.
Hello ! Is it possible that you are going to finish your posts soon ? Your posts describe my goals quite well, but I’d like to see how you advise to deploy Openstack with your network topology. Do you use autopilot or just juju ?
This is a great series of posts. I have completed eveything in a vm environment and I am also waiting for your last post.
Hi! waiting for your next post.. :))
Thank you! Any ETA on Part 6?
Also waiting for part 6 in the series, has been a good primer/intro so far. Looking forward to full deployment…
To all of you, who are asking when will the last part of the series be published, let me try to explain the delay.
Juju Core team (myself included) are working hard to prepare the long-awaited Juju 2.0.0 final release. Since a lot of the ways Juju 2.0 works on MAAS were changed (and significantly improved) over the last few months, I didn’t want to rush with the last article while Juju 2.0 is still not quite ready for “prime time”. We’ve released 2.0rc2 this week, and 2.0.0 is scheduled for release in the beginning of October, likely along with Ubuntu 16.10. Once Juju 2.0 it’s out (or even on the way to be), I’ll update the draft I have and post the last article.
Once again, I apologize for the long delay :/ I’m sure it will be worth it though, and thanks again for your interest (and kind words – esp. a couple of weeks ago in Pasadena, CA where I met some great folks @ 3rd Juju Charmers Summit)!