Advanced Networking: Deploying OpenStack on MAAS 1.9+ with Juju

This is part 5 of my ongoing “Deploying OpenStack on MAAS 1.9+ with Juju” series. In the last post – Nodes Networking: Deploying OpenStack on MAAS 1.9+ with Juju – I described how to configure all 4 nodes in MAAS 1.9/2.0, so they have multiple network interfaces with addresses on all the subnets and VLANs OpenStack needs.

This post will be a little different, as we’ll focus on Juju, and how to take full advantage of the enhanced networking features Juju supports on MAAS 1.9+/2.0+.  Those features, however exciting, are quite new and still being properly documented, with few examples available. I’d like to think about this and the following post as (hopefully useful), tour of Juju’s enhanced networking with examples. We’ll start with an overview, then see how Juju 2.0 improves networking support for charmers and users. Returning to the original goal, deploying OpenStack, I can tell you from experience: a lot can go wrong due to subtly broken network connectivity. Therefore the user experience can be improved dramatically, if we have an easy way to verify whether our thus-far configured MAAS nodes, once deployed satisfy the connectivity requirements of OpenStack.

We’ll do that with an example charm I wrote, called ‘pinger‘. Pinger can verify the connectivity across all machines where its units are deployed to. In other words, by deploying pinger to all 4 MAAS nodes, letting it automatically verify the connectivity across all of them, and seeing a successful result, you can be reasonably certain the OpenStack deployment on the same MAAS will also work. Since the pinger charm is quite simple, it can also serve as an example for charmers on how to use Juju 2.0’s new networking features.

However, since the post got quite long I decided to describe the features first, and leave the actual deployment of OpenStack, and the pinger “pre-flight” check for the next (and final) post in the series.

Juju 2.0 Concepts (in less than 600 words)

I’ll try to explain the core concepts of Juju briefly, highlighting 2.0 specific bits in green. Feel free to skip or skim through if already familiar with those concepts. If you have no experience writing charms, I’d recommend reading the Developer Guide for all the details and hands-on tutorials.

Models, Charms, Applications, Units, and Machines

Juju models applications and their interactions. A Juju model (used to be called “environment” before Juju 2.0) contains everything managed by Juju, including machines, containers, applications, units, etc. Multiple models are can be created (and used like whiteboards) in 2.0, once a Juju controller machine is bootstrapped (controllers used to be called Juju “state servers” or “API servers”).

Charms are about deploying applications (used to be called “services” before Juju 2.0). Applications are what you care about – the actual workloads Juju can help you model, orchestrate, scale, and manage (e.g. Apache Hadoop cluster or OpenStack private cloud). Applications are comprised of one or more units, each one deployed from the same charm. Units are assigned to machines, which can be physical (like on MAAS), or virtual (in public clouds like AWS, GCE, Azure, Joyent, Rackspace, CloudSigma; or private clouds like OpenStack, vSphere, even locally with LXD). All of those machines can host containers (LXC, replaced by LXD in Juju 2.0), and KVM instances (on real hardware or cloud instances with HVM support).

Relations

Applications rarely work on their own, even if they appear self-contained, like a PostgreSQL database cluster. Juju models reusable, well defined “protocols” (or interfaces) for applications to interoperate using named “channels”, called relations. Charms can define 3 types of relations in their metadata: “provides”, “requires”, or “peers”. Any number of those relations can be defined by a single charm. “Provided” relations model the “server-to-client” protocol, while “required” relations model the reverse – “client-to-server”. Peer relations are a combination of both, but within the same application, a “peer-to-peer” protocol. For example, the mysql application can be related – at the same time – to both mediawiki and wordpress, because mysql provides a relation endpoint called “db:mysql”, matching the relation endpoint “db:mysql” required by both mediawiki and wordpress (“db” is the relation name, “mysql” is the supported interface, or “protocol”, the combination of both is a relation endpoint, which represents one side(end) of an established provider-requirer relation or the only side of a peer relation).

Hooks

Hooks are executables (scripts or binaries) in the “hooks” directory of a charm, following the naming scheme defined by Juju (e.g. “install”, “config-changed”, “peer-relation-joined”, “db-relation-changed”). Hooks are called by Juju as events relevant to the charm happen, and allow the charm to respond to those events by performing application-specific tasks. Juju executes hooks with a few useful environment variables set (e.g. $JUJU_UNIT_NAME) and provides a number of hook tools the charm can use (e.g. unit-get, relation-set, open-port, etc.). Hooks are “transactional” by design: nothing a hook does is actually “committed” unless the hook succeeds (returns zero exit code). Hooks also need to be idempotent, as failed hooks are automatically retried by Juju, or can be re-run manually by the user when a unit is error state due to a failed hook.

In the earlier example, when mediawiki is related to mysql, “db-relation-joined”, then “db-relation-changed” are triggered on all units of both applications. MySQL responds by creating a database, username, and password and setting those as relation settings using relation-set. This triggers a “db-relation-changed” hook where mediawiki reads the relation settings with relation-get, and configures its database connection to MySQL accordingly.

Charms and Networking: until Juju 1.25

Charms can ask Juju to open TCP/UDP ports (or port ranges) on machines hosting units of the application. Since different units can be co-located on the same machine, Juju tracks all open ports on each machine to detect port conflicts before they happen, and notifies the charm to give it a chance to try a different port.

Each machine, and by inference any units hosted on that machine, always have a “private-address” local setting (accessible with unit-get private-address) provided by Juju. Any remote unit in a relation also has a “private-address” relation setting  (accessible with relation-get private-address [remote-unit]; remote unit name is optional when relation-get is run within a relation hook, like db-relation-changed), also always available and set by Juju.

Public addresses are used to advertise publicly accessible application endpoints, comprised of the public address and an open port, to external clients. For the application to be accessible from outside the Juju model, it needs to be “exposed“. Exposing an application (e.g. juju expose wordpress) instructs Juju to do what’s needed to allow external access to the application. Unlike “private-address”, there is no “public-address” setting populated by Juju. Usually, a charm uses e.g. open-port 8080/tcp and leaves it to the user to expose the application, then (somehow) notify external clients of the public address and port(s) (visible in the juju status output for the unit) the application is accessible at.

Charms and Networking: since Juju 2.0

Historically, every machine (and the units it hosts) is assumed to be have a single IPv4 address (or hostname) considered “private”, and the same or different “public” address (likewise, a single IPv4 address or hostname).

A single private/public address might be a simple and convenient concept, but also quite inadequate for applications that need anything more sophisticated Also, IPv6 is coming sooner than you might think.

Juju 2.0 introduces a number of new concepts related to networking: spaces, subnets, spaces constraints, endpoint bindings, and a new hook tool: network-get.

We have covered the important aspects of spaces and subnets in previous posts in the series: spaces contain one or more subnets with the same security and traffic segmentation/isolation concerns.

Spaces constraints, like all Juju constraints, can be used to select a suitable machine for hosting a unit of an application with specific connectivity requirements. Like all constraints, spaces constraints can be changed at any time by the user (using juju set-constraints for applications or juju set-model-constraints for the whole model). Changing them does not affect existing machines or units, only newly deployed ones (see below why this is significant). Currently, spaces constraints are supported fully on MAAS 1.9/2.0, and partially on AWS.

Example: juju deploy cs:xenial/percona-cluster mysql --constraints 'arch=amd64 spaces=internal,admin,^storage'

On MAAS this will select amd64 machines connected to the “internal” and “admin” MAAS spaces, but not connected to the “storage” space. On AWS, machines connected to the “internal” space will be selected, ignoring the remaining entries in the list.

Endpoint Bindings and network-get

By far, the one of the most important new networking features in 2.0 is the introduction of endpoint bindings – distinct application endpoints, which can be “bound” (connected) to user-specified spaces, at deployment time. 

What is an (application) endpoint? NOTE: Not to be confused with a relation endpoint – the pair of relation name and interface (“db:mysql”), representing one side of a relation (with all participating units).

Deploying each unit of an application with endpoint “foo” bound to space “bar” guarantees all machines hosting those units will have a dedicated address (not necessarily on a dedicated network interface though) from one of the subnets in space “bar”. The address is accessible to the unit under the name “foo” (using network-get, see below), at any and all times: soon after the host machine is started, for the entire lifetime of the unit, and likely until the machine is removed.

How can a charm define named endpoints? Two ways:

1. All existing charm relation names (in “requires”, “provides”, “peers”) can be used as named endpoints to –bind (see below).
2. Using the new extra-bindings section in the charm metadata.yaml (more on this further down).

Example: juju deploy cs:xenial/percona-cluster mysql --bind 'shared-db=internal-api public-api'

On MAAS this will select a machine with at least 2 network interfaces, first one on the “internal-api” space, the other on the “public-api” space. It also means, the charm can internally use the new network-get hook tool to get the “internal-api” address it has on for the “shared-db” binding, and any other binding will return the “public-api” address (not specifying ‘<binding>=’ before the space name means default space, used for any not explicitly bound endpoints).

Example (in a charm hook): network-get shared-db --primary-address

The above will return e.g. 10.100.0.111 (assuming, like described in previous posts, subnet 10.100.0.0/20 is in space “internal-api”). NOTE: --primary-address is currently required. 

Example (in a charm hook): network-get cluster --primary-address

Since the “cluster” endpoint was not explicitly bound, but there is a default space specified (“public-api”), 10.50.0.111 will be returned (again, assuming subnet 10.50.0.0/20 is in the “public-api” space).

What happens if we used --bind 'shared-db=internal-api' only, and try the above? When no default space is specified, any unbound endpoint will use the legacy “private-address” value. In other words, network-get cluster --primary-address and unit-get private-address will return the same address. This is convenient, works on all clouds, not just MAAS (private-address is supported everywhere), but most importantly: new charms should use network-get <binding> --primary-address instead of unit-get private-address (<binding> is an existing relation name or extra-binding name).

Why bother with the longer syntax? Network-get will evolve further in the upcoming releases, while staying backwards-compatible (e.g. currently requiring –primary-address offers a drop-in replacement for unit-get private-address). The no-arguments behavior will return all relevant addresses, with arguments to support filtering  (including by type – IPv4/IPv6), and more details (like network interfaces info).

Like with all hook tools, you can run juju help-tool network-get to get information about the hook tool command, including its arguments and examples. The usual --format=smart|yaml|json and -o|--output=file arguments are also accepted by network-get.

extra-bindings

Imagine you’re writing a charm for an application, which needs two separate network connections (one for admin traffic, another for client traffic). Those connections are only used by the clients of that application, and not by the application itself.

One way to model that is to define 2 peers relations (e.g. “admin” and “client”), and bind them as needed. Why peer relations? Because they created automatically by Juju, at deployment time, unlike the provided/required relations, which are only created when a user adds the relation between them (juju add-relation app1:rel1 app2:rel2). Peer relations are useful for modeling internal application workflows, like clustering, load-balancing, replication.Simply adding a named  entry to the “peers:” section of the charm’s metadata.yaml is enough to be able to bind that entry by name (no need to implement any hooks). An important limitation of peer relations is that once defined, they cannot be removed by upgrading to newer versions of the same charm (using juju upgrade-charm). See LP bug #1386926 for details. In a future Juju release this will likely change, but until it does – bear in mind each peer relation you add is there to stay for the lifetime of your charm. So 2 peer relations are excellent if you really need 2 separate peer-to-peer “protocols” each going over a dedicated connection with a known address.

There is another way to model those 2 connections, especially in cases where we just need to be reachable on 2 separate addresses (by our clients). In Juju 2.0, a new top-level section can be added to the charm’s metadata.yaml: extra-bindings. It contains a simple list of names, which can be used as bindings with juju deploy .. --bind ‘<binding>=<space>… ‘, in addition to existing charm relation names.

Example (excerpt from cs:xenial/keystone’s metadata.yaml):

name: keystone
summary: OpenStack identity service
# ... more lines below omitted ...
tags:
  - openstack
  - identity
  - misc
extra-bindings:
  public:
  admin:
  internal:
# ... yet more lines ...

That’s all! Adding that extra-bindings section now allows us to deploy this charm like this:

juju deploy cs:xenial/keystone --bind 'public=public-api admin=admin-api internal=internal-api'

In case you’re wondering why not use a simple list like: extra-bindings: ["public", "admin", "internal"], because in future releases of Juju, additional properties will be added to each binding entry (e.g. the ability to request a dedicated Layer-2 network interface, so you can deploy Neutron and run a DHCP server on it).

Endpoint Bindings and Bundles

When using a bundle to deploy multiple related applications, you can add a “bindings:” subsection for each application (a.k.a. service – NOTE, the naming transition from “service” to “application” in Juju 2.0 is still in progress, and for compatibility with existing bundles, both “services:” and “applications:” sections are accepted, but treated the same).

Example (modified mediawiki bundle – added bindings are highlighted):

services:
  haproxy:
    charm: cs:trusty/haproxy-13
    num_units: 1
    bindings:
        website: public
        reverseproxy: internal
        peer: internal
    expose: true
  mediawiki:
    charm: cs:trusty/mediawiki-3
    num_units: 1
    bindings:
      cache: internal
      website: internal
      db: db
    options:
      name: MAAS + Juju Network Spaces Wiki
  mysql:
    charm: cs:trusty/mysql-29
    num_units: 1
    bindings:
      db: db
      cluster: internal
series: trusty
relations:
- - haproxy:reverseproxy
  - mediawiki:website
- - mediawiki:db
  - mysql:db

Deploying this bundle is as simple as:

juju deploy ~/bundles/mediawiki-single.yaml

Roughly equivalent to the following individual commands:

juju deploy cs:trusty/haproxy-13 --bind 'internal website=public'
juju deploy cs:trusty/mediawiki-3 --bind 'db=db internal' --config name='MAAS...'
juju deploy cs:trusty/mysql-29 --bind 'db=db cluster=internal'
juju add-relation haproxy:reverseproxy mediawiki:website
juju add-relation mysql mediawiki:db
juju expose haproxy

When deploying a bundle, Juju will rollback all changes, if the constrains, endpoint bindings, or storage requirements for any unit cannot be fulfilled. 

Unfortunately this is one of the few examples, as the feature is quite new. More information, albeit now a bit outdated, can be found on our Juju Sapphire (a.k.a. “the networking guys” 🙂 ) Team’s site here.

Fortunately, the next post will demonstrate how easy it is to deploy OpenStack from a bundle, using all the nice new features we learned today! 🙂

Finale: OpenStack Deployed on MAAS with Juju!

Thanks for your support, and I hope you’ll like the last chapter even more!

Convenient links to all articles in the series:

1. Introduction
2. Hardware Setup
3. MAAS Setup
4. Nodes Networking
5. Advanced Networking (this one)
6. Finale