Commissioning deployed machines: Request granted

We want to make it possible to deploy MAAS in an existing datacenter, and have it keep track of machines that already have a deployed workload — without disturbing machine or workload. Currently, in order to get a machine into MAAS, with correct hardware information, you have to network boot the machine and let MAAS inspect the machine in an ephemeral environment.

This isn’t feasible if a machine is already running a workload — but there’s still value in having the machine in MAAS:

• It helps keep track of all of your machines,
• and it will allow you to easily re-use the machine, when you don’t need to run the workload there anymore.

We’re working on some solutions; though they may not all get fully implemented right away (or ever), we do plan to provide some level of deployed machine enlistment with the release of MAAS 3.1. We’d like to share our thinking. We welcome your feedback, so please comment directly in reply to this blog.

How we’ve scoped the problem

Thinking like MAAS users, based on a lot of discourse feedback, we made some assumptions to help us scope this problem.

Users want to add deployed machines to MAAS

MAAS users have expressed a desire to add already-deployed machines to MAAS, so that they can use MAAS to keep track of existing workloads If a user has an already-deployed machine that they want to see in MAAS, they can create the machine in MAAS as normal, but set the initial status to Deployed, to prevent MAAS from rebooting and/or reloading it.

We thought about this problem, and decided that this kind of a feature should probably play out like this:

• The user shouldn’t have to specify a power driver.
• The user should be able to add a machine without specifying a MAC address, if they choose Unknown or IPMI as the power driver
• If the machine has an Unknown power driver and no MAC, the user should be expected to provide a MAC when setting the power driver, unless, of course, it’s IPMI.
• When a machine is released, it probably has to go back to the New state, since it doesn’t have any commissioning data, and, by definition, the user is not expecting the machine to continue running its current workload.
• If the machine doesn’t have a MAC, any operation (apart from commissioning) that requires a netboot can’t be allowed.
• After adding the machine as Deployed, it should be visible in the machine listing.

Users want updated H/W info for deployed machines

Users most likely want to update hardware information for deployed machines to MAAS — without rebooting the machine — so that they can use MAAS to keep track of running workloads. If a user has machines in MAAS that are deployed, they should be able to update the hardware information, without having to reboot the machine.

Here’s how we envisioned this feature coming together:

• When the user updates the hardware information, it should be visible in MAAS, as if it were commissioned normally.
• When the machine is released, it should go back into the New state, so that commissioning can be run; that is, it should become just another, normal MAAS machine at that point.

Users want to get H/W info from deployed machines

We assumed it’s okay for users to use SSH to connect to a machine, and then run a script that will (1) add the machine to MAAS, and (2) update the commissioning data, without disturbing the machine’s workload. This builds on the last assumption, but like our enlistment scripts, it will register the machine in MAAS as a deployed machine.

Some ideas about how we wanted to implement this feature:

• The user should be able to download a script from MAAS and run it on the deployed machine.
• This script should add the deployed machine to MAAS — as Deployed — and gather commissioning data without disturbing the machine’s workload.
• If the script is run again, the machine won’t be registered again, but the existing machine will have its commissioning data updated.
• The current hostname of the machine is used to name the machine when initially added.
• If the hostname of the machine changes and the script is run, it’s okay for the name of the existing machine to stay the same.
• The user should expect to provide administrative credentials to enlist the deployed machines.

Users want to release & re-deploy without rebooting

We understand that MAAS users want to release a machine that’s been added to MAAS, and then deploy another workload — without having MAAS reboot the machine in an ephemeral environment to gather commissioning data. If you added a deployed machine, MAAS hasn’t run its commissioning scripts, so information might be missing. We should encourage the user to re-run commissioning in that case, but what kinds of compromises can we make to avoid rebooting the machine?

Well, we’ve considered these points:

• When the machine is released, any non-physical interfaces, like VLANs, bridges, and VFs, are removed, and the machine gets configured like a machine that has been commissioned.
• When a machine that has hardware information is released, it goes into a Ready state.

Users want to add deployed machines without SSH

MAAS users probably want to add already-deployed machines for which they have no SSH permissions. We assumed earlier that users would be okay logging in to the running machine and collecting hardware information from the system. There are ways, though, of using out-of-band management (like Redfish) to get the hardware information.

This is useful not only for machines where you don’t have SSH permissions, but also machines where you can’t collect hardware info in the normal way (e.g, Windows). While having MAAS collect the information itself would be great, it’s just too far out of scope for this first effort. Instead, we’ll document how to update the hardware information, so that someone can write a script that queries the BMC, or run on a Windows host, to collect the information and send it to MAAS.

In other words, we’d be happy here just to post an example script that queries a Redfish BMC and updates a machine in MAAS.

Users want deployed controllers to be visible

We would think that a MAAS user would want machines deployed as controllers to show up in the machine listing. That way, they can be reused, if desired. Currently when you add a controller in MAAS, we either link it to an existing machine (which might be Deployed, or in any other state, really), or we create a new machine in the New state.

It would be good to unify the way we update machines and controllers. This means showing the machine in the machine listing, since it’s no different than other workloads. If you need machines for a workload, you might choose to re-use a controller that’s not really needed, or you might choose to replace the controller with a different machine.

Here’s how that feature might play out:

• If a machine is created when a new controller is added to MAAS, its status becomes Deployed.
• When upgrading to the latest version of MAAS, all machines that are linked to a controller are marked as Deployed.
• The machine that is linked to a controller is visible in the machine listing, just like a normal machine.
• When linking a machine to a controller, the machine is locked.
• When removing a controller, its machine gets unlocked.
• When removing a controller, if the machine doesn’t have a BMC, it’s removed from MAAS.
• When a machine is linked to a controller, it has a workload annotation indicating what it’s a MAAS controller.

Users want to periodically update hardware information

We expect that MAAS users would want write script that can periodically update hardware information for a deployed machine. When you have a running machine, you might want to have something running on that machine that periodically checks the hardware information and tells MAAS about it if it has changed.

While this should already be possible if some of the above features are implemented, in this case, you wouldn’t make use of your own credentials. Instead, you would generate a set of credentials that are limited to one machine only. This is important, since everyone with root on the machine will be able to see those credentials — so they need to be as restrictive as possible.

Feature considerations include the following:

• You can get a set of credentials from MAAS which allows the hardware information to be updated for a single machine only.
• The credentials for a machine may be regenerated, in which case the old credentials are no longer valid.
• Only the owner of the machine is able to generate, and see, new credentials. Anonymous and non-owner users may not.

What we’re testing at the moment

We’ve made some progress toward 3.1 features, which we’ll summarize here.  Understand that we’re just getting started, and as mentioned before, there’s no guarantee we’ll implement all these features — but here’s how we’re doing at the moment.

How to enlist a machine running a workload

In order to add machine that’s already running a workload, there are three options:

UI: In the “Add machine” form on the machine listing page, add the machine as normal, but check the “Deployed” checkbox (this work hasn’t been done yet, but it’s on our list).

API: Create a machine passing the deployed flag:

$ maas $profile machines create deployed=true \
hostname=mymachine architecture=amd64 
mac_addresses=00:16:3e:df:35:bb power_type=manual

On the machine itself: If the machine is running Ubuntu, you can download a helper script from MAAS and create the machine that way. This is the way we’d recommend that users depend on:

$ wget http://$MAAS_IP:5240/MAAS/maas-run-scripts
$ chmod 755 maas-run-scripts
$ ./maas-run-scripts register-machine --hostname mymachine \
http://$MAAS_IP:5240/MAAS $MAAS_API_TOKEN

Now you have a machine in MAAS that’s in the deployed state, with no hardware information yet.

How to update H/W information for a deployed machine

The best way to update the hardware information for a deployed machine is to download the maas-run-scripts script, as shown above, and run it on the machine itself:

$ wget http://$MAAS_IP:5240/MAAS/maas-run-scripts
$ chmod 755 maas-run-scripts
$ ./maas-run-scripts register-machine --hostname mymachine \
http://$MAAS_IP:5240/MAAS $MAAS_API_TOKEN

If you created the machine with the maas-run-scripts, you should have such a mymachine-creds.yaml file already. If not, it should look like this:

reporting:
maas:
consumer_key: YGT6QKSH65aap4tGnw
endpoint: http://10.5.32.98:5240/MAAS/metadata/status/wxwwga
token_key: Lyy9BS4tKsQakDQScy
token_secret: V8vta8Azwn6FZVkfHnuTvLGLScAvEufB

In this example, 10.5.32.98 is the IP/hostname of the MAAS deployment, wxwwga is the system id of the machine in MAAS, and the rest are the API token for the machine. You can get the credentials from the MAAS API like this:

$ maas $profile machine get-token wxwwga
Success.
Machine-readable output follows:
{
"token_key": "Lyy9BS4tKsQakDQScy",
"token_secret": "V8vta8Azwn6FZVkfHnuTvLGLScAvEufB",
"consumer_key": "YGT6QKSH65aap4tGnw"
}

Summary

So we just wanted to share what we’re thinking, and some of the directions we’re taking to support using deployed machines without bringing them down or hampering their workload. We really would welcome some feedback here.