Troubleshooting VMs From Deleted Host Aggregates Tagged as Availability Zones

 In this solution:

If a Metacloud administrator deletes a host aggregate that was tagged as an Availability Zone (AZ), it affects your ability to resize or migrate any instances that were attached to that AZ when they were launched. 

How Instances Get Attached to AZs Based on Host Aggregates

Note: For additional context on host aggregates that are labeled as AZs, see Clarifying what "Availability Zone" Means in Metacloud.

A Metacloud administrator in your organization may have defined host aggregates and labeled them as AZs for various purposes, such as the following:

  • Segregating resources according to operating system (OS), storage type, or similar groupings.
  • Designating resources for different teams of users in your organization.
  • Dedicating resources for specific projects, such as testing a new cloud application.

If these types of AZs are defined in your cloud, users may attach instances to them when launching them using either of the following methods.

In the Dashboard workflow for launching an instance, users can select a specific AZ:

instance_launch_select_AZ.png

In the command-line interface (CLI), users can add an AZ parameter to the command for launching an instance, as in the following example:

$ openstack server create server_1 --availability-zone <az_name> --flavor m1.small --image ubuntu_1404_server_cloudimg_amd64 --nic net-id=<network_uuid>

How Deleting an AZ Based on a Host Aggregate Affects Attached Instances

If a user associates a new instance with an AZ when launching it, the instance record becomes permanently attached to the AZ in the OpenStack database. An administrator may later delete the host aggregate (and the related AZ) for various reasons. For example, the device grouping on which the aggregate was based is no longer necessary, and a new one is required. 

Deleting an AZ based on a host aggregate affects attached instances in the following ways:

  • Administrators cannot "cold-migrate" the instances, meaning they cannot shut down and migrate them. During the migration process the scheduler attempts to verify all the properties in the database record of the instance, including the AZ name, if one was originally attached. If that AZ name no longer exists, the Compute service (Nova) scheduler cannot proceed with the migration. 
  • Administrators cannot "live-migrate" the instances, or migrate them while they're still running, without specifying a destination Metacloud Hypervisor (MHV).
  • Users cannot resize the instances. The resizing process is similar to migration: OpenStack creates a new, resized disk on a different MHV and then attempts to migrate the instance to that MHV. 

In these scenarios, the scheduler displays an error message, such as one of the following in the Dashboard:

AZ_orphan_no_resize_message.png

 AZ_orphan_no_live_mig_sans_destination_message.png

In the CLI, you would see one of the following error messages:

ERROR (BadRequest): No valid host was found. There are not enough hosts available. (HTTP 400) (Request-ID: req-[ID])
No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-[ID])

Tip: If you submit a Support request for a migration or resize problem, and if you know that the affected instance was attached to a deleted AZ, it is helpful to include the name of the AZ in the request. Also, include the request ID if it is displayed in the error message.

Another problem is related to an issue in the 4.0 (Liberty) version of OpenStack: If a host aggregate tagged as an AZ is deleted, any instance that was attached to that AZ at launch appears to be associated with a different AZ in the Dashboard or command-line interface (in output from the openstack server show command); however, the database still associates it with the deleted AZ. 

Workarounds and Best Practices

Specifying a Destination MHV in a Live Migration

An administrator can live-migrate an instance attached to a deleted host aggregate tagged as an AZ if he or she specifies a destination MHV, as seen in the following screenshot and CLI sample:

AZ_orphan_liv_mig_destination.png 

$ openstack server migrate <INSTANCE_UUID> --<destination_MHV_name>

Not Attaching Instances to AZs when Launching Them

Avoid attaching a new instance to an AZ when launching it, unless it is necessary. 

If a user selects the option for Any Availability Zone when using the Dashboard to launch an instance, or does not specify an AZ when running openstack server create in the CLI, the Compute service "assigns" the instance to the AZ with the most hosting resources available; however, the instance shows a value of NULL for Availability Zone in the database and is not permanently associated with any AZ. This prevents migration or resize problems.

The following screenshot shows the launching of an instance in the Dashboard with Any Availability Zone selected.

instance_launch_any_AZ.png

 

 

 

Have more questions? Submit a request
Powered by Zendesk