[OpenStack] Clearing out a Dead VM
While working on an OpenStack cloud (the ‘Train” release), I ran into an issue where a VM did not get created properly because my controller didn’t (fully) see my compute node. This created an issue where the VM was not assigned to any host, and a misleading and incorrect error that was reported as a “Permission Denied” error. Here’s the I got from the nova-api.log:
2019-12-05 13:57:35.266 29922 WARNING nova.compute.api [req-b2caee1d-350c-4269-9785-0634a518ace6 1375ef6956d24237bc82a134a9313bbb abe39009279847a48ebb7705c60b1ca9 - default default] [instance: fc568f19-50fb-4ec3-acc0-dbfc772a6524] instance's host None is down, deleting from database
2019-12-05 13:57:35.940 29922 WARNING keystoneauth.discover [req-b2caee1d-350c-4269-9785-0634a518ace6 1375ef6956d24237bc82a134a9313bbb abe39009279847a48ebb7705c60b1ca9 - default default] Failed to contact the endpoint at http://controller:8778 for discovery. Fallback to using that endpoint as the base url.: Forbidden: Forbidden (HTTP 403)
2019-12-05 13:57:35.946 29922 WARNING keystoneauth.discover [req-b2caee1d-350c-4269-9785-0634a518ace6 1375ef6956d24237bc82a134a9313bbb abe39009279847a48ebb7705c60b1ca9 - default default] Failed to contact the endpoint at http://controller:8778 for discovery. Fallback to using that endpoint as the base url.: Forbidden: Forbidden (HTTP 403)
2019-12-05 13:57:35.949 29922 WARNING nova.scheduler.client.report [req-b2caee1d-350c-4269-9785-0634a518ace6 1375ef6956d24237bc82a134a9313bbb abe39009279847a48ebb7705c60b1ca9 - default default] Unable to delete allocation for instance fc568f19-50fb-4ec3-acc0-dbfc772a6524. Got 403 while retrieving existing allocations: (<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /allocations/fc568f19-50fb-4ec3-acc0-dbfc772a6524
on this server.</p>
</body></html>
)
2019-12-05 13:57:35.950 29922 INFO nova.api.openstack.wsgi [req-b2caee1d-350c-4269-9785-0634a518ace6 1375ef6956d24237bc82a134a9313bbb abe39009279847a48ebb7705c60b1ca9 - default default] HTTP exception thrown: Failed to delete allocations for consumer fc568f19-50fb-4ec3-acc0-dbfc772a6524. Error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /allocations/fc568f19-50fb-4ec3-acc0-dbfc772a6524
on this server.</p>
</body></html>So… can’t create a working VM, then can’t delete the bad VM. How do I fix this? While I researched and tried lots of things, the thing I found that eventually worked was to manually set the “state” of the VM, then you’d be able to delete the thing.
The commands were as follows:
[root@server ~]# openstack server list
+--------------------------------------+------+--------+----------+-----------------+----------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------+-----------------+----------------+
| fc568f19-50fb-4ec3-acc0-dbfc772a6524 | test | ERROR | | centos7-generic | 2-2-40 |
| 7a14d479-04ee-419f-abe0-6cc1b92faab3 | test | ERROR | | centos7-generic | 2-2-40 |
+--------------------------------------+------+--------+----------+-----------------+----------------+
[root@server ~]# openstack server set --state active fc568f19-50fb-4ec3-acc0-dbfc772a6524
[root@server ~]# openstack server delete fc568f19-50fb-4ec3-acc0-dbfc772a6524
[root@server ~]# openstack server list
+--------------------------------------+------+--------+----------+-----------------+----------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------+-----------------+----------------+
| 7a14d479-04ee-419f-abe0-6cc1b92faab3 | test | ERROR | | centos7-generic | 2-2-40 |
+--------------------------------------+------+--------+----------+-----------------+----------------+I had to do the same thing with the other test VM that’s there, but I was happy that I found a way to delete it at all. Since I couldn’t find any other resource that mentioned how to delete a dead VM like this, I posted this here. I hope this helps someone.