# How to securely decommission a MicroCloud deployment #### IMPORTANT This process will erase all data associated with your MicroCloud deployment. Make copies of any data that you need to preserve before proceeding. Refer to [How to back up instances](https://canonical.com/microcloud/docs/default/lxd/howto/instances_backup/#instances-backup) and [How to back up custom storage volumes](https://canonical.com/microcloud/docs/default/lxd/howto/storage_backup_volume/#howto-storage-backup-volume) for relevant details. This guide walks you through the steps to decommission an entire MicroCloud cluster. If you only need to decommission a single cluster member, first [remove the member from the cluster](https://canonical.com/microcloud/docs/default/how-to/member_remove/index.html.md#howto-member-remove). After removing the member, [update the certificate](https://canonical.com/microcloud/docs/default/lxd/howto/cluster_manage/#cluster-manage-update-certificate) on the cluster remaining in production. Then, return to this guide, skip ahead to [Remove snaps](#howto-decommission-remove-microcloud), and follow the instructions in the sections that follow. Some commands used to decommission MicroCloud are LXD or MicroCeph commands. Refer to the [LXD decommissioning guide](https://canonical.com/microcloud/docs/default/lxd/howto/decommission/#howto-decommission) and the [MicroCeph guide to removing disks](https://canonical.com/microcloud/docs/default/microceph/how-to/remove-disk/) for related information. ## Remove offline cluster members Use the `--force` flag to [remove any offline cluster members](https://canonical.com/microcloud/docs/default/how-to/member_remove/index.html.md#howto-member-remove) (you will remove online cluster members later in the process): ```bash sudo microcloud remove --force ``` ## Revoke remote access List all identities that have access to LXD, then delete each identity: ```bash lxc auth identity list lxc auth identity delete / ``` ## List projects Instances, profiles, and custom volumes are scoped by [project](https://canonical.com/microcloud/docs/default/lxd/projects/#projects). For deployments with more than one project, you must repeat some steps for **each** project, each time using the `--project` flag. You do not need to use the `--project` flag to decommission deployments with only one project. Run this command to get a list of all projects: ```bash lxc project list ``` #### NOTE You can also delete a project (except the `default` project) and all of its project-level entities with: ```bash lxc project delete --force ``` ## Delete data You can run the commands in this section on any online cluster member to delete data. #### IMPORTANT Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks. To prevent unauthorized data recovery, you must [destroy and sanitize your data](#howto-decommission-destroy-data). ### Stop and delete instances For each project, stop all instances: ```bash lxc stop --all --project ``` Next, for each project, list all instances, then delete each instance: ```bash lxc list --project lxc delete --project ``` If you are unable to stop or delete an instance, use the `--force` flag: ```bash lxc stop --force --project lxc delete --force --project ``` ### Delete profiles For each project, list all profiles: ```bash lxc profile list --project ``` Each project has a `default` profile that cannot be deleted. Delete all other profiles: ```bash lxc profile delete --project ``` ### Remove disk devices from `default` profiles You cannot delete a storage pool used by an instance, profile, or custom volume. You must, therefore, remove any disk devices used by the `default` profiles in order to delete any storage pools or custom volumes referenced by those devices. At a minimum, the `default` profile of the `default` project has a disk device named `root` that references a storage pool. Remove this device with: ```bash lxc profile device remove default root --project default ``` To check for additional disk devices, view information about the `default` profile of each project: ```bash lxc profile show default --project ``` Remove any remaining disk devices that reference storage pools: ```bash lxc profile device remove default --project ``` ### Delete custom volumes To delete [custom volumes](https://canonical.com/microcloud/docs/default/lxd/explanation/storage/#storage-volume-types), you must specify the storage pools used by the volumes. First, list all storage pools across projects: ```bash lxc storage list ``` Next, for each storage pool, list the custom volumes. Use the `--all-projects` flag to view all custom volumes across projects: ```bash lxc storage volume list type=custom --all-projects ``` Use the `PROJECT` column in the output to identify the project associated with each custom volume. Then delete each custom volume, specifying both the storage pool and the project: ```bash lxc storage volume delete --project ``` ### Delete storage pools #### NOTE Storage pools are not scoped by project, so you do not need to use the `--project` flag with `lxc storage` commands. List all storage pools, then delete each one: ```bash lxc storage list lxc storage delete ``` ### Delete monitoring data Delete data from any external systems that you used to monitor [LXD events](https://canonical.com/microcloud/docs/default/lxd/howto/logs_loki/#logs-loki), [LXD metrics](https://canonical.com/microcloud/docs/default/lxd/metrics/#metrics), or [Ceph logging](https://canonical.com/microcloud/docs/default/microceph/explanation/security/security-overview/), such as [Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), or [Grafana](https://grafana.com/). Refer to the documentation for those systems for details. ## Remove MicroCeph OSDs To [remove MicroCeph OSDs](https://canonical.com/microcloud/docs/default/microceph/how-to/remove-disk/), list all disks, then remove each one: ```bash microceph disk list sudo microceph disk remove ``` Finally, verify that the OSDs have been removed: ```default microceph disk list ``` #### NOTE If you are unable to remove an OSD, use the `--bypass-safety-checks` flag: ```bash sudo microceph disk remove --bypass-safety-checks ``` ## Remove remaining cluster members After deleting data, you can remove the online cluster members from the cluster. First, list all cluster members: ```bash microcloud cluster list ``` You can then [remove most cluster members](https://canonical.com/microcloud/docs/default/how-to/member_remove/index.html.md#howto-member-remove) with: ```bash sudo microcloud remove ``` However, before reducing the cluster from two members to one member, you must [clean up the Ceph monitor map](https://canonical.com/microcloud/docs/default/how-to/member_remove/index.html.md#howto-member-remove-reduce-cluster). #### NOTE As you remove each member, you can run `microcloud status` on the remaining cluster members to verify the removal. ## Remove snaps #### IMPORTANT Run these commands on **every** machine that you decommission. Removing MicroCloud **does not** erase [ZFS pools (zpools)](https://canonical.com/microcloud/docs/default/lxd/reference/storage_zfs/#storage-zfs) or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs). To securely decommission MicroCloud, you must [destroy and sanitize your data](#howto-decommission-destroy-data). Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps. Use the `--purge` flag, or a snapshot of your data will be preserved: ```bash sudo snap remove microcloud --purge sudo snap remove lxd --purge sudo snap remove microceph --purge sudo snap remove microovn --purge ``` #### NOTE The MicroCeph and MicroOVN snaps may not be installed if you deployed a MicroCloud without those components. Verify that the snaps and associated data were removed. The following commands should report that none of these snaps are installed and that the `/var/snap/microcloud/`, `/var/snap/lxd/`, `/var/snap/microceph/`, and `/var/snap/microovn` directories do not exist: ```bash snap list microcloud lxd microceph microovn ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/ ``` ## Destroy and sanitize data Data deleted with MicroCloud, LXD, or MicroCeph commands remains readable and can be recovered by users with access to disks used in your deployment. To prevent unauthorized recovery, you must physically overwrite the data. Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning. If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics. For clusters [configured with OIDC](https://canonical.com/microcloud/docs/default/lxd/authentication/#authentication-openid), consult your OIDC identity provider for the steps to remove any data associated with your profile. Likewise, if you used [ACME services to issue server certificates](https://canonical.com/microcloud/docs/default/lxd/authentication/#authentication-server-certificate), refer to the service provider for the steps to remove any associated data. #### IMPORTANT Sanitized data is irreversibly destroyed and cannot be recovered.