SUMMARY
"Filesystem is still mounted elsewhere, or is unclean" error appears in cloudadm logs or when using cloudadm connect command line
ISSUE
Cloud backup copies are failing.
The cloud storage returns the message: "Filesystem %s is still mounted elsewhere, or is unclean"
This error may appear if the command line below is tried, or, may appear in cloudadm logs in /usr/bp/logs.dir
RESOLUTION
If you have confirmed the filesystem is not already mounted elsewhere, it is unclean. This may be related to a problem with Amazon S3's drivers creating invalid local cache data. With release 9.0.0-13, Unitrends detects this issue automatically and clears the cache files and reconnects. To confirm you do not have this issue, simply update the Unitrends Backup appliance to release 9.0.0-13 or higher. Then, from the UI you can attempt to check the "online" box for the cloud storage and confirm if it comes online or if you are provided a further error.
Note: Starting with release 9.0.0-13, Cloud Storage will no longer automatically mount. This was a purposeful change to avoid potential for filesystem corruption when a archive is not active. Cloud storage will auto-mount only when a job is in process and will then be dismounted at job completion, and will reflect "offline" in the UI at all other times. Please do not be concerned by this status.
If you continue to have issues connecting, the next troubleshooting step is to additionally, check the output from the following command and ensure the "/" partition is 80% or less full. If it is more than 80% full, run the following command:
# rm -rf /tmp/*
If the above steps continue to fail to mount your storage, the Amazon filesystem may in fact be corrupt. You can attempt an auto-check and repair of the filesystem by manually running the following:
- Use the storage name %s from the message where you see <storage> below. If the cloudhook storage entry was successfully added, you can use this command:
# /usr/bp/bin/cloudadm connect -f <storage>
- If the cloudhook storage entry is not in the database (may be needed when connecting another appliance to a prior cloud archive or if cloud storage was removed as an attempt to recreate the mount), use this command:
# /usr/bp/bin/cloudadm fsck <storage> <address> <bucket> <cloud> <user> <password>
Cloudadm Parameters
<storage> = the storage name used as a mount point
<address> = ‘s3’ if Amazon, ‘gs’ if Google, ‘swift’ if Rackspace
<bucket> = the bucket or container name, including prefix if any
<protocol> = always ‘cloud’ here
<user> = the cloud API user or Access Key
<password> = the API password: called Secret, Secret Access Key or API Key
After all above troubleshooting steps, if the S3 storage continues to not mount, you are certain no other systems (including other unitrends appliances) are mounted to the same S3 bucket, then a new bucket should be created and tested. If a new bucket works with the same credential and this one does not, contact Amazon Support for more details.
Reference
Additional CloudHook Information CAUSE
The "filesystem is either already mounted to a system or is unclean" error can be caused by 3 possible issues:
- the S3 driver cache is corrupt locally. This only impacts Amazon S3 and is an issue their software will present in some cases, most commonly where connection reliability to Amazon S3 cannot be made perfectly reliable.
- the /tmp filesystem is full on the Unitrends Appliance.
- the filesystem is "unclean" or corrupt and requires repair. This is unfortunately a potential risk when dealing with remotely mounted filesystems. to limit risk of a large data loss, customers should change buckets periodically, and not put all their backups in a single bucket. Repair instructions are above as well as in the referenced article.