SUMMARY
Notes, advanced configuration, and troubleshooting for Cloudhook.
ISSUE
Describe additional Cloudhook information that may be useful.
RESOLUTION
CloudHook Notes
Bucket/Container Names
Note that cloud bucket names are only supported with lower case letters, numbers, dash '-' and period '.'. Each of the cloud providers has different syntax rules for bucket names and their access methods, and these are the common characters allowed for all cloud providers. From the CloudHook menu, the bucket/container name is validated against these characters before saving it.
Note that Rackspace containers for CloudHook must be preceeded by the hostname ‘auth.api.rackspacecloud.com’. For example: auth.api.rackspacecloud.com/mycontainer. This hostname is automatically added when the Unitrends UI creates a Rackspace cloud storage instance.
Rackspace Container Types
CloudHook currently supports Rackspace ‘Private’ containers, but not Rackspace Public/CDN containers.
Amazon Container Types
CloudHook supports AWS S3 containers and new with release 10.3.x also supports S3 IA (Infrequent Access) containers, but does not support Glacier containers. CloudHook supports AWS Signature V2 and Signature V4. To enable V4 signatures, see Enabling AWS V4 Authentication.
Google Cloud Container Types
CloudHook supports Google Cloud standard and DRA containers.
Other "S3 compatible" containers from 3rd parties:
Though unsupported by Unitrends directly, and titled entirely under "use at your own risk", it may be possible for end users to create CloudHook connections for cold copy use from a Unitrends appliance to other providers that advertise full S3QL compatibility via standard S3 API. Customers are advised 1) this process should be heavily tested including not just simple upload and recovery, but recovery of "aged" data as well, 2) a full understanding Unitrends Support staff will refuse to assist with, diagnose, or escalate any issues with cloud storage systems other than Amazon's own S3 or S3 IA, or Google's Nearline and that data may be at risk of loss or inability to recover or be subject to fees from 3rd party. This stated, customers may use comand line tools to try and test connectivity with other providers at their own risk using this command syntax:
cloudadm create <storagename> s3c <hostname:port/bucketname> cloud <APIkey> <SecretKey>
Username/Password
Cloud provider terms used for CloudHook username and password (API access):
Folders
When using folders for Amazon and Google, they appear in the cloud provider menu as a filename prefix only, unless you specify the folder with a trailing ‘/’ (e.g. mybucket/myfolder/). Then the data block files will appear in the cloud provider web menu underneath the folder.
Firewall / Proxy Issues
Sometimes the customer network will have a firewall or proxy that prevents read/write access to certain ports. The ports that CloudHook requires are ports 80 and 443, and if those ports are restricted, it can cause the underlying handshake functions to hang, waiting on a response from the port. The solution is to open ports 80 and 443 for the CloudHook system.
DNS is required
Resolving the IP address of cloud storage requires DNS, so make sure a DNS server is configured.
Connection Speed to the Cloud Storage
You can verify that your connection speed to the external network is sufficient to handle archiving to the cloud by running /usr/bp/bin/speedtest.sh. This will run a standard speed test from speedtest.net. Some sample results are Download: 74.31 Mbits/s, Upload: 78.23 Mbits/s.
Limits on archive throughput to Cloud Storage
Depending on your connection speed, the number of CPUs, and your available memory, you can determine how much data can be archived per day. With the sample speed results above, 16 CPU cores, and 48 GB of memory, a test CloudHook archive job throughput was 8.2 MB/s (66 Mbps), so this would yield up to 696 GB over 24 hours. A test case with 8 cores and 16GB memory showed throughput of 3.4 MB/s (27 Mbps), yielding throughput up to 287 GB over 24 hours. Your speed and memory size will likely be different, and results would vary depending on other activity on the system. Conduct your own test in your environment to understand your throughput. You will have to restrict your CloudHook archiving if your archive data size is beyond what your connection speed and throughput supports.
Rotate to a new Cloud Storage bucket periodically
Since accumulating lots of cloud filesystem data over time slows down cloud archives, adds overhead and some volatility to the remote cloud filesystem, it is recommended that you open a new bucket for use at least once every year, or more often if the archive size is large. The previous buckets are still there for history.
CloudHook data encryption
The s3ql software does use SSL encryption for data in transit by default, unless the --no-ssl option is specified, and CloudHook never specifies --no-ssl. See https://bitbucket.org/nikratio/s3ql/overview and http://www.rath.org/s3ql-docs/backends.html for details.
RequestTimeTooSkewed Error with AWS S3
This happens when your local system time is more than a few seconds different than the AWS server. To compare the local and remote times:
Get the UTC/GMT time from Amazon
# http://s3.amazonaws.com -v
Get the UTC on your local machine
# date –u
To resolve this, configure ntp in the Setup Wizard or /etc/ntp.conf, including these servers:
server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst
If not using the Setup Wizard, you also need to run
service ntpd restart
HTTPError: 403 Forbidden error with Amazon S3
This happens when the AWS key for this bucket either has not been created or does not have full rights to S3 storage. From the AWS web console, create the AWS key if necessary, and set the AWS policies to give that key full rights to S3 storage. Consult Amazon support for further questions.
BusyError: database is locked error in cloudadm.log
This error message has two possible causes:
- There is another process using the s3ql database with a lock pending, which is true in this case. Check for this with:
ps –ef |grep mount.s3ql
If so, resolve this by applying cloudadm v1.41 or later from EAPP-1116. After that, future mounts of the cloud storage will succeed.
- This can happen when the CloudCacheDir (default=/backups/cloudcache) resides on an NFS share. If this system is a UEB with /_Stateless mounted on NFS, then the latency of the cloudcache can cause BusyError. If so, resolve this by changing /usr/bp/bpinit/master.ini so that the [CloudHook] CloudCacheDir=/var/cloudcache, or a similar directory on a local filesystem. This can also be done from the RRC UI Settings/System/General(Advanced) menu.
"FullError: database or disk is full" in cloudadm.log
This error message indicates that the size of the cloud filesystem database has exceeded the CloudCacheSize allotted. You can increase this size from 512000 to 1024000 in /usr/bp/bpinit/master.ini with this command.
/usr/bp/bin/bputil -p CloudHook CloudCacheSize 1024000 /usr/bp/bpinit/master.ini
Note that the cloud bucket size is probably too large if this occurs and should be rotated to a fresh cloud bucket.
Bucket is already mounted by <asset>
This error message indicates that the other <asset> either currently has the bucket mounted, or the other asset went down uncleanly and left the bucket with unflushed cache metadata.
The best resolution is to go to the other asset and cleanly unmount the cloud storage bucket. If that asset cannot be accessed, there will likely be some loss of in-flight data, and there is a custom manual process to mount the bucket cleanly on the new asset system from a previous copy of the metadata which pre-dates the metadata corruption.
This manual process usually will require a Unitrends support escalation, refering to "Recovering CloudHook bucket from unclean metadata".
CloudCache directory is not fast enough
Sometimes, if the /backups/cloudcache directory is on a NAS device, it may not be fast enough to support the s3ql filesystem. If so it will produce various traceback errors with messages that have these lines:
Uncaught top-level exception:
Traceback (most recent call last):
You can move the cloudcache directory to a different local location by doing these steps:
/usr/bp/bin/bputil -p CloudHook CloudCacheDir /usr/bp/cloudcache /usr/bp/bpinit/master.ini
/usr/bp/bin/cloudadm status (make sure no cloud storage has ONLINE=t)
/usr/bp/bin/cloudadm cleancache
Bucket is in an AWS Signature V4-only region
If you get an error message like this, it indicates that the s3 bucket is in a region that allows AWS Signature V4 only and does not support V2.
S3Error: InvalidRequest: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
To enable AWS Signature V4, see Enabling AWS V4 Authentication.
More Cloud Storage Provider Information
The Unitrends CloudHook feature utilizes S3QL software in Linux to transparently access various Cloud storage providers. See these references for more information about configuring the Cloud provider storage backend.
Advanced CloudHook Settings
You can access these settings from the ‘Settings / System, Updates, and Licensing / General Configuration (Advanced)’ menu. They are stored in the /usr/bp/bpinit/master.ini file.
CloudCacheSize 512000
Maximum size of cloud cache in KB. Default setting is 512,000 KB, which should be optimal for most cases. You might want to increase the cloud cache size for archive sets larger than 500 GB.
EnforceMax 1
Enforces the user-specified purging threshold. Setting is enabled (1) by default. When this setting is enabled, archive jobs fail if they require storage space beyond the threshold. If this setting is disabled (0), the appliance still attempts to archive to the storage.
AlwaysPurgeGB 0
Always purge. Setting is disabled (N=0) by default. Enabling it and specifying a number (N) causes all cloud archive jobs to purge if there is less than N GB of free space below the purging threshold. This has the same effect as if the purge checkbox had been checked for those jobs.
RemoveFiles 0
Remove files also when removing an archive set manually from the Archive Status menu. Setting is disabled (0) by default. If you enable this setting (1), archived data is deleted from the archive media when you manually remove an archive set from the Unitrends database. If this setting is disabled, only archive reference information is removed when you remove an archive set and the archived data is not impacted.
CloudCacheDir /backups/cloudcache
Specifies the cloud cache directory. If you change this directory, be sure the new location has enough space for the cloud cache.
NOTES
Additional Information
