SUMMARY
Image-level and host-level (VM) Instant Recovery (IR) instances appear to hang after functioning without issue.
ISSUE
During instant recovery, when the guest VM writes data to its virtual disks, disk writes may be stored on the appliance until the VM disks are fully migrated. The underlying fuse processes (fusevdisk, fusevhd, fusevxdisk, fuseblk) handle these writes. There are default reservation sizes limit the total amount of bytes written. When a write limit is reached, the VM may hang, halt or suspend (Hyper-V) due to out of disk space.
When the limit is hit, the following message will appear in the appropriate fuse log:
fusevdisk.log, fusevhd.log, fusevxdisk.log:
... GB write limit exceeded, returning -ENOSPC
fuseblk.log:
Total Writes exceeded. Used: ... bytes of ...
An easy way to check for the above mentioned is by logging onto the appliance via CLI and running the following command:
egrep -i 'enospc|exceeded' /usr/bp/logs.dir/fuse*
RESOLUTION
Use the override setting in appliance master.ini to set a higher write limit.
The IR of VMware and Hyper-V backups use fusevdisk, fusevhd, and fusevxdisk.
The write limit override for these processes is recognized since unitrends-rr release 10.3.4-1.
The IR of image level backups use fuseblk process.
This write limit override is recognized by fuseblk since unitrends-rr release 10.3.8-1.
In the FuseProcesses section, the setting is "write_limit_gb=n", and it is either commented out or not there by defailt.
Example, write_limit_gb=25 sets 25 GB write limit.
[FuseProcesses]
max_fuse_processes_allowed=100 ; Maximum number of fuse processes allowed to run on this UB-RS appliance. NOTE: Increasing this to a very high value can cause issues on the UB-RS appliance.
write_reserve_gb=25 ; Amount to reserve per VM disk from IR storage allocation for writes.
write_limit_gb=25 ; Write limit allowed per VM disk when writes reach the VM disk reserved space.
The fuse processes default to 25 GB limit as a result of the reserve option, even if not defined in the master.ini.
Once the reserve limit is hit, the fuse process will use and regularly check the write_limit_gb setting in the master.ini to see if there is a newer limit which would allow the writes to continue. The fuse process will check no more than once a minute, and only when a write arrives in fuse. After changing the write limit, it may be necessary to wait a few minutes for the client to re-try the write, or to click on a dialog on the host to retry the write if such a dialog is displayed.