SUMMARY
system_storage file system is 100% full
ISSUE
Purpose
This article addresses discusses how to address an issue that results in the system_storage file system filling up causing is to become 100% full.
Description
Vaulting will use local storage to create “delta” files. Delta file contain the change blocks between backups which are used to reduce the bandwidth requirements for offsite synchronization. If there are more than 400G of changed blocks, the default file system may become full and cause vaulting failures.
Resolution
To resolve this issue, either the amount of space used by delta creation must be decreased or the space allocated to create deltas files must be increased.
Option 1. To decrease the amount of space delta files consume:
- In the UI, navigate to Settings > Vaulting > Vaulting Attributes > Connection Options and Vaulting Control.
- Decrease the amount of space allocated in the Maximum Space (GB) field.
Option 2. To increase the amount of space available for delta file creation:
*** Caution: For advanced users only. ***
*** May cause un-repairable damage! ***
- Use putty.exe to open a bash console on the backup system or login at a console and hit “F2” for a bash console.
- Verify that the system_storage file system if full to determine whether these steps are necessary. In the example below, system_storage is contained on the software raid partition md4. This may differ based on you backup system model.
[root@system ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 14G 1.6G 12G 12% /
/dev/md4 429G 429G 11K 100% /var/opt/unitrends/system_storage
/dev/md3 370G 262M 351G 1% /var/opt/unitrends/database
/dev/sdc1 236M 19M 205M 9% /mnt/boot_sync
/dev/md2 47G 958M 43G 3% /usr/bp
/dev/sdb1 236M 19M 205M 9% /boot
tmpfs 24G 0 24G 0% /dev/shm
/dev/sda 28T 3.0T 25T 11% /backups
3. Change your current working directory to /backups.
[root@system ~]# cd /backups/
cd /backups
4. View a listing of files/directories within the /backups directory. If this applies to your system, you will see the tmp directory linked (->) to /var/opt/unitrends/system_storage/tmp.
[root@system backups]# ls -lah
total 20K
drwxr-xr-x 13 root root 4.0K Oct 12 11:45 .
drwxr-xr-x 24 root root 4.0K Sep 14 16:25 ..
drwxr-xr-x 2 root root 6 Jul 30 14:43 D2Dbackups
-rw-r----- 1 root root 439 Oct 12 11:45 .DPUusers
lrwxrwxrwx 1 root root 39 Jul 30 14:44 dumps -> /var/opt/unitrends/system_storage/dumps
drwxr-xr-x 3 root root 19 Jul 31 08:42 samba
-rw------- 1 root root 8.0K Oct 15 09:59 SISinfo
lrwxrwxrwx 1 root root 37 Jul 30 14:44 tmp -> /var/opt/unitrends/system_storage/tmp
drwxrwx--- 2 root root 22 Sep 25 09:48 trash
lrwxrwxrwx 1 root root 45 Jul 30 14:44 UnitrendsDataBase -> /var/opt/unitrends/database/UnitrendsDatabase
5. Remove the symbolic link which link tmp to /var/opt/unitrends/system_storage/tmp. If your results do not match exactly, abort the process using CTRL-C and consult an expert.
[root@system backups]# rm tmp
rm: remove symbolic link `tmp'? y
6. Create a tmp directory inside /backups.
[root@system backups]# mkdir tmp
7. Now, remove the contents of the /var/opt/unitrends/system_storage/tmp directory.
[root@system backups]# rm –rf /var/opt/unitrends/system_storage/tmp/*