Ask the Community
Groups
system_storage file system is 100% full - Connect IT Community | Kaseya
<main> <article class="userContent"> <h2 data-id="summary"><strong>SUMMARY</strong></h2> <p>system_storage file system is 100% full</p> <h2 data-id="issue"><strong>ISSUE</strong></h2> <p></p> <h3 data-id="purpose">Purpose </h3> <p>This article addresses discusses how to address an issue that results in the system_storage file system filling up causing is to become 100% full.</p> <h3 data-id="description">Description</h3> <p>Vaulting will use local storage to create “delta” files. Delta file contain the change blocks between backups which are used to reduce the bandwidth requirements for offsite synchronization. If there are more than 400G of changed blocks, the default file system may become full and cause vaulting failures.</p> <h3 data-id="resolution">Resolution</h3> <p>To resolve this issue, either the amount of space used by delta creation must be decreased or the space allocated to create deltas files must be increased.</p> <p> </p> <p><i>Option 1</i>. To decrease the amount of space delta files consume:</p> <ol><li> In the UI, navigate to Settings > Vaulting > Vaulting Attributes > Connection Options and Vaulting Control.</li> <li>Decrease the amount of space allocated in the Maximum Space (GB) field.</li> </ol><p> </p> <p><i>Option 2</i>. To increase the amount of space available for delta file creation:</p> <p><span style="color: #FF0000;"><b>*** <i>Caution</i>: For advanced users only. ***</b></span></p> <p><span style="color: #FF0000;"><b>*** May cause un-repairable damage! ***</b></span></p> <ol><li>Use putty.exe to open a bash console on the backup system or login at a console and hit “F2” for a bash console.</li> <li>Verify that the system_storage file system if full to determine whether these steps are necessary. In the example below, system_storage is contained on the software raid partition md4. This may differ based on you backup system model.</li> </ol><blockquote class="blockquote"> <p><span style="font-family: courier;">[root@system ~]# df -h</span></p> <p><span style="font-family: courier;">Filesystem Size Used Avail Use% Mounted on</span></p> <p><span style="font-family: courier;">/dev/md0 14G 1.6G 12G 12% /</span></p> <p><span style="font-family: courier;">/dev/md4 429G 429G 11K 100% /var/opt/unitrends/system_storage</span></p> <p><span style="font-family: courier;">/dev/md3 370G 262M 351G 1% /var/opt/unitrends/database</span></p> <p><span style="font-family: courier;">/dev/sdc1 236M 19M 205M 9% /mnt/boot_sync</span></p> <p><span style="font-family: courier;">/dev/md2 47G 958M 43G 3% /usr/bp</span></p> <p><span style="font-family: courier;">/dev/sdb1 236M 19M 205M 9% /boot</span></p> <p><span style="font-family: courier;">tmpfs 24G 0 24G 0% /dev/shm</span></p> <p><span style="font-family: courier;">/dev/sda 28T 3.0T 25T 11% /backups</span></p> </blockquote> <p> 3. Change your current working directory to /backups.</p> <blockquote class="blockquote"> <p><span style="font-family: courier;">[root@system ~]# cd /backups/</span></p> <p><span style="font-family: courier;">cd /backups</span></p> </blockquote> <p> 4. View a listing of files/directories within the /backups directory. If this applies to your system, you will see the tmp directory linked (->) to /var/opt/unitrends/system_storage/tmp.</p> <blockquote class="blockquote"> <p><span style="font-family: courier;">[root@system backups]# ls -lah</span></p> <p><span style="font-family: courier;">total 20K</span></p> <p><span style="font-family: courier;">drwxr-xr-x 13 root root 4.0K Oct 12 11:45 .</span></p> <p><span style="font-family: courier;">drwxr-xr-x 24 root root 4.0K Sep 14 16:25 ..</span></p> <p><span style="font-family: courier;">drwxr-xr-x 2 root root 6 Jul 30 14:43 D2Dbackups</span></p> <p><span style="font-family: courier;">-rw-r----- 1 root root 439 Oct 12 11:45 .DPUusers</span></p> <p><span style="font-family: courier;">lrwxrwxrwx 1 root root 39 Jul 30 14:44 dumps -> /var/opt/unitrends/system_storage/dumps</span></p> <p><span style="font-family: courier;">drwxr-xr-x 3 root root 19 Jul 31 08:42 samba</span></p> <p><span style="font-family: courier;">-rw------- 1 root root 8.0K Oct 15 09:59 SISinfo</span></p> <p><span style="font-family: courier;">lrwxrwxrwx 1 root root 37 Jul 30 14:44 tmp -> /var/opt/unitrends/system_storage/tmp</span></p> <p><span style="font-family: courier;">drwxrwx--- 2 root root 22 Sep 25 09:48 trash</span></p> <p><span style="font-family: courier;">lrwxrwxrwx 1 root root 45 Jul 30 14:44 UnitrendsDataBase -> /var/opt/unitrends/database/UnitrendsDatabase</span></p> </blockquote> <p> </p> <p>5. Remove the symbolic link which link tmp to /var/opt/unitrends/system_storage/tmp. <b>If your results do not match exactly, abort the process</b> using CTRL-C and consult an expert.</p> <blockquote class="blockquote"> <p><span style="font-family: courier;">[root@system backups]# rm tmp</span></p> <p><span style="font-family: courier;">rm: remove symbolic link `tmp'? y</span></p> </blockquote> <p>6. Create a tmp directory inside /backups.</p> <p><span style="font-family: courier;"> [root@system backups]# mkdir tmp</span></p> <p>7. Now, remove the contents of the /var/opt/unitrends/system_storage/tmp directory.</p> <blockquote class="blockquote"> <p><span style="font-family: courier;">[root@system backups]# rm –rf /var/opt/unitrends/system_storage/tmp/*</span></p> </blockquote> </article> </main>