This article will act as a parent article to provide and consolidate information to setup Hot Backup Copy (Replication), as well as, troubleshoot some of the more common errors that can occur while utilizing the Hot Backup Copy (Replication) functionality.
Hot Backup Copy (Replication) is an advanced feature of the Unitrends Enterprise Backup software and its performance can be affected by many different environmental factors. These factors can be broken down into three major areas of focus:
- Source appliance issues
- Target appliance issues
- Network or connection issues
Hot Backup Copy (Replication) Overview
Hot Backup Copy (Replication) is the logical synchronization of backup data from one Unitrends system to another or the Unitrends Cloud, in which the systems are connected by LAN or WAN. The originating system is referred to as the source, while the off-site system onto which the data is replicated is the target. Replication enables off-site storage of mission-critical data to protect against data loss in the event of a disaster. The source system is used to protect against loss of files, folders, and individual machines. The target system protects against loss of client data, as well as providing protection against the loss of the entire source system.
Replicated backups look like their counterparts on the source system and are restored in the same way. The target system is configured for retention while in replication view, using the same procedure as any source system (see About retention control). For a given client, you may have more than one backup group on the target, unlike legacy vaulting in which only the most recent backup is stored. As with source systems, the amount of retention on the target is dependent on space available and retention settings.
The target system can be deployed as a private cloud or as a multi-tenant cloud. The replication architecture ensures that the local source systems that replicate to a single target only have access to their data. This secure architecture is the basis of a multi-tenant architecture.
The replication process is fully managed from the target or source system. Using the replication dashboard, you can immediately gauge the status of replication by viewing active, previously completed, and pending replication jobs.
Unitrends’ replication leverages a secure tunnel based on the UDP protocol that creates a secure VPN tunnel and also provides resiliency to intermittent network failures via UDP knitting. If there is a network drop during replication, the process utilizes advanced checkpoint controls to proceed with replication at the time of failure. For details, see About secure tunnels for Unitrends systems.
The initial transfer of data from the source system to the target can occur over the WAN. However, for large data sets it is recommended to use a disk seeding mechanism to transfer the initial data set. Even with deduplication minimizing the amount of data being transferred, transfer speed is primarily governed by the size of the network pipe between the source and target systems. Seeding is also recommended in cases where available bandwidth is used for servicing end users at specific times during the day and cannot be used for replication.
Windows Instant Recovery is supported on the replication target system. See Windows Instant Recovery for more information.
Troubleshooting Hot Backup Copy (Replication) Issues
Hot Backup Copy (Replication) Setup
Failed: Failed to hash backup This error is rare, but means that the software was unable to compute a hash value for part of the backup. The deduplication system cannot work unless all blocks are hashed. In this case, the system will attempt to send the entire backup to the target in an undeduplicated format, which will take significantly more space, time, and replication bandwidth. No corrective action is needed.
Failed: Local backup status is failed. Cannot replicate failed backups. This error simply means that the replication subsystem evaluated the backup, but did not replicate it because it had a status of failed. No action is necessary.
Failed: Unable to send blocks. This replication will now retry This error indicates that there was a problem either sending blocks to the target, or an error when the target system was receiving and processing the blocks. If the error is network related, additional text will be displayed (see below). If not, the error is due to a condition on the target side. In most cases, the target problem is temporary (such as a high resource load, or a temporary shortage of backup space), and a retry will automatically resolve the problem. However, if the problem persists for an extended period of time, check that the target has sufficient resources to receive the backup.
Failed: Unable to send initial meta info. This replication will now retry This error indicates that the replicated backup cannot be created on the target. If the error is network related, additional text will be displayed (see below). Otherwise, this is likely due to a configuration mismatch between the source and target. Check to ensure the target is configured for replication properly and has its encryption setting matched to the source’s setting.
Failed: Unable to send final metadata. This replication will now retry This error is similar to the “unable to send blocks” error above, but occurs later in the replication, during the processing phase. This error indicates that all of the backup’s data blocks have been sent, but the metadata describing the ordering of the blocks in the backup could not be sent. As above, if the error is network related, additional text will be displayed (see below). If not, the error is due to a condition on the target side. In most cases, the target problem is temporary (such as a high resource load, or a temporary shortage of backup space), and a retry will automatically resolve the problem. However, if the problem persists for an extended period of time, check that the target has sufficient resources to receive the backup.
Failed: Failed building on the target This error indicates that all of the backups blocks and metadata have been successfully transferred to the target system, but the target system is unable to convert those blocks into a backup. A common reason this can happen is that encryption is enabled on the source system, but not the target system. If any source system has encryption enabled, the target system must also have encryption enabled. Another reason for this failure may be to lack of space in one of the D2D devices to store the backup file. Note that when using multiple D2D devices, all of them should be a similar size because the backup data is distributed evenly among them. One full D2D device can cause this error even if there is space in the other ones. Ensure that the target system has the proper encryption setting and adequate resources for the replications.
Failed: Backup is not in progress or complete on the target. This replication will now retry. This error indicates that the source system is out of sync with the status of the replication on the target. This may happen if activity on the target system causes communication to break (such as an upgrade or other maintenance activity on the target). The system will automatically recover by retrying.
Failed: Replication aborted by source. Cause: Failed. This error typically indicates that the replication was manually removed from the queue on the source, or that the replication was reset on the source without using the option to let current replications complete.
CANCELLED: Backup cancelled on the target. This error typically indicates that the replication was forcibly killed on the source system or the source system was shut down.
The following text may be appended to any of the error messages above, giving additional information about the errors:
Network: no error This message means that the root problem of the error condition is not network related
Network: couldn't connect to server This message means that there is a network connectivity issue. The replication protocol is RESTful, so connectivity between the source machine and port 80 (unencrypted) or port 443 (encrypted) is required for the protocol to work.
Network: failed writing received data to disk/application This message means that the data may have been corrupted during the network transfer. Please verify the health of your network link.
Network: server returned nothing (no headers, no data) This message means that the source sent a request to the target, but the target did not respond. Please check the target system for errors.