Scaling out

When it comes to managing thousands of virtual machines like BaaS providers do, it might then become difficult to simply configure and run a SureBackup against a linked job containing thousands of images. Attention must then be paid to timing and resource consumption.

Using linked jobs

A first option to scale out SureBackup jobs consists in leveraging the infrastructure resources at their maximum to test all the guests contained in the “linked job” attached to the job.

Here are a few guidelines to run the biggest possible SureBackup jobs.

Leverage non-production infrastructure: SureBackup allows to test the images on any host. This could be infrastructure dedicated to SureBackup, preproduction or development hosts.
Run heartbeat tests only: Commonly backup providers don’t have administrative credentials on the objects they manage. That prevents application testing.
Rely on fast read repository: SureBackup performance mostly depends on the boot time of the tested machines. That requires fast (random) read performance of the repository.

Dedicating backup jobs

The most critical guests might be added to small dedicated backup jobs comprising less than 20 guests to make it possible to spread the the linked jobs tests on the week days.

Using random guests in application groups

An other option is to daily randomly select a set of virtual machines while eliminating the ones already tested. These VMs are included in the application group and then tested.

A PowerShell script that implements that concept is available on Github.

Assuming you could test 100 machines daily, this would be 3000 VMs in 30 days.

You can also choose to make this even more simple and randomly select 10 VMs every day, getting a very good idea on possible backup corruption statistics while modifying the script.

Note: Per design, SureBackup fails at the point when one machine of an application group fails. That leaves other machines untested. Also, all VMs of the application group will stay powered on, until the SureBackup job finishes. This limits by design the number of machines to simultaneously test.

Source: VirtualToTheCore

Using custom scripts

To test any kind of application, custom scripts can be used. The custom script will be invoked on the Veeam Backup Server (launched by the account that controls the Veeam Backup Service), using the credentials specified in the SureBackup job settings.

The custom script will not be pushed on the tested server but will be invoked on the Veeam Backup Server.

Windows scripting

Authentication

Veeam Backup & Replication starts a new shell (cmd.exe) as the user running the Veeam Backup & Replication service (default being “Local System Account”) using the switch “/NETONLY” to use the specified credentials (e.g. database user in the tested environment) only when utilizing a remote connection. This is imposed by the fact that the credentials needed for testing (specified in the credentials configuration tab) might not be existent in the domain where Veeam Backup Services are running.

For example, application group and tested VMs might boot up on domain A, while Veeam backup is running inside domain B, A and B not being aware of each other.

Remote scripting

Another way to run application tests is to store the test script on the production machine. The test script will be backed up with the machine and is available for tests inside the virtual lab environment.

PS remoting and authentication

Another way to test an application in the virtual lab is to remotely launch a set of PowerShell commands on the client residing inside the virtual lab.

To do so, PS remoting and authentication has to be configured properly. The Veeam Bbackup server (where the remote shell is invoked) and tested guest are by definition considered as residing on different domains since the virtual lab appliance is routing all the traffic.

On the Veeam Backup & Replication, the tested guest translated IP must be added to the trusted hosts list.

set-item wsman:\localhost\Client\TrustedHosts -value <translated IP>

If necessary, a set of credentials can be passed to the script after being stored in an encrypted file stored on the backup server.

$Password="MyPassword"
$SecureStringPwd = $Password | ConvertTo-SecureString -AsPlainText -Force
$SecureStringText = $SecureStringPwd | ConvertFrom-SecureString
Set-Content "C:\tmp\ExportedPassword.txt" $SecureStringText

Note: Script will be launched by the user running Veeam Backup Service. In case it is the local system account psexec.exe (or scheduled task) shall be used to encrypt the password into a file accessible by the user.

Then, in the script itself, credentials can be used as described in below example.

$sqlclientinvlab = <translated IP of the server in virtual lab>
$UserName = <User name to invoke the commands inside the virtual lab>
$PwdTxt = Get-Content "C:\tmp\ExportedPassword.txt"
$SecurePwd = ConvertTo-SecureString -String $PwdTxt
$creds = New-Object System.Management.Automation.PSCredential -ArgumentList $UserName, $SecurePwd
...
Invoke-Command -Credential $creds -Computername $sqlclientinvlab -ScriptBlock {
...
Your powerShell script Here
...
}

Linux scripting

SSH connection

If the tested guest is running a Linux OS it is possible to use Plink.exe to remotely run commands. As part of the PuTTY toolbox, Plink is a remote command line interface for Linux guests. Putty toolbox is located on Veeam server directory: C:\Program Files\Veeam\Backup and Replication\Backup\Putty.

As it will connect to the tested guest through SSH, Plink will need to get and store the SSH key.

Note: to avoid sending clear passwords as arguments, SSH connection can be secured by using key generated with puttygen.exe and plink with the -i key option instead of -l user -pw password. Depending on the user running Veeam Backup, there will be two possibilities to store the keys:

If Veeam Backup is running as a service account Plink must be run manually under this account to initiate a first connection to the tested guest. This will store the SSH key in the service account registry. For example run a shell with “runas” and from there run Plink.
If Veeam Backup is running as local system account you will have to manually add the keys to the system user. To do so, you must do the following:
- Logon as any relevant user and run an SSH connection against the tested image.
- Using regedit export the SSH key stored in the user registry on the Veeam backup server.
- Edit the export file and replace the user with the default system user.
  - Before : [HKEY_CURRENT_USER\SOFTWARE\SimonTatham\PuTTY\SshHostKeys]
  - After : [HKEY_USERS\.DEFAULT\SOFTWARE\SimonTatham\PuTTY\SshHostKeys]
- Import the file back to the registry using the “import” function of regedit and check that the keys are stored in the HKEY_USERS\.DEFAULT\Software\SimonTatham\PuTTY\SshHostKeys container.

Note: If the key was not stored when the SureBackup job was run the job will endlessly wait for the SSH key acknowledgment in the running test script step. To exit such situation it is possible to manually kill the plink process on the Veeam Backup server to fail the SureBackup job.

SureBackup tips

Logfiles location

Log in into the vLab appliance

For troubleshooting purpose it is possible to login to the vLab Linux appliance. You can refer to this official KB article for further information.

Once logged in, users can observe NAT rules, IP addresses, and troubleshoot the virtual Lab configuration from the inside.

Note: Since each virtual lab reconfiguration will redeploy the virtual lab appliance any modification applied while logged on the appliance will be erased.