M365 Throttling

Microsoft might be throttling activity on the M365 APIs due to high load to protect the service quality for all customers. There are ways to identify whether your backup operations are throttled and ways to mitigate or disable throttling impact.

Most probably you will encounter throttling on the initial full backup and should not see it often in day to day incrementals.

Identify Throttling

Microsoft is communicating throttling through HTTP response codes like 500 (Internal Server Error) or 429 (Too Many Requests).

You can find the amount of throttled requests per application and day in Veeam.Archiver.Proxy logfiles:

[18.10.2022 14:32:03]   47 (6004) Accounts throttling statistics:
[18.10.2022 14:32:03]   47 (6004)  9f75f5ab-3971-4b48-97a6-943f95cdf4f8: requests 87998, throttled 1076
[18.10.2022 14:32:03]   47 (6004)  f6be4beb-014a-475e-a757-9b10e3230423: requests 88796, throttled 1209
[18.10.2022 14:32:03]   47 (6004)  fbc26ac0-849b-48df-8645-a86f7baa3af3: requests 89193, throttled 1155

Use the following PowerShell command to scan through the logs. Either do this on every proxy from the C:\ProgramData\Veeam\Backup365\Logs folder or within a log export which contains logs from all proxies.

Get-ChildItem -Recurse -Filter *.log | select-string "throttled [^0]"

On the main VB365 server you can also check the job logs by looking for log lines like

[18.10.2022 13:19:43]  139 (9648)  Web error occurred: 429 TOO MANY REQUESTS (status: ProtocolError)

You can use this PowerShell command to find them easily from the VB365 log folder at C:\ProgramData\Veeam\Backup365\Logs:

Get-ChildItem -Recurse -Filter *.log | select-string "too many requests"

Add | Measure-Object -Line when you’re only interested in the amount of throttled requests or modify the path or filter when you’re only interested in a single job or single job run.

Mitigate Throttling

The throttling that Microsoft applies has different shapes based on the service and authentication methods.

Exchange Online

The Exchange Online throttling threshold is based on user mailboxes, so you likely encounter throttling when you have very large single mailboxes. It is possible to disable the EXO throttling via M365 self-service for up to 90 days. The detailed procedure how to do this is described in Veeam KB4198.

SharePoint Online & OneDrive for Business

SharePoint Online and OneDrive for Business (which is based on the former) is throttled based on the account which is accessing the SPO resources.

As explained within the authenitication section you should use Modern-Only authentication. When you’re using modern-only authentication with backup applications to add the Microsoft 365 organization, it’s not recommended by Microsoft to use multiple applications (AppIDs) as it ended up exhausting the tenant’s resource and causing multiple applications to be throttled in the tenant.

Check out about Backup Applications and Backup Accounts in the Organization Config section.

Resources

• Veeam KB4198 - How to temporarily disable Exchange Web Services throttling
• Veeam Help Center - Backup Applications
• Veeam Help Center - Backup Accounts
• Microsoft Documentation - Throttling explained