What is impacted?
This notice only applies to customers using the Google Workspace / G Suite Classic Sites feature
What is NOT impacted?
This notice DOES NOT apply to
- Dark Web Monitoring
- Auditing and Other Spanning System Components
What is happening?
Spanning has noticed a significant decrease in API capacity for Classic Sites. This is resulting in some sites backups failing or completing partially with the Spanning error code ‘99999’. We have validated that these errors are NOT due to product defects and confirmed that the API calls are being rejected. We will provide updates on this situation as it develops.
The specific error we are hitting is a 429 Error, “rateLimitExceeded”.
Why does Google Set Limits and Quotas?
It is common for Cloud platforms, Google Workspace included, to put limits in place to protect the overall health and performance of the service for all customers. These limits prevent programmatic errors, unplanned spikes in activity and malicious outsiders from negatively impacting the service.
Sometimes these limits can also hinder the performance of perfectly legitimate tasks and operations, like a backup or the migration of a site using scripts.
For more details on API limits and quotas you can visit: https://developers.google.com/admin-sdk/directory/v1/limits
Why is this happening now?
We believe this sudden rise in 429 errors is in response to users, third parties and others heavily using the Classic Sites technology to update, export and manage their Classic Sites migration to the new and improved Modern Team Sites experience in Google Workspace as the deadline fast approaches:
Starting September 1, 2021, Classic Sites will no longer be viewable by anyone. If you want your websites to continue to be viewable, you must convert and publish them to the new Google Sites experience before September 1, 2021.
Starting January 2022 (Previously December 2021) Google will begin the final phase of the sites transition. At this point:
- Classic Sites will not be available in any form
- Sites will be archived
- Sites will be moved to "Deleted Sites" and deleted after 30 days
Source: https://support.google.com/a/answer/9958187?hl=en , https://support.google.com/sites/answer/10598263?hl=en&ref_topic=6372873
What is Spanning doing?
About the 429 Errors
When this error is encountered it is recommended by Google that a service ‘back off’ and retry later when the service is less busy. Engineering best practices recommend a technique known as “exponential backoff”.
Spanning Backup has previously and will always use engineering best practices to efficiently call Google APIs. We implement exponential backoff as best practice and we have not made any modifications to our product code for Classic Sites backup.
We will continue to try to improve our retry logic and workaround the limitations we are encountering from the API
For more details on Exponential backoff: https://developers.google.com/admin-sdk/directory/v1/limits#backoff
About Classic Sites Deprecation
- Customers will still be able to export their Classic Sites backup data from Spanning Backup for at least 90 Days after the feature is completely deprecated by Google and after Spanning has launched Modern Sites Support to all customers
- Customers will still be able to backup Classic Sites data until 12/31/2021
Thank you for your patience. We will provide updates on this situation as it develops.