[CSS] Throttling during provisioning

[kyichii]

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Run the following command
docker run -it -e "IotHubConnectionString=[redacted]" -e DeviceCount=1000 mcr.microsoft.com/oss/azure-samples/azureiot-simulatordeviceprovisioning

Provisioning crashes with a ThrottlingBacklogTimeout that causes the script to stop running.

Any log messages given by the failure

PS C:\Users\kyichii>
Starting device provisioning
2021-10-12T07:17:21.6511837Z: 100 devices have been created @ 63.20/sec
2021-10-12T07:17:21.8319529Z: 200 devices have been created @ 553.18/sec
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:55f0784980964ecf83ec9bf4cce2bc7c-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:3ea3022e253e4d7b95ae629f77c7c403-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:474cf77e1c214b409f7429e4ed439825-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:a6b0b0fd21ab40c0a341fbe65a366d54-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163

Expected/desired behavior

This should not crash when throttling. Ideally it would allow us to set the SKU rate for the IoT Hub to avoid the throttling all together and test provisioning at the maximum limits of a specific iot hub

OS and Version?

Windows 10

Versions

Latest

Mention any other details that might be useful

This is behavior that inconsistently crashes but consistently throttles on lower end IoT Hub due to the registry operations. I am also attempting to get some additional logs from customers that are encountering the same issuje.

[kyichii]

Hello team, are there any updates to this issue?