Paging through a considerable amount of queues in a namespace results in inconsistent page- and total item count results

[sergevm]

Description

I have been building a little service bus explorer tool that runs on the major operating systems, and that has proven useful in my day job. The tool is using the Azure.Messaging.Servicebus nuget package to work on messages etc., but also to list and manage storage resources. One of the features that is very helpful, is the ability to filter on queue / topic / subscription names.

Older SDK packages (predecessors), which supported server-side filtering, are deprecated, and unfortunately the current SDK does not support server-side filtering, so I had to fall back to client side filtering instead.

Unfortunately, I found that, in case of a namespace with a lot of queues e.g. (+1900 in my case), the AsyncPageable result does not provide consistent pagination results: both the number of pages returned and the total number of resources (queues e.g.) varies on subsequent runs, which makes the search results inconsistent.

I got a suggestion to use Azure.ResourceManager.Servicebus, but it exposes the same behavior it seems. To verify this, I used a minimal console app, where I simply list and count the queues in a namespace:

// See https://aka.ms/new-console-template for more information

using Azure.Core;
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.ServiceBus;

var armClient = new ArmClient(new InteractiveBrowserCredential(
    "[TENANT ID]",
    "[ENTRA APP ID]",
    new InteractiveBrowserCredentialOptions
    {
        RedirectUri = new Uri("http://localhost")
    }));

var namespaceResource = armClient.GetServiceBusNamespaceResource(new ResourceIdentifier($"/subscriptions/[SUB ID]/resourceGroups/[RG ID]/providers/Microsoft.ServiceBus/namespaces/[NS NAME]"));
var asyncPageable = namespaceResource.GetServiceBusQueues().GetAllAsync();
var count = 0;
await foreach (var queue in asyncPageable)
{
    count++;
    Console.WriteLine(queue.Data.Name);
}

Console.WriteLine($"Total queues: {count}");

Actual Behavior

1. Subsequent runs show a different count of queues

Expected Behavior

1. Subsequent runs show a consistent count of queues

[SeanFeldman]

While I understand the suggestion to treat entity listing as a resource operation and use a separate library, there are several scenarios when using a single library to perform operations on entities and messages that are valuable. A good example of that is Service Bus Explorer. I'd love to see this issue being resolved in a way that allows the main library perform entities listing.

[EldertGrootenboer]

Thank you for your feedback. We have opened an investigation task for this in our backlog, and will update this issue when we have more information.

[EldertGrootenboer]

This item in our backlog, however we currently don't have an ETA on when development might start on this. For now, to help us give this the right priority, it would be helpful to see others vote and support this item.

[kimsey0]

We are also affected by this bug and would very much like to see it addressed. We originally thought it was an issue with the Azure SDK and filed it in the relevant repository: Azure/azure-sdk-for-net#37985

@EldertGrootenboer: We found that throttling our paging fixes the issue, which might suggest there's some timing component to this. The relevant sample code from the linked issue:

public async Task GetAllQueues()
{
    var armClient = new ArmClient(new DefaultAzureCredential(new DefaultAzureCredentialOptions { TenantId = TenantId }));
    var subscription = await armClient.GetDefaultSubscriptionAsync();
    var resourceGroupResponse = await subscription.GetResourceGroupAsync(ServiceBusResourceGroup);
    var resourceGroup = resourceGroupResponse.Value;
    var namespaceCollection = resourceGroup.GetServiceBusNamespaces();
    var serviceBusNamespace = await namespaceCollection.GetAsync(ServiceBusNamespace); // Existing service bus namespace with 3.000 queues.

    var serviceBusQueues = await serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync().ToListAsync();
    Console.WriteLine($"Count without waiting: {serviceBusQueues.Count}"); // Example run: Count without waiting: 1918
    
    // Wait a minute to reset the throttling
    await Task.Delay(TimeSpan.FromMinutes(1));

    var serviceBusQueues2 = new List<ServiceBusQueueResource>(); 
    await foreach (var queue in serviceBusNamespace.Value.GetServiceBusQueues().GetAllAsync())
    {
        serviceBusQueues2.Add(queue);
        await Task.Delay(10); // To avoid throttling
    }
    Console.WriteLine($"Count with waiting: {serviceBusQueues2.Count}"); // Example run: Count with waiting: 3000
}

[kimsey0]

Also of note is that the skip value often starts jumping around 1500 to 2000 queues in, then continues being irregular for the rest of the enumeration. That makes it easiest to reproduce in a namespace with 3000 or more queues. For us, it's not an intermittent issue, but happens 100% of the time for namespaces with enough queues.