ReactiveUI.Testing With in SchedulerExtensions

[senften]

According to the documentation the "With" extension in the SchedulerExtensions for ReactiveUI.Testing says

public static void With<T>(this T scheduler, Action<T> block)
 in class ReactiveUI.Testing.SchedulerExtensions  T is TestScheduler 
With is an extension method that uses the given scheduler as the default Deferred and Taskpool schedulers for the given Action

so I am expecting the following testcase to pass with or without the scheduler being explicitly set

using System.Reactive.Linq;
using System.Reactive.Subjects;
using Microsoft.Reactive.Testing;
using NUnit.Framework;
using ReactiveUI.Testing;

namespace ReactiveUITestingTest;

[TestFixture]
public class SimpleSubjectShould
{

    [TestCase(false)]
    [TestCase(true)]
    public void ControlWithScheduler(bool observeOn)
    {

        new TestScheduler().With(scheduler =>
        {

            var sub = new Subject<Visibility>();

            var actualValue = Visibility.Hidden;

            if (observeOn)
            {
                sub.AsObservable().ObserveOn(scheduler).Subscribe(v => { actualValue = v; });
            }
            else
            {
                sub.AsObservable().Subscribe(v => { actualValue = v; });
            }

            TestContext.WriteLine("Clock {0}", scheduler.Clock);
            scheduler.Start();
            TestContext.WriteLine("Clock {0}", scheduler.Clock);

            Assert.That(actualValue, Is.EqualTo(Visibility.Hidden)); // default
            
            sub.OnNext(Visibility.Visible);
            sub.OnNext(Visibility.Collapsed);

            scheduler.AdvanceBy(1);
            TestContext.WriteLine("Clock {0}", scheduler.Clock);

            Assert.That(actualValue, Is.EqualTo(Visibility.Visible));

            scheduler.AdvanceBy(1);
            TestContext.WriteLine("Clock {0}", scheduler.Clock);

            TestContext.WriteLine("Clock {0}", scheduler.Clock);

           Assert.That(actualValue, Is.EqualTo(Visibility.Collapsed));
        });
    }

    private enum Visibility
    {
        Visible,
        Collapsed,
        Hidden
    }
}

but when the scheduler is not explicitly set, the test fails
Assert.That(actualValue, Is.EqualTo(Visibility.Visible))
Expected: Visible
But was: Collapsed

at ReactiveUITestingTest.SimpleSubjectShould.<>c__DisplayClass0_0.b__0(TestScheduler scheduler) in D:\senften\work\Brew\Reactive\ReactiveUITestingTest\ReactiveUITestingTest\SimpleSubjectShould.cs:line 46

Is my expectation/understanding wrong?