FasterKv out of memory exceptions

[HenkOtterloo]

I have a very simple program that uses 'large' classes, I created a test program to figure out how to get it up and runing with FasterKv.
Unfortunatly I keep runing into out of memory exceptions, I can't figure out what I'm doing wrong.
Help will be appreciated.

private static void Main()
{
Console.WriteLine("This is a test program for FasterKv to test how the log works in case of heavy memory consumption");
Console.WriteLine("I expected the usage of temporary disk log to prevent out of memory exceptions (persistence is not needed)");
Console.WriteLine("This example uses .Net framework 4.8, and uses X64 solution");
Console.WriteLine("The program runs on 32 GB ram laptop, with 25 GB free memory");
Console.WriteLine();
Console.WriteLine("Issue 1) it consumes all ram >> This is behavior that I didn't expect");
Console.WriteLine("Issue 2) it lead to on out of memory exception");
Console.WriteLine();
MyInMemoryWithLogForOverflow();
Console.WriteLine("Press to end");
Console.ReadLine();
}

    /// <summary>
    /// Just a simple class with a very large string value to force memory consumption.
    /// </summary>
    internal class ThingClass : IThing
    {
        public ThingClass() { }
    
        public ThingClass(long key)
        {
            Key = key;
            // Create large strings for heavy  memory load
            Load = new string('$', 1_000_000);
        }

        public long Key { get; set; }

        public object Load { get; set; }
    }

    /// <summary>
    /// Runs the application in transient mode and only use a log to prevent out of memory exceptions.
    /// </summary>
    private static void MyInMemoryWithLogForOverflow() 
    {
        const int maxNumberOfKeys = 1_000_000;
        Console.WriteLine(new string('-', 100));
        Console.WriteLine($"Load : {maxNumberOfKeys:#,##0} of type {nameof(ThingClass)} values into the dictionary");
        var settings = new LogSettings()
        {
            // The main log.
            LogDevice = Devices.CreateLogDevice(@"C:\Tmp\FasterKv\myLog.log"),
            // Class storage
            ObjectLogDevice = Devices.CreateLogDevice(@"C:\Tmp\FasterKv\myLog.obj.log"),
        };
        var store = new FasterKV<long, ThingClass>(1L << 20, settings);
        var functions = new SimpleFunctions<long, ThingClass>((a, b) => b);
        using (var session = store.NewSession(functions))
        {
            long key;
            var stopwatch = Stopwatch.StartNew();
            for (var i = 0; i < maxNumberOfKeys; i++)
            {
                key = i;
                var value = new ThingClass(key);
                session.Upsert(ref key, ref value);
            }

            stopwatch.Stop();
            Console.WriteLine($"Nr of values {maxNumberOfKeys:###,0}, duration {stopwatch.Elapsed}");
            stopwatch.Restart();
            key = 912_342;
            var found = session.Read(key);
            stopwatch.Stop();
            Console.WriteLine($"{key} {found}, duration {stopwatch.Elapsed}");
        }
        settings.LogDevice.Dispose();
        settings.ObjectLogDevice.Dispose();
        Console.WriteLine(new string('=', 100));
    }

[badrishc]

See this doc and sample: https://microsoft.github.io/FASTER/docs/fasterkv-tuning/#log-memory-size-with-c-heap-objects

In general, with objects you need to tune the page size and memory size to be much smaller. E.g., 1KB pages, and say 64 such pages in memory. This is because the long only has 8 byte pointers to the actual data, thus the log size in memory only controls the number of pointers (references) in memory.