aspnetcore on linux-arm64, crash on startup "Could not load file or assembly Datadog.Trace"

[gfody]

Describe the bug
I'm trying to use Datadog.Trace.Bundle to simplify my deployment process, but my site crashes on startup with:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Datadog.Trace, Version=3.6.1.0, Culture=neutral, PublicKeyToken=def86d061d0d2eeb'. The system cannot find the file specified.
File name: 'Datadog.Trace, Version=3.6.1.0, Culture=neutral, PublicKeyToken=def86d061d0d2eeb'

To Reproduce
Steps to reproduce the behavior:

I followed the examples from here and setup my environment like so:

CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
CORECLR_PROFILER_PATH=/myapp/datadog/linux-arm64/Datadog.Trace.ClrProfiler.Native.so
DD_DOTNET_TRACER_HOME=/myapp/datadog
DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=Serilog

my project targets net8.0 and has these packagerefs

    <PackageReference Include="Datadog.Trace.Bundle" Version="3.6.1" />
    <PackageReference Include="Serilog.AspNetCore" Version="9.0.0" />
    <PackageReference Include="Serilog.Sinks.Datadog.Logs" Version="0.5.4" />

my initialization looks like this:

    Tracer.Configure(TracerSettings.FromDefaultSources());

    builder.Services.AddLogging(config =>
        config.AddSerilog(new LoggerConfiguration().WriteTo.DatadogLogs(apiKey).CreateLogger()));

    var app = builder.Build();

the new LoggerConfiguration() triggers the FileNotFoundException

if I copy the files from of /myapp/datadog/net6.0 and /myapp/datadog/linux-arm64 into /myapp then it starts and works fine

Expected behavior
site should just work?

Runtime environment (please complete the following information):

• Instrumentation mode: manual with NuGet package
• Tracer version: 3.6.1
• OS: amazon linux 2023
• CLR: net8.0

[andrewlock]

Hi @gfody, sorry to hear you're having issues.

A couple of clarifications - you've shown the ENV VAR configuration that's required for .NET Framework (COR_ variables) but you actually need the .NET Core variables (CORECLR_). I assume that's just a typo in the issue, as otherwise I wouldn't expect anything to work.

Aside from that, I wasn't able to reproduce the bahaviour you're describing. Would it be possible to provide a minimal repro so that we can look into it?

Alternatively, could you contact support and they will walk through collecting logs etc so that we can figure out what's going on here. Thanks!

[gfody]

ah yes here's my actual config now after reverting (I had changed the profiler_path and tracer_home to point to my apps install directory where Datadog.Trace.Bundle was) .. there must be some other precondition because after I rebooted the machine I stopped being able to reproduce it. If I can get a minimal repro I'll update here

Environment=CORECLR_ENABLE_PROFILING=1
Environment=CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
Environment=CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
Environment=DD_DOTNET_TRACER_HOME=/opt/datadog
Environment=DD_RUNTIME_METRICS_ENABLED=true
Environment=DD_ENV=development
Environment=DD_SERVICE=myservice
Environment=DD_VERSION=xxxxxxx
Environment=DD_LOGS_INJECTION=true
Environment=DD_DYNAMIC_INSTRUMENTATION_ENABLED=true
Environment=DD_SYMBOL_DATABASE_UPLOAD_ENABLED=true

this might actually be a bug in Serilog, it seems not to repro w/o DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=Serilog (which also seems unnecessary - the logs and traces are linking fine w/o it)

[andrewlock]

this might actually be a bug in Serilog, it seems not to repro w/o DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=Serilog (which also seems unnecessary - the logs and traces are linking fine w/o it)

Oh, that's interesting information, thanks! The DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS variable is definitely not required for your setup, and in fact is probably "harmful" 😄 That variable enables you to send logs directly to Datadog, without using the Serilog.Sinks.Datadog.Logs package (behind the scenes it uses automatic instrumentation to do effectively the same thing as that package).

If you can confirm that enabling that key still causes the issue, that would be great, but I'm still not able to reproduce the issue locally...

The only scenario we can think where you could get into this problem would be if you had two versions of the tracer loaded 🤔 I wonder if there are some "left over" dlls lying around in the output folder causing some strange issues (I would love to know if so, so we can replicate it!)

If you can still replicate it, it would be useful if you could raise a support ticket, so that we can see all the log files so that we can get to the bottom of this 🙂