Unable to add proto files from nested directories used in imports

[siewers]

When I add proto definitions, the WireMockProtoFileResolver will only add the files if the filename (the first line comment in each proto file) does not contain invalid file name characters.
However, when imports contain relative paths to other proto files, it fails to find them because they are not added.

I'm testing WireMock using the OpenTelemetry proto files, but they are organized in the following directory structure, which fails to work with WireMock.
I'm manually parsing each proto file and injecting the filename as the first line comment in each file; however, it doesn't work since the filename has to match the one in the imports, and I'm not allowed to use invalid filename characters, which the forward slash (/) is.

opentelemetry
└───proto
    ├───collector
    │   │
    │   ├───metrics
    │   │   └───v1
    │   │           metrics_service.proto
    │   │
    │   ├───profiles
    │   │   └───v1development
    │   │           profiles_service.proto
    │   │
    │   └───trace
    │       └───v1
    │               trace_service.proto
    │
    ├───common
    │   └───v1
    │           common.proto
    │
    ├───metrics
    │   └───v1
    │           metrics.proto
    │
    ├───profiles
    │   └───v1development
    │           profiles.proto
    │
    ├───resource
    │   └───v1
    │           resource.proto
    │
    └───trace
        └───v1
                trace.proto

Can I make this work, or do I need to find another tool for the job?

I was thinking about creating a custom IProtoFileResolver implementation, but the WireMockProtoFileResolver is hard coded in ProtoBufMatcher.DecodeAsync, and I don't want to reimplement the entire thing as it seems like a missing feature in WireMock.

[StefH]

I think some changes can be done to WireMockProtoFileResolver:

• Use GetInvalidPathChars instead of GetInvalidFileNameChars
• only strip the comment characters and some spaces and keep the rest

This would allow:

• "// \\path\\filename.proto"
  or
• "// /path/filename.proto"

[siewers]

That would be great.
Would it be possible to provide the proto definitions with the file path expected in the imports?

If I have an import referencing protos/v1/something.proto, it would be great if I could create a dictionary with these files upfront, like this:

Dictionary<string, string> protoDefintions = 
[
    { "protos/v1/something.proto", "proto file contents" }
]

Or perhaps using an type:

var protoDefinitions = new ProtoDefintions
{
    new ProtoDefinition(filePath: "protos/v1/something.proto", fileContents: "proto file contents")
}

This could perhaps even be extended to something like this:

var protoDefinitions = ProtoDefinitions.FromDirectory(directory: pathToProtoFiles, relativeTo: Directory.GetCurrentDirectory());

That way I wouldn't need to manually "hack" the files to provide it with a filename inside the file contents itself.

[StefH]

The last one is the most user friendly I guess. The ProtoDefinitionHelper.FromDirectory(...) method returns a dictionary which should be usable in WireMock. However, this needs a lot a changes.

So maybe adding a method ProtoDefinitionHelper.FromDirectory which modified the files by adding the first line comment, and then returning a IReadOnlyCollection<string> protoDefinitions with all the modified protos --> this only needs this new method.

[siewers]

That would be an acceptable compromise I'd say 👍
The comments themselves are harmless and as long as you are not inspecting the in-memory data yourself, there should be no confusion.

[siewers]

On a possibly unrelated note, would it be possible to get access to the deserialized proto instance through the log entries? I'm using this to assert the received messages in WireMock, but I cannot find a way to get the deserialized instances.

Let me know if I should create a separate issue for this.

[StefH]

On a possibly unrelated note, would it be possible to get access to the deserialized proto instance through the log entries? I'm using this to assert the received messages in WireMock, but I cannot find a way to get the deserialized instances.

Let me know if I should create a separate issue for this.

By deserialized, do you mean the bytes or the json?

[StefH]

That would be an acceptable compromise I'd say 👍 The comments themselves are harmless and as long as you are not inspecting the in-memory data yourself, there should be no confusion.

I think this PR should fix it, can you review?
#1263

Preview version is: 1.7.4-ci-19802 (https://github.com/WireMock-Net/WireMock.Net/wiki/MyGet-preview-versions)

[siewers]

On a possibly unrelated note, would it be possible to get access to the deserialized proto instance through the log entries? I'm using this to assert the received messages in WireMock, but I cannot find a way to get the deserialized instances.

Let me know if I should create a separate issue for this.

By deserialized, do you mean the bytes or the json?

I would like to have access to the deserialized JSON from the gRPC type.
I guess it isn't realistic to provide access to the gRPC instance as the types are generated dynamically, but perhaps available as a dynamic object could be helpful?

[StefH]

On a possibly unrelated note, would it be possible to get access to the deserialized proto instance through the log entries? I'm using this to assert the received messages in WireMock, but I cannot find a way to get the deserialized instances.
Let me know if I should create a separate issue for this.

By deserialized, do you mean the bytes or the json?

I would like to have access to the deserialized JSON from the gRPC type. I guess it isn't realistic to provide access to the gRPC instance as the types are generated dynamically, but perhaps available as a dynamic object could be helpful?

Should already work?

[siewers]

I mean the request message. When I make a gRPC request to WireMock, I don't see the JSON unless I set the transformer on the response.

[StefH]

I mean the request message. When I make a gRPC request to WireMock, I don't see the JSON unless I set the transformer on the response.

OK, in that case, it;s better to create a new issue.

[siewers]

I must be doing something wrong. The new version works, but I'm confused about how to use it.

First thing, the imports in the proto files use a forward slash (/) and not a back slash (\), like this:

import "opentelemetry/proto/logs/v1/logs.proto";

This could be solved by normalizing the path separators in the WireMockProtoFileResolver.Exists method. I'm unsure if this path separator is the proto file convention, but on Windows, the implementation in the ProtoDefinitionHelper doesn't work.

It also only seems to work if I add the proto definitions at the request level, but then I need to specify the "primary" proto file as the first one.

With OpenTelmetry, there are multiple primary files, so I cannot access them with this new implementation.

This is what I'm currently doing:

internal sealed class OpenTelemetryMockServer : IDisposable
{
    private readonly WireMockServer _server;

    public OpenTelemetryMockServer(ITestOutputHelper? testOutputHelper = null)
    {
        var serverSettings = new WireMockServerSettings
        {
            UseHttp2 = true,
        };

        if (testOutputHelper is not null)
        {
            serverSettings.Logger = new TestOutputHelperWireMockLogger(testOutputHelper);
        }

        var openTelemetryProtoDefinitionsPath = Directory.GetCurrentDirectory();

        var protoDefinitionsMap = Directory.EnumerateFiles(openTelemetryProtoDefinitionsPath, "*.proto", SearchOption.AllDirectories)
                                           .ToDictionary(Path.GetFileNameWithoutExtension, ParseProtoFile);

        _server = WireMockServer.Start(serverSettings);

        var defaultResponse = Response.Create()
                                      .WithTransformer()
                                      .WithStatusCode(200);

        _server.Given(
                   Request.Create()
                          .WithPath("/opentelemetry.proto.collector.trace.v1.TraceService/Export")
                          .WithBodyAsProtoBuf([
                              protoDefinitionsMap["trace_service"],
                              protoDefinitionsMap["trace"],
                              protoDefinitionsMap["resource"],
                              protoDefinitionsMap["common"],
                          ], "OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest")
               )
               .RespondWith(defaultResponse);

        _server.Given(
                   Request.Create()
                          .WithPath("/opentelemetry.proto.collector.metrics.v1.MetricsService/Export")
                          .WithBodyAsProtoBuf([
                                  protoDefinitionsMap["metrics_service"],
                                  protoDefinitionsMap["metrics"],
                                  protoDefinitionsMap["resource"],
                                  protoDefinitionsMap["common"],
                              ],
                              "OpenTelemetry.Proto.Collector.Metrics.V1.ExportMetricsServiceRequest")
               )
               .RespondWith(defaultResponse);

        _server.Given(
                   Request.Create()
                          .WithPath("/opentelemetry.proto.collector.logs.v1.LogsService/Export")
                          .WithBodyAsProtoBuf([
                                  protoDefinitionsMap["logs_service"],
                                  protoDefinitionsMap["logs"],
                                  protoDefinitionsMap["resource"],
                                  protoDefinitionsMap["common"],
                              ],
                              "OpenTelemetry.Proto.Collector.Logs.V1.ExportLogsServiceRequest")
               )
               .RespondWith(defaultResponse);
    }

    public string Endpoint => _server.Url!;

    public void Dispose()
    {
        _server.Dispose();
    }

    public IReadOnlyList<JObject> GetTelemetryItems()
    {
        return [.. _server.LogEntries.Select(entry => entry.RequestMessage.BodyAsJson).OfType<object>().Cast<JObject>()];
    }

    private string ParseProtoFile(string filePath)
    {
        var sb = new StringBuilder(File.ReadAllText(filePath));

        sb.Insert(0, Environment.NewLine);
        var relativeProtoFilePath = Path.GetRelativePath(Directory.GetCurrentDirectory(), filePath)
                                        .Replace("\\", "/");

        sb.Insert(0, $"// {relativeProtoFilePath}"); // Used by WireMock to identify the file

        return sb.ToString();
    }
}

[StefH]

Thanks for testing.

1. imports in the proto files indeed should use a forward slash (/), this is fixed, please test the latest preview.
2. Currently, the first file is skipped because it's assumed to be the main proto file.
   But as you noticed, this is not correct.
   A solution could be to add an extra argument to FromDirectory(string directory) to indicate which proto file is the main proto, and make sure to add this as first (and do not add a comment)

What do you think?