bug: [0.5.8] OpenAI .Net library chokes on "finish_reason": null in response from Jan server

[debruyckere]

Jan version

v0.5.6

Describe the Bug

The response from the Jan server looks like so (from the built-in example):

{
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "message": {
        "content": "Hello user. What can I help you with?",
        "role": "assistant"
      }
    }
  ],
  "created": 1700193928,
  "id": "ebwd2niJvJB1Q2Whyvkz",
  "model": "_",
  "object": "chat.completion",
  "system_fingerprint": "_",
  "usage": {
    "completion_tokens": 500,
    "prompt_tokens": 33,
    "total_tokens": 533
  }
}

It is parsed by the OpenAI library (https://www.nuget.org/packages/OpenAI, v2.1.0-beta). The response contains 'null' for 'finish_reason' and the library attemps to parse that in ChatFinishReasonExtensions.ToChatFinishReason, using this code:

public static ChatFinishReason ToChatFinishReason(this string value)
{
    if (StringComparer.OrdinalIgnoreCase.Equals(value, "stop")) return ChatFinishReason.Stop;
    if (StringComparer.OrdinalIgnoreCase.Equals(value, "length")) return ChatFinishReason.Length;
    if (StringComparer.OrdinalIgnoreCase.Equals(value, "tool_calls")) return ChatFinishReason.ToolCalls;
    if (StringComparer.OrdinalIgnoreCase.Equals(value, "content_filter")) return ChatFinishReason.ContentFilter;
    if (StringComparer.OrdinalIgnoreCase.Equals(value, "function_call")) return ChatFinishReason.FunctionCall;
    throw new ArgumentOutOfRangeException(nameof(value), value, "Unknown ChatFinishReason value.");
}

Consequently, an ArgumentOutOfRangeException is thrown because the value is null. The expected value is 'stop'.

Steps to Reproduce

1. Start local server with 'llama3.2-3b-instruct' model
2. Run this code in .Net, using the official OpenAI library:

var uri = new Uri("http://127.0.0.1:1337/v1");
var credential = new ApiKeyCredential("123");
var client = new ChatClient("llama3.2-3b-instruct", credential, new OpenAIClientOptions {Endpoint = uri});

var messages = new ChatMessage[] { ChatMessage.CreateUserMessage("Say something") };
var response = client.CompleteChat(messages);

Screenshots / Logs

System.ArgumentOutOfRangeException: Unknown ChatFinishReason value.
Parameter name: value
    at OpenAI.Chat.ChatFinishReasonExtensions.ToChatFinishReason(String value)
   at OpenAI.Chat.InternalCreateChatCompletionResponseChoice.DeserializeInternalCreateChatCompletionResponseChoice(JsonElement element, ModelReaderWriterOptions options)
   at OpenAI.Chat.ChatCompletion.DeserializeChatCompletion(JsonElement element, ModelReaderWriterOptions options)
   at OpenAI.Chat.ChatCompletion.FromResponse(PipelineResponse response)
   at OpenAI.Chat.ChatClient.CompleteChat(IEnumerable`1 messages, ChatCompletionOptions options, CancellationToken cancellationToken)
   at OpenAI.Chat.ChatClient.CompleteChat(ChatMessage[] messages)

What is your OS?

• MacOS
• Windows
• Linux

[louis-jan]

Hi @debruyckere, please update to version 0.5.8 and see if the issue persists.

[debruyckere]

Same behavior in 0.5.8.

[louis-jan]

@debruyckere Could you kindly share the stream response on your end?

[debruyckere]

The JSON returned by the server is in the description. Or are you referring to something else?

[louis-jan]

The JSON returned by the server is in the description. Or are you referring to something else?

Yes, @debruyckere, please ensure that the JSON response from the server indicates a successful request.

[louis-jan]

Hi @debruyckere, it would be great to have the response json so that we can take a look at.

[debruyckere]

Using v0.5.11 I can no longer reproduce the issue, thanks for fixing!

The API Playground documentation still mentions 'null' for 'finish_reason', but that is no big deal of course:

[louis-jan]

Thanks @debruyckere, we are going to replace that swagger page with a better one. #3895