Failed to evaluation on driving-with-language with "[ERROR] Evaluation failed 'choices'"

[dszpr]

Hi, many thanks for the great dataset and competation.

I tried to make 3 submissions today on driving-with-language leaderboard. However, all submissions encountered "[ERROR] Evaluation failed 'choices'" and I failed to get a score. Could you please help figure out what’s going wrong?

Looking forward to your reply!

[ChonghaoSima]

Could you provide your submission id? We will check it later.

…

On Thu, 12 Dec 2024 at 10:41, dszpr ***@***.***> wrote: Hi, many thanks for the great dataset and competation. I tried to make 3 submissions today on driving-with-language leaderboard. However, all submissions encountered "[ERROR] Evaluation failed 'choices'" and I failed to get a score. Could you please help figure out what’s going wrong? Looking forward to your reply! — Reply to this email directly, view it on GitHub <#135>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM6RGLSXGT2ZE2Z44EMZ6HL2FDZVDAVCNFSM6AAAAABTOZQVTOVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4ZTINJXGEZDMOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dszpr]

The 3 failed submission are listed below:
0cbe734c0f86eb8b0e02b93647ff2e7e29da7f7b__d0ab1914-c257-44a0-bec0-7184503bec33
fdaa9313e5953c6308909084a28177d2b4d411e1__e5914edc-164a-41bb-8127-fb7206d8314f
6ccc0746a3fb93bdd347681a52ba04a1fff4a4eb__5f73c46b-5601-4f30-b280-0061020f2056

I have also sent an email to you this morning, you may find it in your mail box.

[ChonghaoSima]

Should be fixed now. I tested with the baseline submission and it works.

[dszpr]

Hi Chonghao, it seems the test server meet some error again. I submitted a json file just now and get an [ERROR] Evaluation failed. 'choices' The team name in your submission is [test12].

The submission ID is 7b623ecc82957c86f4744acf0755ffa4c54fe68b__f26c98d9-62bd-43ee-8bed-2d5b75a263d3.

Could you please help figure out what’s going wrong?

Looking forward to your reply!

[aurora-xin]

Same question. I submitted just now and received an [ERROR] Evaluation failed. 'choices'
My submission ID is a6ca59d3f5e4f11860875a2a96cd5184c92a4380__c094ba5c-addb-4f66-9528-71a2d2377597.

[ChonghaoSima]

We are replacing the api key, should be fixed soon

[dszpr]

@ChonghaoSima
Hi Chonghao, really sorry to keep bothering you these days.

We are working on our own VLM and use DriveLM as one of the benchmarks. So we frequently submitted our results and it seems that we run out of your OpenAI tokens every day. And "[ERROR] Evaluation failed. No available key left! " happened again just now.

Could you please check it and replace the api key?

Besides, do you have a plan to disclose the labels of the DriveLM val set? If so, we are able to evaluate by ourselves and reduce your maintenance pressure‌.

Thanks and apologize for the bothering again!

[ChonghaoSima]

We do meet some problem in the stability of those keys and the server and keep fixing them with the key provider. Really apologize for the inconvience. For now we don't have a plan to release the label as we want to keep them confidential so the results are grounded.