Metrics Are Not Accurate

[nnikolov3]

The metrics do not represent the real code accepted. I think, you should count how many times it says "I am sorry" or "I understand your frustration ".

It gets in a loop making changes, apologizing , redo, the undo many times and counts it as 100% success and acceptance.

But did not write a single line of useful code, rather it created bugs. Yet I was billed for all the tokens in the process.

In other words, 90% accepted, without anything to show, plus more bugs. An odd paradox.

[gemini-cli]

This issue reports inaccurate metrics and poor model performance, including generated code quality and failing to follow instructions. It is classified as a medium-priority bug under the models area. Adding 'status/need-information' as the user has not provided the output of the /about command.

[nnikolov3]

I will provide more information.

One way to reproduce it is by setting up a linter with more strict rules.
For example, the cyclomatic complexity in my Go implementations is 5. Which is generally very strict. The linter will flag implementations with cyclomatic complexity above 5.

I will ask the agent to "fix ONE SINGLE linter issue at a time, lint, test, and if the linting issue is gone, go to the next. If not gone, refactor again until the issue is fixed"

Then it will refactor the code, but add more (linting issues), then it will try solving the 'new' linting issues it introduced. It fixes them, but again hits the same issue with the cyclomatic complexity. So it erases it, starts over. In the entire process it is working ON THE SAME file. Once I give it access to the file, every edit counts as a acceptance. In 30 minutes, it more or less uses 2/3 of the daily quota, until I stop it with Ctrl+C.

Generally, the agent will have access to entire source, and I will provide (in GEMINI.md) the design principles and coding standards from here https://github.com/nnikolov3/docs/tree/main. Usually, it will get all the information from the linter, like line number, and what is failing. I add context too, as what is the purpose, what is the objective etc.

I have to say, it seems it is not the model the issue.

These same issues were with Claude-code, and Cursor. This is why I switched to Gemini.

However, it seems everyone is using the same MCP setup with nodejs. ALL, regardless if it is Gemini-2.5-pro or Opus (Anthropic) focus mainly on 'getting work done'. It does not necessarily matter if it is completed successfully. For example, Gemini will print some nice check marks and say "I have done this and this and now your code is production ready" . So the tool seems to not have the notion of completing work correctly and providing a working solution.

What is more, often it will prefer to use 'hacks' to solve the problem. Like instead of refactoring the code and eliminate the error or linter failure, it will add something like nolint. Instead of fixing a test or creating a new one, it will create a test, and then just add a 'pass'. Despite that in the GEMINI.md is explicitly stated nolint is forbidden, and it should be making single change at a time, it still updates the entire file with all sort of changes.

I am aware that LLMs can make up information, and it is not entirely a bug. On the other hand, something is not right with this MCP approach everyone is using. OR it is a business approach to make people pay more.

[gemini-cli]

Hello! As part of our effort to keep our backlog manageable and focus on the most active issues, we are tidying up older reports.

It looks like this issue hasn't been active for a while, so we are closing it for now. However, if you are still experiencing this bug on the latest stable build, please feel free to comment on this issue or create a new one with updated details.

Thank you for your contribution!