use tree-sitter things to make syntactically-correct code transformations and see if the tests fail

[silky]

they probably should.

[utdemir]

Sounds like Mutation Testing, but likely tree-sitter can make them extra fancy.

Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways.[1] Each mutated version is called a mutant and tests detect and reject mutants by causing the behaviour of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants.