Guidelines for writing tests

The following are guidelines for writing quality tests for Hyperksill projects. Please do your best to stick to them.

1. Soften tests in places where it is not particularly important (in terms of program logic) what the user program outputs. Soft tests will increase the diversity of user solutions, and the softer the tests, the more solutions will differ from each other, and this is good. But the core logic of the program should always be tested completely and thoroughly.

2. When you split() a string, always check how many elements it contains, and if less than the required number, end the testing using WrongAnswer with the necessary feedback. ArrayOutOfBoundsException (IndexError) happens all the time in tests and it should be fixed in tests. Better to eliminate this kind of error beforehand.

3. The user may accidentally print a lowercased word rather than capitalized, and the testing should not result in WrongAnswer if the program works correctly in all other parts. It is recommended to check all strings in lower case (unless the case is absolutely important and is an essential part of program logic). NOTE: Keep original strings if you want to show them in the feedback. If you show a lowercased string in the feedback, the user will not realize that their program outputted that string and will be confused.

4. In addition to lower case, get rid of whitespace at the beginning and end of the string before comparing using the .strip() method. Excessive whitespace or \n characters will give users a lot of trouble when debugging.

5. Try to make sure that extra empty lines in any place do not affect the test result. Users like to add or remove empty lines to/from the output, they think it looks prettier that way. Extra or missing empty lines make the logic of the program remain unchanged. If empty lines are really necessary (for example, for convenient output parsing in tests), it is worth emphasizing to the user that their program should separate blocks of output with an empty line.

6. It is preferable to use the occurrence checker instead of the line equality checker. Try to check for occurrence of only the most important words in the output, not the whole sentence. Use .contains() instead of .equals() (Java) and in instead of == (Python).

7. The previous point allows users to pass the tests pretty easily just by printing every possible answer. For example, just by printing yes no the user can pass both the tests that search for yes and the tests that search for no because tests use contains instead of equals checks. So, you should check the output both for the existence of some answer and for the lack of any other answer.

8. Try to create comprehensible feedback that makes it clear to the user what went wrong. For example, Your answer is wrong or Incorrect output is an unclear feedback, better to show users what was expected and what the user program actually outputs.The better feedback would be The number of people is wrong, should be "45", but your program printed "44" - the user will immediately understand where to look for the error and what it might be.

9. Try to give the user different feedback for different errors. If the feedback is the same when checking different parts of the output, e.g. Incorrect Output, this means that the feedback is incomprehensible. Incorrect feedback can help users in comments, for example, where people have the same problem. If different people have different errors but the same feedback, then users can give each other advice in comments that won't work.