-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to write tests #269
Comments
A related question: every time I create a gridworld I seem to get different IDs |
To summarize your question; How can I test MATRX code that depends on complex interactions between objects/agents and the grid world? Where The quick and clear answer is that supporting MATRX users in simplified and automated testing of their world is not easily possible, and is unlikely to be supported in future MATRX releases. Below I outline some potential solutions and argue why they will not actually solve the issue. Potential Solution 1 Potential Solution 2 In the case of the So even if we implement methods that allow the creation of a Potential Solution 3 Ofcourse it might be possible I missed an approach, so if I did; please say so! :) |
This is by (crappy) design; the However, we acknowledged some time a go that the heavy reliance, even on the MATRX user's side, on Our current working solution is to hide this |
This seems to me the best way, next to mocking the gridworld instead of making a real one If I could create a GridWorld at arbitrary state, that would include the time stamp history required in this case. But I think the timestamp info is not part of GridWorld, it's part in fact of the CollectionGoal. So this is part of the code that I need to test. The state should be clear and targeted at the thing I want to test. Unfortunately I don't see how to do this, because gridworlds are build using the builder; otherwise they don't work. running a gridworld in a real simulation is not unit testing, but end-to-end testing. That may have its place but that's not what I'm after at this point. |
@jwaa ok, so I have to work around this issue with the object_id. How do I reset the object_id to ensure my test run gets consistent object_ids? |
Have you tried resetting the global object counter (used to set the ID)? builder = create_builder() # Create your builder
world = builder.get_world() # Get a world
... # Do stuff with that world
EnvObject.object_counter = 0 # Reset the global counter for objects
other_world = builder.get_world() # Create another world
... # Do stuff with this other world (supposedly having the same ID's) Disclaimer; not tested and this might have unforeseen side effects. Edit: Fixed |
How would you want to create a Yes the tick is stored in So I'm curious how you foresee the feature of creating a Without accounting for this "history", the |
As a side note to all (future) readers; this discussion is scoped for unit tests that a MATRX user wants to write for MATRX code to test their own world, not for unit tests of MATRX components as a MATRX developer. Within the package namespace the complexity is (somewhat) reduced although the same issues still stand. Just to a lesser degree, as within the package namespace one can unit test all the separate codeblocks that make-up a functionality instead of only testing the public methods. As the latter is more of E2E testing mixed with a Unit test approach. |
Thanks for all the help.
Indeed. May I add that testing of all functions and classes that require a world is not easily possible either. And therefore that this has a much broader scope than suggested. Because of these issues we halted our code unit testing efforts. I highly recommend that this is addressed. Testing starts with testability. Testability is an important factor related to code quality, modularity , reliability and robustness. |
What is your question?
My question is about mocking and testability. Generally it seems hard to test
Let's give a concrete example
Suppose I want to test the CollectionGoal of BW4T.
So I have to call the goal_reached(grid_world) function with -supposedly- a Mock of the grid_world
Now grid_world is a highly complex object. Just creating a Mock and filling it up a bit seems not going to work.
If you would follow general recommendations in such a situation eg https://www.ibm.com/developerworks/rational/library/oct06/pollice/index.html you would first rewrite the whole object into an interface that you can then properly mock. This seems out of the question as (1) python has no interface (2) extracting the interface calls will be difficult (3) interface calls are hidden behind parameter access calls.
For (1), a workaround could be to use an abstract class instead of an interface. For the others, I have no idea.
Because of that, I ended up creating not a mock but a real gridworld object. Which is created through the worldbuilder.
This gives additional problems with extra threads that seem to be created, that are still running at the end of the test, and then cause python to give errors that these threads were not terminated. I ignored these for now.
Next, I decided to completely hard code the blocks to be moved, to test if the goal is reached. Because the test code must be simple, not doing searches through gridworlds...
Now apparently CollectionGoal needs to keep track of timestamps, it does this itself by checking if there are blocks with unknown placement times and storing the times. That means I also have to work around this, by placing blocks one by one in the world and repeatedly calling CollectionGoal so that it can keep track of the block movements.
Overall, this looks way too complex for something simple as 'check that CollectionGoal returns true if the target blocks have been placed'.
I can't find any example test code in matrx for inspiration either.
How is testing supposed to be done? Am I missing something?
To what is your question related?
The text was updated successfully, but these errors were encountered: