enumerate_tests takes a long time

[Almaz5200]

Tests enumeration takes a really long time in a big project, comparable to the time it takes to run the tests. Looking at the logs of running -enumerate-tests manually I see what can only be explained by launching the test classes without running them. We have some logic for tests in obj-c -load method that runs on module initialization and I see a lot of those logs, which implies those tests launching. For example, in my project for an incremental build of a single test class it took 20 seconds to build and 211 seconds to enumerate the tests again. Is there a way to either disable test explorer and enumeration fully, enumerate only manually, or enumerate while running the tests?

[wojciech-kulik]

Yeah, enumeration is very slow for some reason, I couldn't find any good alternative to do the same (besides greping all files, but I'm not sure If I want to do that). I skip enumeration if you perform reruns (:XcodebuildTestRepeat) or run tests from Tests Explorer.

I don't remember now why I don't just rely on logs from the run instead of enumerating tests. I will investigate that :).

[wojciech-kulik]

Could you also create a thread there: https://developer.apple.com/forums/topics/developer-tools-and-services/developer-tools-and-services-xcode

Apple engineers reply there quite often. Maybe they would be able to help. If not, you can also create a ticket in Feedback Assistant. I reported one issue with their CLI tool and they fixed it, so there is a better chance of getting fix than if you report iOS bugs :D. I guess CLI tools are not as crowded as iOS SDK :).

[wojciech-kulik]

I checked and potentially it could be changed to generate the tree dynamically while running tests. However, I'm not sure when I'll find time to try it.

If you have some spare time, I could give you some directions how to approach it. There are 4 main things in this feature:

1. Parsing tests from xcodebuild --enumerate-tests - enumeration_parser.lua
2. Building a tree for Test Explorer - explorer.lua:99 - it assumes that tests are sorted, so if it encounters a new class it creates a new parent node, the same for a target.
3. Currently, there is some dynamic insert for the case when we rerun some class and a new test appears, but it assumes that the parent class must exist in the tree - explorer.lua:750.
4. runner.lua has an option called skipEnumeration to indicate whether enumeration should be skipped.

So, to implement this feature:

• we could skip test enumeration completely (delete all the related code in runner.lua + enumeration_parser.lua) (I guess)
• improve (3) to be able to locate the right place in a tree and inject test. Possibly using some code from (2).
• unlock showing Test Explorer when tests are not loaded (explorer.lua:886)
• verify if the solution is reliable. I don't know if I remember correctly, but I think sometimes the logs are weird when running tests (the target might be missing or something). Please refer to unit tests, there are different cases there.

[Almaz5200]

I am not sure if I have time to do that right now, but I certainly will look into that as soon as I can. Thank you for your detailed response!

[wojciech-kulik]

Just to let you know, I started working on it :)

[wojciech-kulik]

@Almaz5200 I finished it, would you like to try it? #198

[Almaz5200]

Sorry for not replying earlier. Thank you, I really appreciate it! I'll give it a try today and will get back to you

[Almaz5200]

@wojciech-kulik it works wonderfully! I would even say it's more correct now. Previous versions showed some phantom test class for every test suite due to the fact that we subclass from our own "MyTestCase" rather than "XCTestCase", and this is now gone as well!

But most importantly of course its really quick now. Also there is another improvement as a side effect of that – Test explorer now only shows tests that you ran, which removes all the clutter and makes it a lot more useful, at least for me :)
Thank you again, it's amazing!

[wojciech-kulik]

Great to hear! 🔥 I'm merging those changes then :)

[wojciech-kulik]

@Almaz5200 you might be interested in checking out #201 :)

[Almaz5200]

👀 I certainly am! Thank you for letting me know, I'll try that as soon as I can and will let you know how it goes!