Why is ranking for a challenge object required before generating podium plots

[ff98li]

Hi, thank you for making the amazing package! I have a few questions regarding to the podium plot:

1. As in the title. Ranking (e.g. aggregateThenRank) must be first performed for a challenge object to generate a ranked.list for podium() to dispatch to podium.ranked.list. If I understood the podium plot correctly, the ranks along the x-axis describe how well an algorithm achieves for individual test case; in other words, it is an algorithm-wise comparison for every test case. This means the ranking results in the podium plot does not depend on the aggregated summary values (e.g. mean or median).
2. There is another version of the podium() method for the challenge object, which has an required argument ordering. However, because of the sparse documentation for that method, I'm having hard time deciphering what the proper input for ordering should be.

By all means please feel free to point out any mistakes in my understanding for the podium plot:) explanation from the authors will be a big help for me to use the plot properly🙏

[wiesenfa]

Dear @ff98li , sorry for the late reply.

1. the algorithms within each podium are ordered according to the according to a ranking e.g. obtained by aggregateThenRank. E.g. within podium place "2." at the very left bar/the very left dots represent the proportion of test cases the algorithm ranked best according to for instance aggregateThenRank are ranked "2.". I.e. the ordering within each podium place is according to the overall ranking provided. while the podium places represent the frequencies of rankings on the test case level.
2. yes, indeed the documentation is sparse positively spoken... if ranking is your ranked.list object, then ordering = t(ranking$matlist[[task]][, "rank", drop = F])["rank", ] where task is the task name in case of multi task challenge or just use task=1 otherwise. In words, ordering is a named vector with elements equal to the rank and corresponding algorithm names as element names. E.g. ordering could be e.g. ordering= c(A1=2, A2=3, A3=1) for a challenge with 3 algorithms where e.g. algorithm A2 has rank 3.

Hope this helps!