---

RA-Lib is a benchmarking environment for Rank Aggregation (RA) algorithms. This website contains the current benchmarking results, which have 21 unsupervised RA methods, 7 supervised RA methods and 1 semi-supervised RA methods, these algorithms were tested on our preprocessed datasets. These datasets cover the areas of person re-identification (re-ID), recommendation system, bioinformatics and social choice. The code of tested methods includes both classical and state-of-the-art RA methods that can be funded in https://github.com/nercms-mmap. As well as having all the experimental details and settings on this website.

If you want to add your own algorithm to improve the benchmarking system, please send a package of your algorithm code and a link to the published paper to [email protected].

Unsupervised	Supervised	Semi-supervised
$\textrm{CombMIN}$ [1] $\textrm{CombMAX}$ [1] $\textrm{CombSUM}$ [1] $\textrm{CombANZ}$ [1] $\textrm{CombMNZ}$ [1] $\textrm{MC1}$ [2] $\textrm{MC2}$ [2] $\textrm{MC3}$ [2] $\textrm{MC4}$ [2] $\textrm{Borda count}$ [3] $\textrm{Dowdall}$ [4] $\textrm{Median}$ [5] $\textrm{RRF}$ [6] $\textrm{iRANK}$ [7] $\textrm{Mean}$ [8] $\textrm{HPA}$ [9] $\textrm{PostNDCG}$ [9] $\textrm{ER}$ [10] $\textrm{Mork-H}$ [40] $\textrm{CG}$ [11] $\textrm{DIBRA}$ [12] $\textrm{Borda-Score}$ [21]	$\textrm{wBorda}$ [13] $\textrm{CRF}$ [14] $\textrm{CSRA}$ [15] $\textrm{AggRankDE}$ [16] $\textrm{IRA}_\textrm{R}$ [17] $\textrm{IRA}_\textrm{S}$ [17] $\textrm{QI-IRA}$ [18]	$\textrm{SSRA}$ [19]

Directory Structure

│  .gitignore
│  README.md
│ 
│  
├─datasets
│  ├─ice-cream
│  ├─MovieLens1M  
│  ├─MQ2008-agg      
│  ├─NSCLC  
│  ├─Re-ID     
│  └─World University Ranking 2022 
├─examples
│      matlab_run_examples.m
│      py_run_examples.ipynb
├─results
├─src
│  │  requirements.txt
│  │  __init__.py
│  │  
│  ├─ramatlab
│  │  ├─common
│  │  │      InputType.m
│  │  │      IRAType.m
│  │  │      McType.m
│  │  │      
│  │  ├─semi
│  │  │      SSRA.m
│  │  │      
│  │  ├─supervised
│  │  │      AggRankDE.m
│  │  │      CRF.m
│  │  │      IRA.m
│  │  │      QI_IRA.m
│  │  │      WeightedBorda.m
│  │  │      
│  │  └─unsupervised
│  │          bordacount.m
│  │          borda_score.m
│  │          cg.m
│  │          combanz.m
│  │          combmax.m
│  │          combmed.m
│  │          combmin.m
│  │          combmnz.m
│  │          combsum.m
│  │          dibra.m
│  │          dowdall.m
│  │          er.m
│  │          hpa.m
│  │          irank.m
│  │          markovchainmethod.m
│  │          mean.m
│  │          median.m
│  │          mork_heuristic_maximum.m
│  │          postndcg.m
│  │          rrf.m
│  │          
│  └─rapython
│      │  __init__.py
│      │  
│      ├─common
│      │      constant.py
│      │      params.py
│      │      __init__.py
│      │      
│      ├─datatools
│      │      data_class.py
│      │      data_io.py
│      │      data_process.py
│      │      __init__.py
│      │      
│      ├─evaluation
│      │      evaluation.py
│      │      __init__.py
│      │      
│      ├─semi
│      │      ssra.py
│      │      __init__.py
│      │      
│      ├─supervised
│      │  │  aggrankde.py
│      │  │  crf.py
│      │  │  ira.py
│      │  │  qi_ira.py
│      │  │  weighted_borda.py
│      │  │  __init__.py
│      │  └─CSRA
│      │          
│      └─unsupervised
│              bordacount.py
│              borda_score.py
│              cg.py
│              combanz.py
│              combmax.py
│              combmed.py
│              combmin.py
│              combmnz.py
│              combsum.py
│              dibra.py
│              dowdall.py
│              er.py
│              hpa.py
│              irank.py
│              markovchain.py
│              mean.py
│              median.py
│              mork_heuristic_maximum.py
│              postndcg.py
│              rrf.py
│              scorefunc.py
│              __init__.py
│              
└─test

Demo

4-re-id

Running

1. Run python plot.py to plot results.

Follow-up Plan

We will be updating and adding more RA methods for shared use.

References

[1] Fox, E., & Shaw, J. (1994). Combination of multiple searches. NIST special publication SP, 243-243.

[2] Dwork, C., Kumar, R., Naor, M., & Sivakumar, D. (2001, April). Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web (pp. 613-622).

[3] Aslam, J. A., & Montague, M. (2001, September). Models for metasearch. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 276-284).

[4] Reilly, B. (2002). Social choice in the south seas: Electoral innovation and the borda count in the pacific island countries. International Political Science Review, 23(4), 355-372.

[5] Fagin, R., Kumar, R., & Sivakumar, D. (2003, June). Efficient similarity search and classification via rank aggregation. In Proceedings of the 2003 ACM SIGMOD international conference on Management of data (pp. 301-312).

[6] Cormack, G. V., Clarke, C. L., & Buettcher, S. (2009, July). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (pp. 758-759).

[7] Wei, F., Li, W., & Liu, S. (2010). iRANK: A rank‐learn‐combine framework for unsupervised ensemble ranking. Journal of the American Society for Information Science and Technology, 61(6), 1232-1243.

[8] Burges, C., Svore, K., Bennett, P., Pastusiak, A., & Wu, Q. (2011, January). Learning to rank using an ensemble of lambda-gradient models. In Proceedings of the learning to rank Challenge (pp. 25-35). PMLR.

[9] Fujita, S., Kobayashi, H., & Okumura, M. (2020). Unsupervised Ensemble of Ranking Models for News Comments Using Pseudo Answers. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42 (pp. 133-140). Springer International Publishing.

[10] Mohammadi, M., & Rezaei, J. (2020). Ensemble ranking: Aggregation of rankings produced by different multi-criteria decision-making methods. Omega, 96, 102254.

[11] Ivano Azzini., & Giuseppe Munda. (2020). Azzini, I., & Munda, G. (2020). A new approach for identifying the Kemeny median ranking. European Journal of Operational Research, 281(2), 388-401.

[12] Xiao, Y., Deng, H. Z., Lu, X., & Wu, J. (2021). Graph-based rank aggregation method for high-dimensional and partial rankings. Journal of the Operational Research Society, 72(1), 227-236.

[13] Akritidis, L., Fevgas, A., Bozanis, P., & Manolopoulos, Y. (2022). An unsupervised distance-based model for weighted rank aggregation with list pruning. Expert Systems with Applications, 202, 117435.

[14] Pujari, M., & Kanawati, R. (2012, November). Link prediction in complex networks by supervised rank aggregation. In 2012 IEEE 24th International Conference on Tools with Artificial Intelligence (Vol. 1, pp. 782-789). IEEE.

[15] Volkovs, M. N., & Zemel, R. S. (2014). New learning methods for supervised and unsupervised preference aggregation. The Journal of Machine Learning Research, 15(1), 1135-1176.

[16] Yu, Y., Liang, C., Ruan, W., & Jiang, L. (2020, May). Crowdsourcing-Based Ranking Aggregation for Person Re-Identification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1933-1937). IEEE.

[17] Bałchanowski, M., & Boryczka, U. (2022). Aggregation of Rankings Using Metaheuristics in Recommendation Systems. Electronics, 11(3), 369.

[18] Huang, J., Liang, C., Zhang, Y., Wang, Z., & Zhang, C. (2022). Ranking Aggregation with Interactive Feedback for Collaborative Person Re-identification.

[19] Hu, C., Zhang, H., Liang, C., & Huang, H. (2024). QI-IRA: Quantum-inspired interactive ranking aggregation for person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, pp. 1-9).

[20] Chen, S., Wang, F., Song, Y., & Zhang, C. (2008, October). Semi-supervised ranking aggregation. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 1427-1428).

[21] Boehmer, Niclas, Robert Bredereck, and Dominik Peters. "Rank aggregation using scoring rules." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.

Contacts

If you encounter any problems, you can contact us via email [email protected]