[mieb] SketchyI2IRetrieval hangs in the beginning

[Muennighoff]

@gowitheflow-1998 & I are both experiencing weird long hanging times when running SketchyI2IRetrieval at the start; Would be great to figure out why & maybe fix it; Users may think something is going wrong and cancel the run even though it will eventually start

[izhx]

I think this might be because the qrels is too large (90.6M according to https://huggingface.co/datasets/JamieSJS/sketchy/viewer/qrels), and the mapping of qrels_dict is slow.

https://github.com/embeddings-benchmark/mteb/blob/mieb/mteb/abstasks/Image/AbsTaskAny2AnyRetrieval.py#L99

In my test, this self.qrels.map(qrels_dict_init) costs 45 minutes.

log:

Sun Jan  5 13:29:26 2025 - Done load qrels, before map qrels_dict
Sun Jan  5 14:14:27 2025 - Done qrels_dict

code:

        qrels_dict = defaultdict(dict)
        # f'{time.asctime()} - Done load qrels, before map qrels_dict'

        def qrels_dict_init(row):
            qrels_dict[row["query-id"]][row["corpus-id"]] = int(row["score"])

        self.qrels.map(qrels_dict_init)
        # f'{time.asctime()} - Done qrels_dict'

[gowitheflow-1998]

I think this might be because the qrels is too large (90.6M according to https://huggingface.co/datasets/JamieSJS/sketchy/viewer/qrels), and the mapping of qrels_dict is slow.

https://github.com/embeddings-benchmark/mteb/blob/mieb/mteb/abstasks/Image/AbsTaskAny2AnyRetrieval.py#L99

In my test, this self.qrels.map(qrels_dict_init) costs 45 minutes.

thanks for investigating this quickly! Sketchy definitely worths a downsample then. Not sure about how many classes in total but maybe for each class, keep only 1/n sketches and 1/m real images for each class? Looks like there are 200 real-life images for each class (judging by 90M qrels / 450k queries). Maybe make it 20. wdyt? @Jamie-Stirling