Realistic evaluation of transductive few-shot learning

Introduction

This repo contains the code for our NeurIPS 2021 paper "Realistic evaluation of transductive few-shot learning". This is a framework that regroups all methods evaluated in our paper except for SIB and LR-ICI. Results provided in the paper can be reproduced with this repo. Code was developed under python 3.8.3 and pytorch 1.4.0.

1. Getting started

1.1 Quick installation (recommended) (Download datasets and models)

To download datasets and pre-trained models (checkpoints), follow instructions 1.1.1 to 1.1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM)

1.1.1 Place datasets

Make sure to place the downloaded datasets (data/ folder) at the root of the directory.

1.1.2 Place models

Make sure to place the downloaded pre-trained models (checkpoints/ folder) at the root of the directory.

1.2 Manual installation

Follow instruction 1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM) if facing issues with previous steps. Make sure to place data/ and checkpoints/ folders at the root of the directory.

2. Requirements

To install requirements:

conda create --name <env> --file requirements.txt

Where <env> is the name of your environment

3. Reproducing the main results

Before anything, activate the environment:

source activate <env>

3.1 Table 1 and 2 results in paper

Evaluation in a 5-shot scenario on mini-Imagenet using RN-18 as backbone (Table 1. in paper)

Method	1-shot	5-shot	10-shot	20-shot
SimpleShot	63.0	80.1	84.0	86.1
PT-MAP	60.1 (↓16.8)	67.1 (↓18.2)	68.8 (↓18.0)	70.4 (↓17.4)
LaplacianShot	65.4 (↓4.7)	81.6 (↓0.5)	84.1 (↓0.2)	86.0 (↑0.5)
BDCSPN	67.0 (↓2.4)	80.2 (↓1.8)	82.7 (↓1.4)	84.6 (↓1.1)
TIM	67.3 (↓4.5)	79.8 (↓4.1)	82.3 (↓3.8)	84.2 (↓3.7)
α-TIM	67.4	82.5	85.9	87.9

To reproduce the results from Table 1. and 2. in the paper, from the root of the directory execute this python command.

python3 -m src.main --base_config <path_to_base_config_file> --method_config <path_to_method_config_file> 

The <path_to_base_config_file> follows this hierarchy:

config/<balanced or dirichlet>/base_config/<resnet18 or wideres>/<mini or tiered or cub>/base_config.yaml

The <path_to_method_config_file> follows this hierarchy:

config/<balanced or dirichlet>/methods_config/<alpha_tim or baseline or baseline_pp or bdcspn or entropy_min or laplacianshot or protonet or pt_map or simpleshot or tim>.yaml

For instance, if you want to reproduce the results in the balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:

python3 -m src.main --base_config config/balanced/base_config/resnet18/mini/base_config.yaml --method_config config/balanced/methods_config/alpha_tim.yaml

If you want to reproduce the results in the randomly balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:

python3 -m src.main --base_config config/dirichlet/base_config/resnet18/mini/base_config.yaml --method_config config/dirichlet/methods_config/alpha_tim.yaml

Reusable data sampler module

One of our main contribution is our realistic task sampling method following Dirichlet's distribution.

Our realistic sampler can be found in sampler.py file. The sampler has been implemented following Pytorch's norms and in a way that it can be easily reused and integrated in other projects.

The following notebook exemple_realistic_sampler.ipynb is an exemple that shows how to initialize and use our realistic category sampler.

Contact

For further questions or details, reach out to Olivier Veilleux ([email protected])

Acknowledgements

Special thanks to the authors of NeurIPS 2020 paper "TIM: Transductive Information Maximization" (TIM) (https://github.com/mboudiaf/TIM) for publicly sharing their pre-trained models and their source code from which this repo was inspired from.