TabRepo 2.0 Feature Tracker

[Innixma]

For TabRepo 2.0, several quality of life changes should be made for ease of use. This list will evolve over time.

P0 (Critical)

• Add EvaluationRepository Join Logic #65
• Add a simple method to fit models on datasets #69
• Move config hyperparameter files into the S3 artifacts, rather than being part of the TabRepo source code.
• Add ability for user to specify their own custom ensembling method and method kwargs.
• Add easy way to go from Portfolio List -> AutoGluon hyperparameters dictionary. Add AutoGluon hyperparameters fetching #80
• Rename ensemble_size -> n_iterations

P1

• Hyperparameter dictionary should have keys matching the names elsewhere. Currently there is an inconsistency, sometimes a model is called "LightGBM_c1", another time it is called "LightGBM_c1_BAG_L1". The name should be consistent in all usages and contexts.
• Add a way to recover original and transformed features from OpenML #67
• Consider using a new distributed batch training logic #68
• Add support for callback early stopping in simulations #81
• Document every method in EvaluationRepository
• Cleanup a large portion of the scripts code, move relevant logic into tabrepo/ for ease of use by others for their own experiments.
• Add max_models and max_models_per_type #72
• Add a requirements_frozen file to ensure reproducibility of experiments, or provide a Docker container.
• Release TabRepo on PyPi?
• Each fold model of a bag should be able to get its own validation predictions. Helps to support Add Simulation Support for "Don’t Waste Your Time: Early Stopping Cross-Validation" #60.

P2 (Nice-to-have)

• [new datasets] PMLBmini #82
• Add TabRepo artifacts to HuggingFace #66
• Add binary_as_multiclass to repo.predict methods #64
• Each fold model of a bag should have its own predictions on the test data. This will enable more in-depth simulations involving partial bagged fits / hyperband style strategies. Helps to support Add Simulation Support for "Don’t Waste Your Time: Early Stopping Cross-Validation" #60.
• Linting checks
• Consistent docstring format

P3

• Can theoretically reduce multiclass task storage and memory cost by a factor of num_classes by storing only the prediction probability of the ground truth class. This might be brittle to extensions though, and would only work for log_loss.

[geoalgo]

Another thing that has been my radar for some time is to have tabrepo on huggingface. It will speedup the download time by ~8x (download is very slow from outside) and would make the dataset more visible.

[geoalgo]

Having an example or an API that allows to "join" two repository would be also be quite useful. One could do:

from tabrepo import load_repository
from tabrepo.utils import merge_repositories
repo = load_repository("D244_F3_C1530_30")
repo_with_new_method = load_repository("D244_F3_C1530_30")
# builds a repository from the two, filter models that appear only in all task, underneath, just calls the repo that contains a given model
repo_union = merge_repositories([repo, repo_with_new_method], force_dense=True)

[Innixma]

@geoalgo Yes, repo joining is something I plan to implement. I added a tracking GitHub issue: #65

[geoalgo]

Sorry for the delay again :-) The list you made sounds great!

One thing I want to mention that I think could be quite useful is adding a way to recover original and transformed features from openml.

Something like that:

df, y = repo.openml_dataframe(dataset="airplane", fold=2) # gets the raw columns from the dataset
X, y = repo.openml_transformed_features(dataset="airplane", fold=2)  # gets the features as provided to the model

This would allow to use Tabrepo to train TabPFN models (probably with larger scales that what they currently use). Also it would make it easier to train new models and add them in tabrepo.