Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]: Allow Metrics To Directly Accept Frameworks & Datasets #25

Open
nmichlo opened this issue Feb 6, 2022 · 1 comment
Open
Assignees
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers refactoring Internal changes to the code that do not change the outputs.
Milestone

Comments

@nmichlo
Copy link
Owner

nmichlo commented Feb 6, 2022

Is your feature request related to a problem? Please describe.
Current metrics require that you provide a representation function. This is inconvenient and always repeated. Metrics also always require that the dataset be a DisentDataset.

dataset = DisentDataset(data)
# we always need to produce this same function
get_repr = lambda x: module.encode(x.to(module.device))
# to use with metrics
results = metric_mig(dataset, get_repr)

Describe the solution you'd like
Allow the metrics to directly accept the frameworks instead, and automatically wrap datasets with DisentDataset

# directly use the framework and raw data instead!
results = metric_mig(data, module)

Describe alternatives you've considered

  • Detect if isinstance(obj, DisentFramework) then handle everything automatically!
  • Or add a function to frameworks called get_representation
  • Or detect if an object has the property encode and use that instead!
@nmichlo nmichlo added documentation Improvements or additions to documentation enhancement New feature or request refactoring Internal changes to the code that do not change the outputs. labels Feb 6, 2022
@nmichlo nmichlo added this to the v0.4.0 milestone Feb 6, 2022
@nmichlo nmichlo self-assigned this Feb 6, 2022
@nmichlo nmichlo changed the title [FEATURE]: Allow Metrics To Directly Accept Frameworks [FEATURE]: Allow Metrics To Directly Accept Frameworks & Datasets Feb 6, 2022
@nmichlo
Copy link
Owner Author

nmichlo commented Jun 9, 2022

This will be easier now that metrics use the @metric decorator

@nmichlo nmichlo added the good first issue Good for newcomers label Jun 9, 2022
@nmichlo nmichlo modified the milestones: v0.4.0, roadmap Jun 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers refactoring Internal changes to the code that do not change the outputs.
Projects
None yet
Development

No branches or pull requests

1 participant