Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add model transfer functionality to the package #482

Open
egillax opened this issue Oct 7, 2024 · 0 comments
Open

[Feature] Add model transfer functionality to the package #482

egillax opened this issue Oct 7, 2024 · 0 comments

Comments

@egillax
Copy link
Collaborator

egillax commented Oct 7, 2024

Currently when executing network studies using Strategus we we use a separate ModelTransferModule to download models either from local path, s3 or github. We used this successfully in the Deep Learning Comparison study using s3. This requires that a regular external validation study workflow looks like:

CohortGenerator -> ModelTransfer -> External validation

The issue with this approach is that the logic of which model is used with which prediction problem can't be fully customized upfront since we haven't downloaded the models. Currently I've added logic to match on modelTargetId and modelOutcomeId. This means the prespecified module settings to execute a validation study would look like:

validationComponents <- list(list(targetId=1,
                                  outcomeId=2,                                                                                                                    
                                  modelTargetId=1, 
                                  modelOutcomeId=2, 
                                  populationSettings,
                                  restrictPlpDataSettings, 
                                  recalibrate='weakRecalibration',
                                  runCovariateSummary=TRUE),
                             list(targetId=1, 
                                  outcomeId=3,
                                  modelTargetId=1, 
                                  modelOutcomeId=2, 
                                  populationSettings,
                                  restrictPlpDataSettings, 
                                  recalibrate='weakRecalibration',
                                  runCovariateSummary=TRUE)
)

Then once the models have been downloaded using the ModelTransferModule the downloaded models are matched using targetId=modelTargetId and outcomeId=modelOutcomeId and then the validationDesign is created (only one shown):

design <- createValidationDesign(targetId=1, 
                                 outcomeId=1,
                                 populationSettings = myPopSettings,
                                 restrictPlpDataSettings = myRestrictSettings,
                                 plpModelList,
                                 recalibrate='weakRecalibration',
                                 runCovariateSummary=TRUE)                                                        

A better solution I think is to forego the ModelTransferModule and instead be able to supply the s3 or github paths directly in the plpModelList when creating the validationDesign. Then the models are fetched at runtime and validated. This allows for complete flexibility in what models are used for what prediction problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant