Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracker.load() does not work when using sagemaker pipeline #174

Open
heizari opened this issue Dec 6, 2022 · 1 comment
Open

Tracker.load() does not work when using sagemaker pipeline #174

heizari opened this issue Dec 6, 2022 · 1 comment

Comments

@heizari
Copy link

heizari commented Dec 6, 2022

Describe the bug
Create Experiment and Trial smexperiment. And configure PipelineExperimentConfig.
Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script, Tracker.load() return exception about this

Traceback (most recent call last):
  File "train.py", line 133, in <module>
    main()
  File "train.py", line 66, in main
    tracker = Tracker.load()
  File "/miniconda3/lib/python3.8/site-packages/smexperiments/tracker.py", line 161, in load
    _ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session),
AttributeError: 'NoneType' object has no attribute 'trial_component_name'

Maybe this function return None.
But TrialComponentEnvironment.source_arn is defined.
So, I guess is this line wrong?
Because environment["TRAINING_JOB_ARN"] contains uppercase when using sagemaker pipeline.

Sorry for my poor English.

To Reproduce

  1. configure Experiment, Trial and PipelineExperimentConfig
  2. run training job using pipeline
  3. Tracker.load() in training script

Expected behavior
Tracker.load() load trial_component in pipeline

Environment:
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:

  • sagemaker==2.116.0
  • sagemaker-experiments==0.1.39 (in image: 0.1.41)

Are you using a custom image: yes

@joshcx
Copy link

joshcx commented Apr 10, 2023

I'm facing the same issue with Sagemaker Pipelines. When I remove the ".lower" in the code you mentioned here then it works. Is it necessary to convert the source arn to lowercase?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants