You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Create Experiment and Trial smexperiment. And configure PipelineExperimentConfig.
Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script, Tracker.load() return exception about this
Traceback (most recent call last):
File "train.py", line 133, in <module>
main()
File "train.py", line 66, in main
tracker = Tracker.load()
File "/miniconda3/lib/python3.8/site-packages/smexperiments/tracker.py", line 161, in load
_ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session),
AttributeError: 'NoneType' object has no attribute 'trial_component_name'
Maybe this function return None.
But TrialComponentEnvironment.source_arn is defined.
So, I guess is this line wrong?
Because environment["TRAINING_JOB_ARN"] contains uppercase when using sagemaker pipeline.
Sorry for my poor English.
To Reproduce
configure Experiment, Trial and PipelineExperimentConfig
run training job using pipeline
Tracker.load() in training script
Expected behavior Tracker.load() load trial_component in pipeline
Environment:
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:
sagemaker==2.116.0
sagemaker-experiments==0.1.39 (in image: 0.1.41)
Are you using a custom image: yes
The text was updated successfully, but these errors were encountered:
I'm facing the same issue with Sagemaker Pipelines. When I remove the ".lower" in the code you mentioned here then it works. Is it necessary to convert the source arn to lowercase?
Describe the bug
Create Experiment and Trial
smexperiment
. And configurePipelineExperimentConfig
.Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script,
Tracker.load()
return exception about thisMaybe this function return
None
.But
TrialComponentEnvironment.source_arn
is defined.So, I guess is this line wrong?
Because
environment["TRAINING_JOB_ARN"]
contains uppercase when using sagemaker pipeline.Sorry for my poor English.
To Reproduce
Tracker.load()
in training scriptExpected behavior
Tracker.load()
load trial_component in pipelineEnvironment:
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:
Are you using a custom image: yes
The text was updated successfully, but these errors were encountered: