[huggingface_pytorch] Inference - update for HuggingFace Transformers to 4.46.1 - Accelerate 1.1.0 - PyTorch 2.3

[JingyaHuang]

Issue #3870

• transformers: 4.46.1
• accelerate: 1.1.0
• torch: 2.3.0
• diffusers: 0.31.0
• peft: 0.13.2

Note:

• If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.

• All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.

Description

Tests run

NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"

• I have run builds/tests on commit for my changes.

Confused on how to run tests? Try using the helper utility...

Assuming your remote is called origin (you can find out more with git remote -v)...

• Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin

• Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp origin

• Restore TOML file when ready to merge

python src/prepare_dlc_dev_environment.py -rcp origin

NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:

Expand

• sagemaker_remote_tests = true
• sagemaker_efa_tests = true
• sagemaker_rc_tests = true

Additionally, please run the sagemaker local tests in at least one revision:

• sagemaker_local_tests = true

Formatting

• I have run black -l 100 on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)

DLC image/dockerfile

Builds to Execute

Expand

Fill out the template and click the checkbox of the builds you'd like to execute

Note: Replace with <X.Y> with the major.minor framework version (i.e. 2.2) you would like to start.

• build_pytorch_training_<X.Y>_sm

• build_pytorch_training_<X.Y>_ec2

• build_pytorch_inference_<X.Y>_sm

• build_pytorch_inference_<X.Y>_ec2

• build_pytorch_inference_<X.Y>_graviton

• build_tensorflow_training_<X.Y>_sm

• build_tensorflow_training_<X.Y>_ec2

• build_tensorflow_inference_<X.Y>_sm

• build_tensorflow_inference_<X.Y>_ec2

• build_tensorflow_inference_<X.Y>_graviton

Additional context

PR Checklist

Expand

• I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
• If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true
• If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.)
• (If applicable) I've documented below the DLC image/dockerfile this relates to
• (If applicable) I've documented below the tests I've run on the DLC image
• (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See https://www.apache.org/legal/resolved.html.
• (If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.

NEURON/GRAVITON Testing Checklist

• When creating a PR:

• I've modified dlc_developer_config.toml in my PR branch by setting neuron_mode = true or graviton_mode = true

Benchmark Testing Checklist

• When creating a PR:

• I've modified dlc_developer_config.toml in my PR branch by setting ec2_benchmark_tests = true or sagemaker_benchmark_tests = true

Pytest Marker Checklist

Expand

• (If applicable) I have added the marker @pytest.mark.model("<model-type>") to the new tests which I have added, to specify the Deep Learning model that is used in the test (use "N/A" if the test doesn't use a model)
• (If applicable) I have added the marker @pytest.mark.integration("<feature-being-tested>") to the new tests which I have added, to specify the feature that will be tested
• (If applicable) I have added the marker @pytest.mark.multinode(<integer-num-nodes>) to the new tests which I have added, to specify the number of nodes used on a multi-node test
• (If applicable) I have added the marker @pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">) to the new tests which I have added, if a test is specifically applicable to only one processor type

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[Captainia]

Pasting codebuild logs

=================================== FAILURES ===================================
--
831 | _ test_safety_file_exists_and_is_valid[669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.0-cpu-py311-ubuntu22.04-pr-4392] _
832 | [gw2] linux -- Python 3.8.0 /usr/local/bin/python
833 |  
834 | image = '669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.0-cpu-py311-ubuntu22.04-pr-4392'
835 |  
836 | @pytest.mark.model("N/A")
837 | @pytest.mark.skipif(is_canary_context(), reason="Skipping test because it does not run on canary")
838 | def test_safety_file_exists_and_is_valid(image):
839 | """
840 | Checks if the image has a safety report at the desired location and fails if any of the
841 | packages in the report have failed the safety check.
842 |  
843 | :param image: str, image uri
844 | """
845 | repo_name, image_tag = image.split("/")[-1].split(":")
846 | # Make sure this container name doesn't conflict with the safety check test container name
847 | container_name = f"{repo_name}-{image_tag}-safety-file"
848 | # Add null entrypoint to ensure command exits immediately
849 | run(
850 | f"docker run -id " f"--name {container_name} " f"--entrypoint='/bin/bash' " f"{image}",
851 | hide=True,
852 | warn=True,
853 | )
854 |  
855 | try:
856 | # Check if file exists
857 | docker_exec_cmd = f"docker exec -i {container_name}"
858 | safety_file_check = run(f"{docker_exec_cmd} test -f {SAFETY_FILE}", warn=True, hide=True)
859 | assert safety_file_check.ok, f"Safety file existence test failed for {image}"
860 |  
861 | file_content = run(f"{docker_exec_cmd} cat {SAFETY_FILE}", warn=True, hide=True)
862 | raw_scan_result = json.loads(file_content.stdout)
863 | safety_report_object = SafetyPythonEnvironmentVulnerabilityReport(report=raw_scan_result)
864 |  
865 | # processing safety reports
866 | report_log_template = "SAFETY_REPORT ({status}) [pkg: {pkg}] [installed: {installed}] [vulnerabilities: {vulnerabilities}]"
867 | failed_count = 0
868 | for report_item in safety_report_object.report:
869 | if report_item.scan_status == "FAILED":
870 | failed_count += 1
871 | LOGGER.error(
872 | report_log_template.format(
873 | status="FAILED",
874 | pkg=report_item.package,
875 | installed=report_item.installed,
876 | vulnerabilities=[
877 | entry for entry in report_item.vulnerabilities if not entry.ignored
878 | ],
879 | )
880 | )
881 | >           assert failed_count == 0, f"{failed_count} package/s failed safety test for {image} !!!"
882 | E           AssertionError: 1 package/s failed safety test for 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.0-cpu-py311-ubuntu22.04-pr-4392 !!!
883 | E           assert 1 == 0
884 |  
885 | sanity/test_safety_report_file.py:107: AssertionError

[Captainia]

Sorry

ERROR    test.dlc_tests.sanity.test_safety_report_file:test_safety_report_file.py:97 SAFETY_REPORT (FAILED) [pkg: setuptools] [installed: 68.2.2] [vulnerabilities: [SafetyVulnerabilityAdvisory(vulnerability_id='72236', advisory='Affected versions of Setuptools allow for remote code execution via its download functions. These functions, which are used to download packages from URLs provided by users or retrieved from package index servers, are susceptible to code injection. If these functions are exposed to user-controlled inputs, such as package URLs, they can execute arbitrary commands on the system.', reason_to_ignore='N/A', spec='<70.0.0', ignored=False)]]

Could you add setuptools to install list so they get patched?

[JingyaHuang]

Hi @Captainia, thanks for the log! I just patch the vulnerability with the upgrade of two HF libraries, but it seems that the CIs failed while pulling the image, do you have any idea why?

Error response from daemon: manifest for 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-cpu-py311-ubuntu22.04-pr-4392 not found: manifest unknown: Requested image not found
<class 'invoke.exceptions.UnexpectedExit'> thrown : Encountered a bad command exit code!

Command: 'docker pull 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-cpu-py311-ubuntu22.04-pr-4392'

Exit code: 1

Stdout: already printed

Stderr: already printed


Downloading executions cache from ec2 instance
Downloading executions cache from ec2 instance
Cache file wasn't downloaded: [Errno 2] No such file
Cache file wasn't downloaded: [Errno 2] No such file
No cache file was created
No cache file was created
Terminating Instances for image: 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-cpu-py311-ubuntu22.04-pr-4392
Destroying ssh Key_pair for image: 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-cpu-py311-ubuntu22.04-pr-4392
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded
Error response from daemon: manifest for 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-gpu-py311-cu121-ubuntu20.04-pr-4392 not found: manifest unknown: Requested image not found
<class 'invoke.exceptions.UnexpectedExit'> thrown : Encountered a bad command exit code!

Command: 'docker pull 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.1-gpu-py311-cu121-ubuntu20.04-pr-4392'

Exit code: 1

Stdout: already printed

Stderr: already printed

[Captainia]

Should this be 4.46.1? I see an image in testing repo with 669063966089.dkr.ecr.us-west-2.amazonaws.com/pr-huggingface-pytorch-inference:2.3.0-transformers4.46.0-gpu-py311-cu121-ubuntu20.04-pr-4392, could be the transformer version unmatch is the reason of not finding the image.

[JingyaHuang]

Good catch @Captainia, let me fix it!

[Captainia]

/rerun

[Captainia]

/rerun

[Captainia]

Seeing timeout in the sagemaker tests

signum = 14
--
3979 | frame = <frame at 0x7f6fc1925240, file '/codebuild/output/src2602542180/src/github.com/aws/deep-learning-containers/test/sagem...ubuntu20.04-pr-4392-2024-11-08-20-44-33/lib/python3.8/site-packages/sagemaker/session.py', line 6319, code _wait_until>
3980 |  
3981 | def handler(signum, frame):
3982 | >       raise TimeoutError("timed out after {} seconds".format(limit))
3983 | E       test.sagemaker_tests.huggingface.inference.integration.sagemaker.timeout.TimeoutError: timed out after 1800 seconds

Triggered a rerun

[Captainia]

The sanity check is failing - could you patch or ignore the following vulnerability?

ERROR    test.dlc_tests.sanity.test_safety_report_file:test_safety_report_file.py:98 SAFETY_REPORT (FAILED) [pkg: setuptools] [installed: 68.2.2] [vulnerabilities: [SafetyVulnerabilityAdvisory(vulnerability_id='72236', advisory='Affected versions of Setuptools allow for remote code execution via its download functions. These functions, which are used to download packages from URLs provided by users or retrieved from package index servers, are susceptible to code injection. If these functions are exposed to user-controlled inputs, such as package URLs, they can execute arbitrary commands on the system.', reason_to_ignore='N/A', spec='<70.0.0', ignored=False)]]