v1: sglang ec2 #5595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

DevakiBolleneni wants to merge 45 commits into aws:master from DevakiBolleneni:sgl-ec2

dlc_developer_config.toml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -37,7 +37,7 @@ deep_canary_mode = false
  
    [build]

    # Add in frameworks you would like to build. By default, builds are disabled unless you specify building an image.

    # available frameworks - ["base", "vllm", "sglang", "autogluon", "huggingface_tensorflow", "huggingface_pytorch", "huggingface_tensorflow_trcomp", "huggingface_pytorch_trcomp", "pytorch_trcomp", "tensorflow", "pytorch", "stabilityai_pytorch"]

    build_frameworks = []

    build_frameworks = ["sglang"]

    # By default we build both training and inference containers. Set true/false values to determine which to build.

sglang/build_artifacts/dockerd_entrypoint.sh

-Original file line number
+Diff line change
@@ -0,0 +1,6 @@
+    #!/usr/bin/env bash
+    # Check if telemetry file exists before executing
+    # Execute telemetry script if it exists, suppress errors
+    bash /usr/local/bin/bash_telemetry.sh >/dev/null 2>&1 || true
+    python3 -m sglang.launch_server "$@"

sglang/buildspec-ec2.yml

-Original file line number
+Diff line change
@@ -0,0 +1,56 @@
+    account_id: &ACCOUNT_ID <set-$ACCOUNT_ID-in-environment>
+    prod_account_id: &PROD_ACCOUNT_ID 763104351884
+    region: &REGION <set-$REGION-in-environment>
+    framework: &FRAMEWORK sglang
+    version: &VERSION "0.5.6"
+    short_version: &SHORT_VERSION "0.5"
+    arch_type: &ARCH_TYPE x86_64
+    autopatch_build: "False"
+    repository_info:
+      build_repository: &BUILD_REPOSITORY
+        image_type: &IMAGE_TYPE gpu
+        root: .
+        repository_name: &REPOSITORY_NAME !join [ pr, "-", *FRAMEWORK ]
+        repository: &REPOSITORY !join [ *ACCOUNT_ID, .dkr.ecr., *REGION, .amazonaws.com/, *REPOSITORY_NAME ]
+        release_repository_name: &RELEASE_REPOSITORY_NAME !join [ *FRAMEWORK ]
+        release_repository: &RELEASE_REPOSITORY !join [ *PROD_ACCOUNT_ID, .dkr.ecr., *REGION, .amazonaws.com/, *RELEASE_REPOSITORY_NAME ]
+    context:
+      build_context: &BUILD_CONTEXT
+        deep_learning_container:
+          source: src/deep_learning_container.py
+          target: deep_learning_container.py
+        install_efa:
+          source: scripts/install_efa.sh
+          target: install_efa.sh
+        start_cuda_compat:
+          source: sglang/build_artifacts/start_cuda_compat.sh
+          target: start_cuda_compat.sh
+        sagemaker_entrypoint:
+          source: sglang/build_artifacts/dockerd_entrypoint.sh
+          target: dockerd_entrypoint.sh
+    images:
+      sglang_ec2:
+        <<: *BUILD_REPOSITORY
+        context:
+          <<: *BUILD_CONTEXT
+        image_size_baseline: 26000
+        device_type: &DEVICE_TYPE gpu
+        cuda_version: &CUDA_VERSION cu129
+        python_version: &DOCKER_PYTHON_VERSION py3
+        tag_python_version: &TAG_PYTHON_VERSION py312
+        os_version: &OS_VERSION ubuntu22.04
+        tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *CUDA_VERSION, "-", *OS_VERSION, "-ec2" ]
+        latest_release_tag: !join [ *VERSION, "-", *DEVICE_TYPE, "-", *TAG_PYTHON_VERSION, "-", *CUDA_VERSION, "-", *OS_VERSION, "-ec2" ]
+        skip_build: "False"
+        docker_file: !join [ *FRAMEWORK, /, *ARCH_TYPE, /, *DEVICE_TYPE, /Dockerfile ]
+        target: sglang-ec2
+        build: true
+        enable_common_stage_build: false
+        test_configs:
+          test_platforms:
+            - sanity
+            - security
+            - ec2

sglang/buildspec.yml

Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		buildspec_pointer: buildspec-sm.yml
		buildspec_pointer: buildspec-ec2.yml

sglang/x86_64/gpu/Dockerfile

-Original file line number
+Diff line change
@@ Expand Up @@
      && rm -rf /var/lib/apt/lists/* \
      && rm -rf /root/.cache | true
+    # =======================================================
+    # ====================== EC2 ============================
+    # =======================================================
+    FROM base AS sglang-ec2
+    RUN dpkg -l | grep -E "cuda|nvidia|libnv" | awk '{print $2}' | xargs apt-mark hold \
+      && apt-get update \
+      && apt-get upgrade -y \
+      && apt-get clean
+    RUN rm -rf /tmp/*
+    COPY dockerd_entrypoint.sh /usr/local/bin/dockerd_entrypoint.sh
+    RUN chmod +x /usr/local/bin/dockerd_entrypoint.sh
+    ENTRYPOINT ["/usr/local/bin/dockerd_entrypoint.sh"]
     # =======================================================
     # ====================== sagemaker ======================
     # =======================================================
@@ Expand Down @@

test/dlc_tests/conftest.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -109,6 +109,8 @@ @@
         "pytorch_trcomp_training",
         # Autogluon
         "autogluon_training",
+        # SGLang
+        "sglang",
         # Processor fixtures
         "gpu",
         "cpu",
@@ Expand Down @@

test/dlc_tests/ec2/sglang/__init__.py

Empty file.

test/dlc_tests/ec2/sglang/ec2_tests/__init__.py

Empty file.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1: sglang ec2 #5595

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

v1: sglang ec2 #5595

Are you sure you want to change the base?

Uh oh!

v1: sglang ec2 #5595

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!