Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use an env var SGLANG_SET_CPU_AFFINITY to set cpu affinity; turn it off by default #2217

Merged
merged 3 commits into from
Nov 27, 2024

Conversation

merrymercy
Copy link
Contributor

@merrymercy merrymercy commented Nov 27, 2024

It will conflict with some other libraries (e.g. Ray). We see an error message like below, so we want to turn it off by default.

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/data/dhou/runner/actions-runner/_work/sglang-private/sglang-private/python/sglang/srt/managers/scheduler.py", line 1526, in run_scheduler_process
    gpu_proc_affinity(server_args.tp_size, server_args.nnodes, gpu_id)
  File "/data/dhou/runner/actions-runner/_work/sglang-private/sglang-private/python/sglang/srt/utils.py", line 1053, in gpu_proc_affinity
    p.cpu_affinity(bind_cpu_ids)
  File "/usr/local/lib/python3.10/dist-packages/ray/thirdparty_files/psutil/__init__.py", line [89](https://github.com/xai-org/sglang-private/actions/runs/12041611115/job/33575335155#step:4:90)3, in cpu_affinity
    self._proc.cpu_affinity_set(list(set(cpus)))
  File "/usr/local/lib/python3.10/dist-packages/ray/thirdparty_files/psutil/_pslinux.py", line 1717, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/thirdparty_files/psutil/_pslinux.py", line 2234, in cpu_affinity_set
    raise ValueError(
ValueError: CPU number 96 is not eligible; choose between [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73]
Process Process-6:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ray/thirdparty_files/psutil/_pslinux.py", line 2222, in cpu_affinity_set
    cext.proc_cpu_affinity_set(self.pid, cpus)
OSError: [Errno 22] Invalid argument

@HaiShaw
Copy link
Collaborator

HaiShaw commented Nov 27, 2024

@merrymercy Just curious - how to reproduce the error that you saw? will have a further look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants