inference_demo script loads after some delay

[EmilyWebber]

Please add a note that the suggested command to check the install of NxD inference, inference_demo --help, can take an extended period of time to load the first time it's invoked.

https://github.com/aws-neuron/aws-neuron-sdk/blob/master/libraries/nxd-inference/nxdi-setup.rst#verify-nxd-inference-installation

[AWSNB]

Few minutes is really not what’s expected Is this because of slow boot volume or slow access to S3 ?

…

Sent from my iPhone On Jan 2, 2025, at 2:53 PM, Emily Webber ***@***.***> wrote:  Please add a note that the suggested command to check the install of NxD inference, inference_demo --help, can take multiple minutes to load the first time it's invoked. https://github.com/aws-neuron/aws-neuron-sdk/blob/master/libraries/nxd-inference/nxdi-setup.rst#verify-nxd-inference-installation — Reply to this email directly, view it on GitHub<#1076>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFTRWCLFR3YJXDJH5HRAGET2IWKMLAVCNFSM6AAAAABUQQRXQ2VHI2DSMVQWIX3LMV43ASLTON2WKOZSG43DMNJWGA2TGNI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jluntamazon]

I attempted to reproduce this with a fresh environment (on an existing instance):

python3 -m venv venv
source venv/bin/activate
python -m pip install --extra-index-url https://pip.repos.neuron.amazonaws.com neuronx-cc==2.* neuronx-distributed-inference
time inference_demo --help

This resulted in:

real	0m4.559s

As @AWSNB mentioned this may be due to instance setup effects.

@EmilyWebber Can you validate my example above produces a similar timing to mine?

[EmilyWebber]

I picked a PyTorch environment, then followed the steps for a manual install on the packages listed here.

source /opt/aws_neuronx_venv_pytorch_2_5/bin/activate
pip install --upgrade neuronx-cc==2.* neuronx-distributed-inference --extra-index-url https://pip.repos.neuron.amazonaws.com
time inference_demo --help

The result:

real 2m34.913s

If I use the aws_neuronx_venv_pytorch_2_5_nxd_inference venv, with the packages preinstalled, here's what I get.

source /opt/aws_neuronx_venv_pytorch_2_5_nxd_inference/bin/activate
time inference_demo --help

This results in:
real 3m13.345s

Oddly enough when I create a new Python venv, do a fresh install and invoke the helper script, as suggested above, I absolutely do see the same 4s load time!

[AWSNB]

Instance type and boot volume size ? From: Emily Webber ***@***.***> Reply-To: aws-neuron/aws-neuron-sdk ***@***.***> Date: Friday, January 3, 2025 at 3:09 PM To: aws-neuron/aws-neuron-sdk ***@***.***> Cc: "Bshara, Nafea" ***@***.***>, Mention ***@***.***> Subject: Re: [aws-neuron/aws-neuron-sdk] inference_demo script loads after some delay (Issue #1076) I picked a PyTorch environment, then followed the steps for a manual install on the packages listed here<https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/nxdi-setup.html#install-nxd-inference>. source /opt/aws_neuronx_venv_pytorch_2_5/bin/activate pip install --upgrade neuronx-cc==2.* neuronx-distributed-inference --extra-index-url https://pip.repos.neuron.amazonaws.com time inference_demo --help The result: real 2m34.913s If I use the aws_neuronx_venv_pytorch_2_5_nxd_inference venv, with the packages preinstalled, here's what I get. source /opt/aws_neuronx_venv_pytorch_2_5_nxd_inference/bin/activate time inference_demo --help This results in: real 3m13.345s — Reply to this email directly, view it on GitHub<#1076 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFTRWCOR26ZZMCK3UJASSNT2I4KANAVCNFSM6AAAAABUQQRXQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRZHEYTSMZYGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[EmilyWebber]

Just a trn1.2xlarge - I used the default settings in the launcher maybe 360 GB.