Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] I got memory issues with direct RL env. #1497

Open
sizsJEon opened this issue Dec 4, 2024 · 4 comments
Open

[Question] I got memory issues with direct RL env. #1497

sizsJEon opened this issue Dec 4, 2024 · 4 comments

Comments

@sizsJEon
Copy link
Contributor

sizsJEon commented Dec 4, 2024

[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-12-04 09:23:08 [27,397ms] [Error] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp error: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.1/NVIDIA/Materials/Base/Architecture/Shingles_01.mdl?watch=00007efec5bf1a20(1,1): C100 "mdl" expected
2024-12-04 09:23:08 [27,397ms] [Error] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp error: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.1/NVIDIA/Materials/Base/Architecture/Shingles_01.mdl?watch=00007efec5bf1a20(17,4): C100 IDENT expected
2024-12-04 09:23:08 [27,397ms] [Warning] [rtx.neuraylib.plugin] Loading MdlModule to DB (rtx_scope0) failed: ::Z6Ahttp_3A_2F::Z58omniverse_2Dcontent_2Dproduction_2Es3_2Dus_2Dwest_2D2_2Eamazonaws_2Ecom::Assets::Isaac::ZCD_34_2E1::NVIDIA::Materials::Base::Architecture::Shingles_01
2024-12-04 09:23:08 [27,398ms] [Warning] [rtx.neuraylib.plugin] Loading transaction committed (this thread). MdlModule is NOT in the DB (rtx_scope0): ::Z6Ahttp_3A_2F::Z58omniverse_2Dcontent_2Dproduction_2Es3_2Dus_2Dwest_2D2_2Eamazonaws_2Ecom::Assets::Isaac::ZCD_34_2E1::NVIDIA::Materials::Base::Architecture::Shingles_01
[INFO]: Time taken for simulation start : 64.168856 seconds
[INFO] Event Manager:  <EventManager> contains 2 active terms.

while these error and warnings are rose,
my RAM memory consumption rise to 80GB.

I tried to solve this problems, but currently failing again and again.

any advices will be helpful.

@sizsJEon
Copy link
Contributor Author

sizsJEon commented Dec 4, 2024

accessing
mdl_path="{NVIDIA_NUCLEUS_DIR}/Materials/Base/Architecture/Shingles_01.mdl",
rise those errors.
it is odd that I can access those address in browser and download it.

but still have cpu-memory leakage.

@sizsJEon
Copy link
Contributor Author

sizsJEon commented Dec 4, 2024

it seems to be train with Terrain(self.cfg.terrain.terrain_type == "generator") and recording video result memory leakage
(--headless --video --video_length 100 --video_interval 500)
without video recording option, RAM stay in 8GB.
anybody ever experienced a situation like this?

@RandomOakForest
Copy link
Collaborator

Thanks for posting this. You are using Assets/Isaac/4.1 when they should be 4.2. Upgrading to Isaac Sim 4.2 should resolve the issue.

@Duuuc
Copy link

Duuuc commented Dec 12, 2024

it seems to be train with Terrain(self.cfg.terrain.terrain_type == "generator") and recording video result memory leakage (--headless --video --video_length 100 --video_interval 500) without video recording option, RAM stay in 8GB. anybody ever experienced a situation like this?

I experienced the same. I also observed it with a manager based env (I ran the command ./isaaclab.sh -p source/standalone/workflows/rsl_rl/train.py --task Isaac-Reach-Franka-v0 --headless --video --video_length 1000)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants