Question: best way to benchmark HTEX executor memory usage on single system

[d33bs]

Hello - thanks again for the great work on this project! I'm in the process of benchmarking how Parsl gets used within the context of a single system through the HTEX executor. When I do this I've been using memray to help me understand the isolated execution of Python programs which then go on to spawn an HTEX executor and run tasks. Specifically, I'm using --follow-fork to help make sure we attempt to track any forked procedure memory usage. When I do this I find that the memory usage is surprisingly low and likely may not be tracking the memory effectively (for example, it might be missing the app procedures). My impression is that memray might not be capable of tracking memory use through HTEX execution, or that I might have Parsl, memray, or something in between configured improperly. In contrast, I'm able to use the ThreadPoolExecutor with memray to find what seem to be accurate results.

Are there best practices or guidance when it comes to tracking memory through Parsl's HTEX executor? Ideally I'm seeking data which can then be plotted to best understand how Parsl consumes memory and time.