Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Having files ending in .gz stored next to the kineto trace may have unwanted side effects #163

Closed
fh-TurbaAI opened this issue Oct 21, 2024 · 1 comment

Comments

@fh-TurbaAI
Copy link

Describe the Bug

In the current code base, the directory of the kineto trace is passed to HTA's TraceAnalysis class. This may have unwanted side effects if the directory contains other files besides the kineto trace that was passed to chakra_trace_link. For example, a zipped chakra execution trace (whose name will likely end in .gz) will crash the linker since HTA attempts to read the file ending in .gz, which fails since the chakra trace has a binary format.

Steps to Reproduce

  • Create a folder containing host_trace.json, device_trace.json and some random binary file ending in .gz (for example, a zipped fully converted chakra trace)
  • Run the trace linker on the host and device traces.

Expected Behavior

  • Only host_trace.json and device_trace.json are taken into consideration during linking

Screenshots

  • HTA, which received the directory device_trace.json was in attempts to read all present files, including the one ending in .gz and crashes.
  • Error log
Traceback (most recent call last):
  File "/home/.../PyCharmProjects/chakra/.venv/bin/chakra_trace_link", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_link.py", line 47, in main
    linker.link(args.rank, args.chakra_host_trace, args.chakra_device_trace, args.output_file)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_linker.py", line 74, in link
    sync_deps = self.load_sync_dependencies(rank, chakra_device_trace)
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_linker.py", line 124, in load_sync_dependencies
    trace_analysis = TraceAnalysis(trace_dir=trace_dir)
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/hta/trace_analysis.py", line 36, in __init__
    self.t = Trace(trace_files, trace_dir)
             ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/hta/common/trace.py", line 330, in __init__
    trace_files if trace_files is not None else get_trace_files(self.trace_path)
                                                ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/hta/common/trace_file.py", line 87, in get_trace_files
    ok, rank_to_trace_dict = create_rank_to_trace_dict(trace_path)
                             ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/hta/common/trace_file.py", line 48, in create_rank_to_trace_dict
    data = line.decode() if isinstance(line, bytes) else line

@AlexDenisov
Copy link
Member

I believe this is fixed now via #167

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants