Skip to content

Conversation

@Thomas1664
Copy link
Contributor

@Thomas1664 Thomas1664 commented Oct 29, 2023

My profiling showed that this PR makes install_files_and_write_listfile() 2x faster (tested with catch2 (39.8/ 24.5 ms) & aws-c-common (21.7/ 11.9 ms) on M2 MacBook Air). However, previous tests on Windows indicated even higher speedups (>= 4x).

Previous behavior

For each installed file individuall, the file type was btained and checked and the file was copied to the installed folder. At the same time, the file path was added to a vector containing a list of all installed files.

The listfile was created using fs.write_lines(). fs.write_lines() results in one write() and one put() (for \n) for each line.

Current behavior

  1. Obtain and check the file type of all installed files (parallelized)
  2. Create all required directories (not parallelized to avoid race conditions)
  3. Copy all files in parallel
  4. Write listfile in one go using fs.write_contents_and_dirs().

Comment on lines 120 to 161
fs.remove_all(target, IgnoreErrors{});
fs.remove(target, IgnoreErrors{});
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no difference for regular files between remove_all and remove (provided that vcpkg's implementation is identical to the STL): https://en.cppreference.com/w/cpp/filesystem/remove

@Thomas1664 Thomas1664 marked this pull request as draft October 30, 2023 11:05
@Thomas1664
Copy link
Contributor Author

Thomas1664 commented Oct 30, 2023

@BillyONeal Unit tests on OSX failed here. It seems to work again but I can consistently reproduce the failure on my local machine: The test case captures-output fails with error: calling CreateProcessW failed with 2 (unknown error). I'm not sure if this is the same test that failed in CI. I can confirm that this is most likely not caused by changes in this PR because the changed code inside commands.install.cpp is never hit.

@Thomas1664 Thomas1664 marked this pull request as ready for review October 30, 2023 15:22
@autoantwort
Copy link
Contributor

Btw you can build the format target locally to format all code.

@Thomas1664 Thomas1664 mentioned this pull request Oct 30, 2023
@BillyONeal BillyONeal reopened this Oct 30, 2023
@BillyONeal
Copy link
Member

Sorry!

@Thomas1664

This comment was marked as outdated.

@Thomas1664 Thomas1664 marked this pull request as draft February 16, 2025 22:25
@Thomas1664 Thomas1664 marked this pull request as ready for review February 17, 2025 11:28
@Thomas1664
Copy link
Contributor Author

@BillyONeal I addressed your concerns regarding conflicting directory creations and interleaved terminal output. I changed the code to write the list file from the main thread and I use a mutex to protect terminal output.

@BillyONeal
Copy link
Member

Closed in favor of #1896. Thanks for your submission!

@BillyONeal BillyONeal closed this Jan 13, 2026
@Thomas1664 Thomas1664 deleted the parallel-file-install branch January 13, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants