Skip to content

Conversation

@cr3ativ3cod3r
Copy link

Description

This PR introduces parallel processing of pack and unpack operations. Previously the layers were packed and unpacked sequentially.

  • In savekitfilelayers, calls to savecontentlayer is now executed in a new goroutine. They are also indexed, so that the order of layers is preserved.
  • In unpackrecursive, calls to unpacklayers is now executed in a new goroutine.
  • For the progress bars, created an instance of mpb.Progress and each goroutine adds a bar to this shared instance.
  • In ignorepaths struct, added a mutex and used it in matches method as MatchesOrParentMatches is not safe for concurrent access.

Linked issues

closes #254

AI-Assisted Code

  • This PR contains AI-generated code that I have reviewed and tested
  • I take full responsibility for all code in this PR, regardless of how it was created

Copy link
Member

@gorkem gorkem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some critical issues to address and also missing

  • Race detector tests (go test -race)
  • Concurrent layer ordering tests
  • Performance benchmarks


func (pm *ignorePaths) Matches(path, layerPath string) (bool, error) {
pm.mu.Lock()
defer pm.mu.Unlock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

saveContentLayer -> packLayerToTar -> writeLayerToTar
writeLayerToTar calls ignore.Matches` for each file With the mutex, all file traversal becomes serialized. Parallel implementation may be SLOWER than sequential due to lock contention.

if err != nil {
return err
}
kitfile.Model.Parts[index].LayerInfo = layerInfo
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is introducing concurrent writes to shared Kitfile fields without synchronization, causing potential data races and corruption. The same pattern is repeated on other layers too.

}

if err = eg.Wait(); err != nil {
close(results)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This closes the results channel and returns. However, some goroutines might still be running and will panic when trying to send to the closed channel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-threading for pack/unpack

2 participants