Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Add glob_pattern to LogstreamerInput #1985

Open
michaelgibson opened this issue Sep 3, 2016 · 0 comments
Open

Add glob_pattern to LogstreamerInput #1985

michaelgibson opened this issue Sep 3, 2016 · 0 comments

Comments

@michaelgibson
Copy link
Contributor

michaelgibson commented Sep 3, 2016

The LogstreamerInput uses filepath.Walk() for locating files to compare to the file_match regexes.
This is normally fine in a mostly sane directory structure.
In the case of ridiculously large/complex filesystems with numerous levels containing millions of files, this can be somewhat inefficient.

The pattern matching is useful so not suggesting a replacement for regexing.
This change should still follow the same rules for log_directory and file_match.
I propose adding an additional config option to use globbing specifically for the filepath.Walk portion.
https://github.com/mozilla-services/heka/blob/dev/logstreamer/filehandling.go#L280

glob_pattern = "/this/*/path*/would/*take/*/*/*/forever/to/*/walk/*/in/a*/reasonable/*/*/*/timeframe/*"

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant