File statistics exporter

Prometheus exporter gathering metrics about file size, modification time and other statistics.

Quickstart

Pre-built binaries are available on the GitHub release page.

Usage

Configure target files on command line, passing glob patterns in parameters

./filestat_exporter '*'

Optional flags:

• -config.file <yaml>: The path to the configuration file (use "none" to disable).
• -log.level <level>: Logging level [debug, info, warn, error]. (default: info)
• -version: Print the version of the exporter and exit.
• -web.listen-address <port>: Address to listen on for web interface and telemetry. (default: 9943)
• -web.telemetry-path <URL path>: Path under which to expose metrics. (default: /metrics)
• -web.config <web file>: Path to config yaml file that can enable TLS or authentication.
• -web.systemd-socket: Flag to use systemd socket activation listeners instead of port listeners (Linux only).
• -path.cwd <path>: Change working directory of path pattern collection.
• -tree.name <name>: Change default tree name used as tree label in metrics
• -tree.root <path>: Chnage default root path of files
• -metric.crc32: Generate CRC32 hash metric of files.
• -metric.nb_lines: Generate line number metric of files.

The exporter can read a config file in yaml format (filestat.yaml by default).

exporter:
  # Optional working directory - overridden by parameter '-path.cwd'
  working_directory: "/path/to/my/project"
  # Optional network parameters
  listen_address: ':9943'
  #metrics_path: /metrics
  
  # Optional working directory - overridden by parameter '-path.cwd'
  working_directory: "/path/to/my/project"
  # Optional default tree name and root path - overridden by parameter '-root.name' and '-root.path'
  #tree_name: ""
  #tree_root: ""

  # Default enable/disable of metrics - overridden if not set by parameter '-metric.*'
  enable_crc32_metric: true
  # enable_nb_line_metric: false
  # list of patterns to apply - metrics can be enable/disabled for each group
  files:
    - patterns: ["*.html","assets/*.css","scripts/*.js"]
    - patterns: ["data/*.csv"]
      enable_nb_line_metric: true
    - patterns: ["archives/*.tar.gz"]
      enable_crc32_metric: false
      enable_nb_line_metric: false

  # other trees
  trees: []

Notes:

• if a file is matched by a pattern more than once, only the first match's config is used
• if no tree name is defined, the label is not used

Pattern format

Pattern uses the glob implementation of bmatcuk/doublestar project:

• Doublestar (**) can be used to match directories recursively. It should appear surrounded by path separators such as /**/.
• Usual Glob syntax is still supported.

Pattern can also use golang template format with the following functions:

Function	Description	Example
now	Current time	{{ now.Locale.Year }}/*.tgz

Trees

Trees extract files' statistics relatively to a tree root. If tree root is empty, it is not applied. The name of the tree is used for scoping the metrics.

If defined, the tree root must exists. It can be templated but usual patterns are not used.

- tree_name: name of tree   # optional
  tree_root: path/to/tree/  # optional
  #enable_*_metric: true|false # default for tree
  files: [] # as usual

Exported Metrics

Metric	Description	Labels
file_glob_match_number	Number of files matching pattern	tree, pattern
file_stat_size_bytes	Size of file in bytes	tree, path
file_stat_modif_time_seconds	Last modification time of file in epoch time	tree, path
file_content_hash_crc32 (*)	CRC32 hash of file content	tree, path
file_content_line_number (*)	Number of lines in file	tree, path

Note: metrics with (*) are only provided if configured

Building and running

Prerequisites:

• Go compiler - currently version v1.22
• Linux with make installed
• Essential build environment for dependencies

Building

go get github.com/michael-doubez/filestat_exporter
cd ${GOPATH-$HOME/go}/src/github.com/michael-doubez/filestat_exporter
make
./filestat_exporter <flags> <file glob patterns ...>

To see all available configuration flags:

./filestat_exporter -h

The Makefile provides several targets:

• make check: Running checks and tests
• make run: Run exporter from go
• make version: Print current version
• make build: Build exporter without checks

Cross compiled distribution

To build all distribustion packages

make dist

To build a specific os/architecture package

make dist-<os>-<archi>
make dist-linux-amd64
...

Using Docker

The filestat_exporter is designed to monitor files on the host system.

Try it out in minutes on Katakoda docker playground:

# create local file
docker container run --rm -d -v ~/my_files:/my_files --name my_files bash -c 'echo "Hello world" > /my_files/sample.txt'
# launch exporter watching the files
docker run -d -p 9943:9943 --name=filestats -v ~/my_files:/data mdoubez/filestat_exporter -path.cwd /data '*'
# see file metrics
curl -s docker:9943/metrics | grep file_

TLS and basic authentication

Filestat Exporter supports TLS and basic authentication. This enables better control of the various HTTP endpoints.

To use TLS and/or basic authentication, you need to pass a configuration file using the --web.config.file parameter. The format of the file is described in the exporter-toolkit repository.

License

Apache License 2.0, see LICENSE.