Option to exclude folders from scanning

[falco467]

Your feature request related to a problem? Please describe.

golang-lint-ci --fast is slowed down, because it traverses all directories (even unrelated big directories like a node_modules folder) and there is no option to exclude unrelated folders from scanning/traversal.

Describe the solution you'd like.

Provide an option to ignore/exclude folders from traversal/scanning.

Describe alternatives you've considered.

I tried excluding the folders with the --skip-dirs flag, but golang-lint-ci will still traverse the subfolders and take a lot of time on each run.

Additional context.

I'm using golang-lint-ci as my integrated linter in VS Code (running on file-save), fast execution time is critical. I measured performance with --verbose and Go packages loading at mode 575 takes up the most time (even with caching enabled) it takes almost 9 seconds every run, even though my project has only ~1000 Lines of Go-Code in 6 files.

We have the frontend project in a subfolder of the go project (/frontend) which includes a huge node_modules folder. Deleting this folder speeds up the scan from 12 to 4 seconds. Is there any option to hide this folder from golang-lint-ci ?

[boring-cyborg]

Hey, thank you for opening your first Issue ! 🙂 If you would like to contribute we have a guide for contributors.

[falco467]

This could also probably be fixed by caching. On multiple runs golang-lint-ci could recognize by the modification time on the folder that nothing has changed inside the folder and should not need to scan anything inside the folder. I'm working with go build cache and golang-lint-ci cache. But it is still scanning all folders again even when I don't change any file between runs.

[ldez]

Hello,

A package, if it's used by another part of the code, cannot be skipped.

Related to #4217
Related to #913

[falco467]

This is not about skipping packages, but skipping unrelated Folders without any .go files in it (like a folder with PDFs or JS files)

[falco467]

Here is the output which illustrates the problem:

golangci-lint run --verbose
level=info msg="[loader] Go packages loading at mode 575 (types_sizes|files|imports|name|compiled_files|deps|exports_file) took 10.0107529s"
level=info msg="Execution took 12.4308233s"

After deleting the big folder without any .go files:

golangci-lint run --verbose
level=info msg="[loader] Go packages loading at mode 575 (compiled_files|imports|name|types_sizes|deps|exports_file|files) took 2.3652813s"
level=info msg="Execution took 4.488697s"

[ldez]

Can you provide a reproducible example?

[falco467]

@ldez sure - here is a simple example:

A project with a single main.go file and a folder "stuff" which contains no files, just empty folders.
golangci-lint --skip-dirs stuff still takes about 30seconds on windows in contrast to 3 seconds if I delete the folder stuff.

package main

import (
	"fmt"
	"os"
)

func main() {
	fmt.Println("generating 10_000 empty folders")
	for i := range 100 {
		for j := range 100 {
			os.MkdirAll(fmt.Sprintf("stuff/f%d/f%d", i, j), 0o777)
		}
	}
}

Here is the whole project as a zip file:
golangtest.zip

[falco467]

I think I located the problem with how the go module system works. If I rename the folder to _stuff or if I put an empty "go.mod" file into the folder it will be ignored by the go module system and build and I get the fast response.
If golangci-lint cannot solve this, we will use the workaround of putting an empty go.mod file with a comment into the subfolder so it gets ignored.

[ldez]

Just to explain, golangci-lint uses the Go pieces and those pieces allow only the same pattern as go build.
The scanning will happen on all trees because Go needs to know where Go files are located.
Directories starting with _ are ignored by Go.
Golangci-lint will not be able to do a magical filter.

[skandragon]

My use case is different. I have some auto-generated code from a Ragel lexer, and it is huge and mostly not my own code. I'm willing to accept slightly broken code there so long as it compiles.

The problem is, this 250k file takes significant time to run, and even if issues are found, it is not likely I will care enough to fix them so long as the code compiles. It is a literal waste of CPU minutes in GitHub Actions to scan this file, and I have to up my time limit to 10m just to get it to be scanned.

If I were to add a //nolint: ignore-whole-file option or something, would this work? I suppose I'd still need to have a syntax scan to get the method fingerprints so return values can be checked in other source.

My use case isn't per-package as I in the same package I have code I do want to be scanned.

[ldez]

As I already explained: the issues reported for a file can be ignored but the files cannot be ignored by the analysis.

• documentation about nolint directives: https://golangci-lint.run/usage/false-positives/#nolint-directive
• documentation about exclusions configuration: https://golangci-lint.run/usage/false-positives/#exclude-or-skip

[falco467]

@skandragon can your code compile without this 250k file? If not than you're probably out of luck, since the whole code-base has to be compiled and linked by the go builder.

But I was thinking - should it not be possible to increase performance with more aggressive caching for excluded files? If the files are in subfolder e.g. other modules, should not build-caching allow the linker to reuse the binary without rebuilding? And the linter could also use cached resources for this file, since it is marked as included?
Or is this something we need to address upstream with the go build team ?

[ldez]

When the Go tooling supports that, we will be able to effectively ignore packages and files.
But for now, the possibilities are extremely limited.

[silverwind]

When the Go tooling supports that, we will be able to effectively ignore packages and files. But for now, the possibilities are extremely limited.

Could also just crawl the files without relying on the go tooling which would also solve other issues like having to run golangci multiple times with different build tags or GOOS values to be able to lint everything.

[ldez]

The Go tooling is the core of golangci-lint, golangci-lint relies on that to have its performance.
We will not reinvent the wheel.

[coin-au-carre]

I have also the same problem, I have a persistend database generated by Docker, in one of my subfolder and I would like that golangci does not try to look into it...

golangci-lint run --fix
ERRO [linters_context] typechecking error: pattern ./...: open /home/user/project/docker/persistent-data/db: permission denied 
make: *** [Makefile:83: lint] Error 7