Deprecation of godog CLI

[vearutop]

Historically godog started as a command-line tool that you can run in your project and it would parse your code and build test executor. This seems convenient at first, but comes with limitations.

Whenever you need something more than running tests (build tags, debugger, test coverage, etc...) you may find lack of control or difficult configuration. This comes from an unfortunate fact that godog CLI takes ownership of some responsibilities that are usually carried out by go tools, those features are rich and it is not feasible to keep up parity.

It is possible (and recommended) to run godog feature tests as subtests of *testing.T, this mode of operation aims to deliver same functionality as godog run with standard go test (and therefore all advanced Go features available too with IDE assistance).

In a recent discussion there was an idea to deprecate and phase out godog CLI to reduce complexity and maintenance effort.

I would like to learn from community if godog in CLI mode is still useful these days. Please vote and share your use cases for godog run/build that do not translate well into go test operation.

Thank you for using godog! ❤️

Relevant issues:
#154 #373 #325 #159

[cpustejovsky]

I couldn't get the godog CLI to work for my needs but when I plugged it into a testing.T framework, I found it very easy to use.

[wichert]

One thing I would like to be able to do is access the testing.T instance from a scenario so I can call T.Tempdir(). Dropping support for the godog cli will make that possible.

[aviator-ua]

Godog CLI does not intervene with the standard go test command behavior.
So it's nice to run quick Unit tests for whole project with simple go test (or press "Run Test" button in IDE), and run separate "heavy" Functional tests.

[vearutop]

Feature tests can also be configured to run separately from unit or other types of tests, a few strategies are discussed here: https://stackoverflow.com/questions/25965584/separating-unit-tests-and-integration-tests-in-go.

[sergiught]

Hey @vearutop 👋🏻

I believe it would be useful to take into consideration the upcoming release of Go version 1.20 and its potential benefits when used in conjunction with godog for running tests against application binaries. One notable advantage is the ability to accurately calculate coverage profiles, which has previously been a limitation when running tests with the godog CLI and testing as a black box. Please check: https://go.dev/testing/coverage/.

If anyone wants to try it out already you can by executing the following:

go install golang.org/dl/go1.20rc3@latest
go1.20rc3 download
go1.20rc build -cover -v -o /path/to/your/desired/binary /path/to/your/main.go
mkdir coverage && GOCOVERDIR=coverage godog run your.feature
go1.20rc3 tool covdata textfmt -i=coverage -o=coverage.out
go tool cover -func coverage.out

The benefit of having a CLI tool, is that it improves the developer experience, specially for folks that are just getting started with godog. They don't have to read through snippets on how to set it up with TestMain, they just execute a couple of CLI commands and are already up and running. You could further double down on the developer experience by introducing additional support commands.

I do however empathize with the maintainers point of view, that of reducing inner complexity and maintainability, as that will also allow you to ship new features faster perhaps. However your users don't care about that, they just want the best developer experience they can get.

[vearutop]

Thank you for feedback, it seems with active deprecation notice there are now more people interested in keeping CLI alive.

I think we can maybe reach a compromise on minimal maintenance burden and useful interface.
godog CLI can be simplified to a facade around go test. So instead of building and invoking instrumented binary, we could create a temporary XXX_test.go file and then execute go test [propagated-flags] XXX_test.go.

Perhaps, we can try to build an experimental prototype of such a tool that would declare and propagate all go test flags, while keeping some compatibility with original godog flags.

[midjones]

Is there an go test equivalent way to generate the cucumber test output? With the godog CLI we did it using godog -f cucumber

[vearutop]

Please check an example.

[midjones]

That worked! Thank you very much 😃

[dskarataev]

@vearutop If I do no use the godog command, how could I generate step definitions based on my .feature file? I remember previously it was possible to generate step definitions and just copy-paste them instead of creating manually

[vearutop]

If you bind godog CLI flags in a Go test, I'd expect most of godog CLI behavior to be available (including step hints), please check an example and see if it works for you:

https://github.com/cucumber/godog/blob/v0.14.1/_examples/godogs/godogs_test.go#L32

[Tochemey]

I will suggest to provide a good and clear document on how to generate go code from the gherkin files. I will not recommend that people should hand-craft the go code from the gherkin files. This will always lead to errors and frustrations. Therefore the cli should only be for generating the the go test files and then we can run them using go test.