add docs on detailed error messages on assertion failures

[vak]

Hi

many thanks for your time invested in BATS. BATS is really helping.

It would be nice, if BATS developers could add a few lines in docs about the way of printing detailed error messages from failing assertions.

The easiest way to get a hint on what happens in a failing assertion is to write assertions with comments like:

  ...
  [[ "${lines[0]}" = "Request succeeded for "* ]] # request succeeded?

So, when it happens one could see not only the corresponding failing statement of the .bats test file, but also the comment line in addition. However if we'd like to see in test logs some hint on why it happened (e.g., invocation details, response details, variables, etc) then we'd need to echo details to stderr in some BATS-friendly manner.

One could add perhaps the following recipe for the BATS-beginners.

# add this definition to your BATS-test or in the source'd-in helper script:
exit_with_error() {
  echo "$1. Exit code: $status" >&2
  exit $status
}

and use it like:

  ...
  cmd="send_request arg1 arg2"
  run $cmd
  [ "$status" -eq 0 ] || exit_with_error "'$cmd' failed."
  [[ "${lines[0]}" = "Request succeeded for "* ]] || exit_with_error "'$cmd' failed, got '${lines[0]}' on stdout" # request succeeded with the correct output?
  ...

or maybe you have even better way to print out it :)

kind regards,
Valery

[ztombol]

I think you are looking for the bats-* family of test helpers. They were created for this exact purpose. Giving useful feedback on failing tests.

The helpers are separated into libraries based on their role.

• bats-assert - common assertions, e.g. output and status
• bats-file - file related assertions, e.g. existence

To help others write similar helpers, the underlying primitives, e.g. output formatting, have been abstracted out into bats-support.

Check out the Readme of each library and the shared documentation bats-docs.

For the philosophy and design of these libraries see the discussion in #110.

[vak]

@ztombol, it's quite a pity that your nice helpers were not merged into BATS and even more pity that they were not mentioned in BATS' docs. I have implemented all my tests using "plain" BATS+bash infrastructure before I got your answer.

By the way, I used BATS to test REST API of a web-service. Now I am asked to output all curl-invocations that have been issued against the web-service. That is, print out all commands passed during invocations of the BATS "run" function. Probably, with your helpers such a verbosity would have been easy to add.

Well, said all that, I am looking forward to use your code in my next tasks.

[ztombol]

@vak The bats-* libraries not being a part of Bats is a good thing. If you follow the discussion in #110 you'll see that there are more pros to making it separate. I, too, initially suggested we include them in Bats, but was persuaded that it's better this way. The biggest reason for me was that development is much easier this way (an update to one of the libraries doesn't require a new Bats release).

Bats has been understaffed for some time (hence the lack of a new releas or merged PRs). The author started gathering manpower in #150 to get development going again, and it's been going well! I've been writing up a list of changes I will suggest to be addressed in the coming releases. Adding a reference to bats-* libraries is one of my priority.

Don't worry if you have already completed your test suite without the use of these libraries. Migration is trivial and quick. Simple regular expression find-and-replace commands will do that for you in no time. I did it for one of my projects.

Regarding your question. The assertions in bats-* only produce output when they fail. If you need feedback on passing assertions you can write a simple helper like below.

#!/usr/bin/env bats

# Log into `test.log'.
log() {
  echo "$@" >> 'test.log'
}

# Test a call to the REST API and log all passing calls.
assert_rest_call() {
  local -r command="curl <options> $@"
  run "$command"

  if (( status )); then
    # Add failure details to be displayed by bats here!
    return 1
  fi

  log "$command"
}

@test 'REST API call' {
  assert_rest_call '<call>'
}

I suggest you write general purpose assertions and helpers and compile them into a helper library for testing REST APIs (or maybe someone has already done so?). This way you can use it in multiple projects and if you publish it others can help you improve it. For examples, see the libraries listed in bats-docs.