Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Functional Parameters #45

Open
12AT7 opened this issue Feb 15, 2020 · 3 comments
Open

Feature Request: Functional Parameters #45

12AT7 opened this issue Feb 15, 2020 · 3 comments

Comments

@12AT7
Copy link
Contributor

12AT7 commented Feb 15, 2020

My favorite Mettle feature is the template parameters to instantiate tests on different types. However, I have also been experimenting with function parameters to solve a different kind of testing problem. Specifically, I need to sample my tests with different numerical parameters. They are always the same type, and there might be a great many of them. While these parameters can be coaxed into compile time arguments, the Mettle compile times then become prohibitive. Some test patterns are just more naturally expressed via sets of function parameters. This is particularly true in the scientific programming I normally do.

Here I demonstrate a prototype mettle::enumerate<>(...) that is supposed to emulate mettle::subsuite<>(...) as closely as possible, except for these differences:

  • It accepts a standard container of parameter values. Actually, it accepts any class that conforms to the range based for interface.
  • the name argument for the tests use a libfmt pattern instead of a plain string, and
  • The subsuite created is populated by enumerating the container, not by explicit _.test() invocations.

I am completely unmarried to my names here, and I am certain extra wizardry would be required to fully support important Mettle features like fixtures and marks. I am posting this as a feature request, not a fully formed pull request, because I am sure some design discussion up front would be helpful.

From my code sample below, this is the output that I get. I am not actually making any test assertions (those work fine); I am more interested here is seeing which tests are collected and how the names are formatted.

bin/ut.enumerate -o verbose                    (02-15 15:54)
enumerate

  vector<int>
    i=0 PASSED
    i=1 PASSED
    i=3 PASSED
    i=4 PASSED
    i=5 PASSED

  vector<array<int,2>>
    i=0,j=0 PASSED
    i=0,j=1 PASSED
    i=3,j=4 PASSED
    i=5,j=5 PASSED

  vector<tuple<int, float>>
    i=0,j=17.0 PASSED
    i=0,j=42.0 PASSED
    i=3,j=37.0 PASSED

12/12 tests passed

Here is my investigation:

#include <mettle.hpp>
#include <tuple>
#include <functional>
#include <fmt/format.h>

namespace mettle {

// format_index() wraps fmt::format(...) to accept a std::array or std::tuple
// of arguments instead of a varargs list.  Ideally, this would be a function
// of libfmt.
template <typename Tuple>
auto format_index(const std::string format, Tuple&& index) ->
decltype(std::tuple_cat(index), std::string())
{
    return std::apply([format](const auto&... i){ 
            return fmt::format(format, i...); 
            }, std::tuple_cat(index));
}

template <typename T>
auto format_index(const std::string format, T index) ->
typename std::enable_if<std::is_scalar<T>::value, std::string>::type
{
    return fmt::format(format, index);
}

// mettle::enumerate<> is supposed to have similar semantics to
// mettle::subsuite<>, except the parameters are dynamic parameters instead of
// template parameters.  This pattern may be useful in preference to
// subsuite<Args...>, when the number of Args is large, and/or there is no
// useful reason to vary the argument type for each test instance.  The API
// difference is the support of the Generator, which is a container (or generic
// object that obeys the range for interface), to instantiate a subsuite of
// tests, one per element of the range.  The element is very similar in purpose
// to a fixture, but that gets a unique but programmable value for each test.
// The second new argument, "name_format", is a libfmt compatible std::string
// that will serve as the "name" argument of each test.
template <typename Builder, typename Generator, typename Func, typename...Args>
auto enumerate(
        Builder&& builder, 
        std::string subsuite_name,
        std::string name_format,
        const Generator& generator,
        Func&& func
        )
{
    return mettle::subsuite<Args...>(builder, subsuite_name, 
            [name_format, generator, func](auto& _) 
            {
                for (auto idx: generator) {
                    _.test(format_index(name_format, idx), [idx, func]() { func(idx); });
                }
        });
}

} // namespace mettle

mettle::suite<> suite("enumerate", [](auto &_) {

    mettle::enumerate(_, "vector<int>", 
            "i={}", 
            std::vector<int> { 0, 1, 3, 4, 5 },
            [](auto idx) {});

    mettle::enumerate(_, "vector<array<int,2>>", 
            "i={},j={}", 
            std::vector<std::array<int, 2>> { {0,0}, {0,1}, {3,4}, {5,5} },
            [](auto idx) {});

    mettle::enumerate(_, "vector<tuple<int, float>>", 
            "i={},j={}", 
            std::vector<std::tuple<int, float>> { {0,17.0f}, {0,42.0f}, {3,37.0f} },
            [](auto idx) {});

    });


@jimporter
Copy link
Owner

I'm not sure about this. I intended for this to be handled with a simple for loop or std::for_each or whatever the ranges version of that is. If we lived in a world where there were easy, standard ways to iterate over types, I might never have added support for type-parameterization either.

On the other hand, there's a certain symmetry to providing a special creator for value-parameterized suites. Though on the third(?) hand, "symmetry" is a pretty weak argument.

@12AT7
Copy link
Contributor Author

12AT7 commented Feb 16, 2020

The way that this works basically is exactly that; just a for loop over a container (and, presumably, ranges will be similar and awesome). In my use, I actually have a custom generator class that I use as well. It is definitely not required that this function be part of Mettle itself to be useful in applications; it is easy enough to adapt this outside of mettle as I have done.

This is sort of similar to the discussion we had a couple of years ago about supporting properties based testing. It has the similar feel of generating a batch of tests from a programmable rule. It has been a while since I looked at that, but I seem to remember that we settled on a similar pattern of using a free function to populate the suite in a specialized way.

You can close this issue, or use it as an example, or whatever you want to do with it. If others find it useful, then it is still going to be here in issues as a reference.

@jimporter
Copy link
Owner

I thought about this a bit more, and I think there's some value in doing this, since it's a bit more difficult to parameterize root-level suites with multiple values (obviously, you can't have a for loop at global scope). I'll probably do the same sort of thing for test cases too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants