Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shrinking doesn't seem to work as advertised for one_of/frequency/tree #97

Open
obrok opened this issue Apr 13, 2018 · 0 comments · May be fixed by #98
Open

Shrinking doesn't seem to work as advertised for one_of/frequency/tree #97

obrok opened this issue Apr 13, 2018 · 0 comments · May be fixed by #98
Labels

Comments

@obrok
Copy link
Contributor

obrok commented Apr 13, 2018

I'm using stream_data to generate a rather large data structure representing an SQL query. I was having problems shrinking the generated examples - it seemed like only minor details like integer values were being shrunk while the query structure as a whole remained the same. I managed to reduce it to the following property (should work after pasting into stream_data tests):

    property "shrinks towards earlier generators" do
      check all size <- positive_integer(),
                seed <- {integer(), integer(), integer()} do
        {:error, result} =
          one_of([:foo, {:bar, integer()}])
          |> check_all(
            [max_shrinking_steps: 100, initial_size: size, initial_seed: seed],
            fn example -> {:error, example} end
          )

        assert result.nodes_visited < 100
        assert :foo = result.shrunk_failure
      end
    end

Error:

  1) property one_of/1 shrinks towards earlier generators (StreamDataTest)
     test/stream_data_test.exs:199
     Failed with generated values (after 5 successful runs):

         * Clause:    size <- positive_integer()
           Generated: 4

         * Clause:    seed <- {integer(), integer(), integer()}
           Generated: {3, 0, 0}

     match (=) failed
     code:  assert :foo = result.shrunk_failure()
     right: {:bar, 0}

Curiously - it does seem to be dependent on the size, even though integer/1 which is used by one_of seems to operate independently of size. If I just set size <- constant(1), then the property passes.

I did some tracing and it does seem like for small initial_size the initial failure found is {:bar, _} and after some shrinking it gets reduced to :foo but that doesn't seem to happen for a larger initial_size. It doesn't seem to happen when the generator tested is just one_of([:foo, :bar]).

This also affects frequency/1 as it has almost the same implementation with the minor difference of how the option is chosen from the list and at the very least tree/2, which is how I originally found it. I assume it also affects anything else that uses frequency/1.

I'd be happy to make some more attempts at fixing this, but I haven't yet understood the code dealing with shrinking fully, so I wanted to make sure I'm on the right track here and also ask for any advice you might have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants