Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential issue with shrinking when combined with scaling #107

Open
keathley opened this issue Jul 17, 2018 · 0 comments
Open

Potential issue with shrinking when combined with scaling #107

keathley opened this issue Jul 17, 2018 · 0 comments
Labels

Comments

@keathley
Copy link
Contributor

I'm not sure if the title is correct or makes sense. Generally I'm having a hard time getting reasonable counter-examples.

I have some code that needs to filter a list of elements based on a provided whitelist. The test for this looks like this:

defmodule MyTest do
  def good_fields(whitelist) do
    sized fn size ->
      whitelist
      |> one_of
      |> list_of(min_length: size)
      |> map(fn list -> Enum.map(list, &Atom.to_string/1) end)
    end
  end

  def whitelist do
    bind scale(list_of(atom(:alphanumeric), min_length: 1), & &1*3), fn list ->
      constant(Enum.uniq(list))
    end
  end

  describe "whitelist/2" do
    property "returns only the whitelisted fields" do
      check all \
        whitelist <- whitelist(),
        real      <- good_fields(whitelist),
        garbage   <- scale(list_of(binary()), & &1*4)
      do
        fields = real ++ garbage
        assert FieldSelector.whitelist(fields, whitelist) == real
      end
    end
  end
end

The actual issue is that I didn't ensure that the garbage fields were not included in the whitelist. StreamData wouldn't find this issue until I scaled both the whitelist generator and the garbage generator. Once I had done that I got this counter example:

  1) property whitelist/2 returns only the whitelisted fields
     Failed with generated values (after 24 successful runs):
     
         * Clause:    whitelist <- whitelist()
           Generated: [:f3s, :mehtwbp, :aRknJKop0hFvA873k1@of3, :P5U_VEToigea5IhP, :_mEylhu, :I, :gTj2sLvk8DSf@Y, :pEpfY87ZPWoc1nMgUK_s0F, :fNHuckgqQmLfJhb32k3dg, :nFX, :szyUhwHfrSF0jX_03qNr@PxLlj, :oz, :e5yQ5EQx2M87M13EwpSLktAwQ, :jZiDjaKT@, :pSxD4DIy7yrMGJNz, :mH9O_XKAg4@0ES6bEy65NF2H, :_li9HD5zl5qU4SN, :qGOGFYfuhkWxuSww, :ebKfOTMcgOom0Yx01bC8e1az, :_LgHwcU, :VsZfa4J, :_IjCozJ9ZpBTHulF, :_, :JYf, :vWX4f2, :_FjJSsaxq456CMRxXBVA@, :_EKG9NLyw103Ft0T0, :Nxp2WmQRSmCEQM, :_j6A, :a, :Qa@AtebCMFv_FYSbE7wooT52, :w@fl6vk6Y1, :lR83aOqI, :pVTD@@I, :g5cj8Cs6rIJ5KJnV0b, :f8lJjMY6YjhCzpy@0l, :ig, :h87XKO3E9DIOL@nn22I, :AUgq, :nhGolyOG_Tt6sonFAFw7a6mrC, :TN78mT0JcFD@vkPQMl, :kGwaw9y, :_19, :xIZEIm, :O7@g0ceYE, :tjzM, :nHIsiP5RW54r, :aC8AOE_AsOCiMC, :_E5DlrUqs8mcSWyjYerfkBQLPN]
         
         * Clause:    real <- good_fields(whitelist)
           Generated: ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU"]
         
         * Clause:    garbage <- scale(list_of(binary()), &(&1 * 4))
           Generated: ["I", <<122, 255, 43, 70, 243, 198, 209, 254, 82, 204, 29, 85, 140, 141, 191, 173, 159, 138, 168, 41, 39, 114, 95, 97, 38, 109, 174, 9, 83, 228, 155, 4, 169, 188, 249, 10, 26, 207, 141, 186, 88, 164, 136, 233, 63, 190, 206, 136, ...>>, <<131, 74, 242, 191, 247, 124, 229, 255, 214, 174, 143, 28, 146, 29, 88, 80, 53, 34, 177, 251, 31, 62, 204, 6, 62, 24, 153, 239, 151, 142, 225, 96, 217, 244, 138, 78, 136, 7, 92, 176>>, <<36, 176, 143>>, <<126, 166, 52, 245, 162, 110, 203, 181, 235, 11, 92, 252, 183, 115, 198, 191, 3, 160, 203, 159, 43, 220, 37, 73, 134, 163, 241, 204, 18, 234, 76, 169, 25, 176, 122, 107, 51, 216, 28, 157, 191, 43, 43, 245, 233, ...>>, <<105, 89, 96, 5, 54, 36, 29, 169, 245, 241, 43, 211, 107, 115, 107, 200, 77, 96, 106, 159, 153, 70, 193, 190, 124, 133, 2, 224, 16, 33, 95, 249, 238, 153, 143, 100, 49, 241, 186, 165, 35, 172, 253, 103, ...>>]
     
     Assertion with == failed
     code:  assert FieldSelector.whitelist(fields, whitelist) == real
     left:  ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU", "I"]
     right: ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU"]

The minimal counter example is:

whitelist = [:I]
real = []
garbage = ["I"]

It seems like stream_data should be able to further shrink this counter example by quite a bit. Is there some interaction between scaling and shrinking? Is there some other way to improve the shrinking here?

@whatyouhide whatyouhide changed the title Potential issue with shrinking when combined with scaling. Potential issue with shrinking when combined with scaling Mar 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants