Composite field {$ROWn} references are only correctly evaluated for other composite fields where the {$ROWn} reference occurs *after* the field being evaluated

[ohingardail]

If I define a dataset thus :
Order Column title Data type Options
1 A Composite B={$ROW2};C={$ROW3}
2 B Number range Between 1 and 10
3 C Composite {math equation="rand(1,10)"}
4 D Composite B={$ROW2};C={$ROW3}

I generate a dataset thus :
A, B, C, D
B=1;C=, 1, 3, B=1;C=3
B=4;C=3, 4, 8, B=4;C=8
B=3;C=8, 3, 3, B=3;C=3
B=6;C=3, 6, 7, B=6;C=7
B=6;C=7, 6, 10, B=6;C=10

That is,

• {$ROW2} (a number range) is correctly evaluated in all cases
• {$ROW3} (a composite field) is only correctly evaluated when it is referenced after row 3
• so it is null in column A in the first row
• and is the value of column C in the previous line in the first field if each subsequent row

The example shown here has been simplified to highlight the issue (and thus there are other solutions that get around the problem). The problem was noticed in a real world example when simulating a dataset wherein a summation field occurs before the fields being summed in a row, and those fields being summed are calculated using a complex formula in a composite field, and there is no obvious workaround (the field order of the CSV and the data within are pre-defined and cannot be changed).

My PHP is rubbish, so I don't really know how to fix the problem myself, otherwise I'd provide a patch.

GenerateData version : 3.0.2 (installed locally on CentOS 6.3)
PHP version : 5.3.3
MySQL version : 14.14 / 5.1.61

[benkeen]

Brilliant, thanks for the extremely clear write-up of the problem.

I'll take a look at this this weekend. Thanks!

• Ben

[benkeen]

Hey @ohingardail,

This is a tricky one! The problem is that any field can reference any other - and users could even put in a circular reference. If that occurs, there's no possible way to make it work.

What I think I'll do is the following:

1. Add some custom code to detect and prevent circular references.
2. Update the backend code to intelligently generate the fields in the correct order.

The second one is going to be pretty hard, I'll be blunt. It'll mean examining the content of each Composite field's "Options" value and determining a generation order that will ensure they all get generated. Then I'd need to update the way the Core passes the information to the Data Types so that for Composite fields it'll get sent ALL fields that are Composite, thus ensuring it has enough information to work with.

So I think I'll pull this out of 3.0.3 and put it into 3.0.4. I'd like to get some other bug fixes out the door today and don't want them to be held up.

Thanks!

• Ben

[benkeen]

Yikes, this one has been put off and off... I'll put it back into 3.0.9 just so I don't forget it. It IS a tricky one!

[benkeen]

After more investigation, I'm still at a loss how to property solve it. Since it's such a fringe case, I'm pulling it out of 3.0.9.