(1)Output should fit inside a single term. Increase MaxTermSize?

[jodavies]

Hello,
This relates to the discussion in #289 (comment)
I have managed to extract a minimal crashing example from a larger set of codes: (rename as .frm)
test-minimal.txt

The example has two modes, inside a function argument (`#define DOINARG') or not.

Outside the function argument, everything runs OK. It also checks in each loop whether the current expression does indeed fit inside a function argument; it does.

Computing inside a function argument, however, it always crashes, however large I set MaxTermSize (10M, for eg, does not work).

For the first (commented) example input, it works for MaxTermSize values larger than 42K. With the default, (with my patches), I get the output

Sorted function argument too long: 41004 words
MaxTermSize: 40000 words
Called from execarg

which is a different error from the crashing second input term.

The code which this has been taken from has run successfully on many millions of very similar input terms. This one causes problems.

Any ideas?
Thanks,
Josh.

[jodavies]

OK, I think the issue is that if you arrive at https://github.com/vermaseren/form/blob/master/sources/sort.c#L963 the problem is not that MaxTermSize is too small, but actually you need larger SubSmallSize or SubTermsInSmall etc.

The documentation does describe subfilepatches, sublargepatches, sublargesize though, should multi-stage sorting not be working here?

Thanks,
Josh.

[vermaseren]

To my knowledge that ‘subsystem’ works, but indeed, you may have to set those to larger values than the default. Much of this was designed before the polyratfun existed and in those days it was hardly ever used. The polyratfun however makes these parameters much more important. Jos

…

On 24 Aug 2018, at 10:13, jodavies ***@***.***> wrote: OK, I think the issue is that if you arrive at https://github.com/vermaseren/form/blob/master/sources/sort.c#L963 <https://github.com/vermaseren/form/blob/master/sources/sort.c#L963> the problem is not that MaxTermSize is too small, but actually you need larger SubSmallSize or SubTermsInSmall etc. The documentation does describe subfilepatches, sublargepatches, sublargesize though, should multi-stage sorting not be working here? Thanks, Josh. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#292 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEnZTSaGjdKqIgaxiwYx3vwYzH4z0ks5uT7WngaJpZM4WJ8KY>.

[jodavies]

Here is a simpler example,

#-                             
                               
#: SubTermsInSmall 12K         
#: MaxTermSize 200K              
                               
CFunction f;                   
AutoDeclare Symbol x;          
                               
Local test1 = x;               
Local test2 = f(x);            
                               
Argument f;                    
   Identify x = (x1+...+x10)^7;
EndArgument;                   
Identify x = (x1+...+x10)^7;   
                               
.end                           

which gives

Time =       0.01 sec    Generated terms =      11440
           test1         Terms in output =      11440
                         Bytes used      =     513660

Time =       0.02 sec    Generated terms =          1
           test2         Terms in output =          1
                         Bytes used      =     675000

So, test1 suggests that the result can easily fit into a single term of 200K words (800KB).

SubTermsInSmall 12K means that we never sort the sub small buffer into the sub large buffer. If this setting is commented out, (the default is 10K) then we get (1)Output should fit inside a single term. Increase MaxTermSize?

It seems something bad happens if the sub small buffer must be sorted to sub large buffer?

Thanks,
Josh.

[jodavies]

In this example, it is a par=1 EndSort with 1 (or more, if the powers in the test script are increased) large patches. We run AR.outfile = newout = AllocFileHandle(0,(char *)0); so newout->handle is -1. We go into MergePatches here:

form/sources/sort.c

Line 918 in 743ed4f

if ( MergePatches(2) ) {

after which, newout->handle is 3, set by PutOut:

form/sources/sort.c

Line 1437 in 743ed4f

fi->handle = (WORD)RetCode;

This causes the message and termination at:

form/sources/sort.c

Line 960 in 743ed4f

else if ( newout->handle >= 0 ) { /* output too large */

It seems that it is assumed that if PutOut created a file, then the output doesn't fit in MaxTermSize. In this example it should certainly fit, however.