Long expression generates an error "too many inputs"

[yonghah]

Problem description

When I send a reqeust with a complex expression (and hence long such as total url's length = 1851 characters), titiler returns "too many inputs" error. When I change the request with the same url and only shrinked expression, titiler returns the result without any issue.

Is there any setting for maximum query length? If so, how can I relax the condition?

Expected Output

Return image response

Environment Information

Titiler 0.11.6 / OAS3
Deployed to AWS with Lambda on Python 3.10

The request with a problem:
https://ou60yuvp0i.execute-api.us-east-1.amazonaws.com/cog/tiles/14/8193/8190.png?url=https://stereo-e16.s3.amazonaws.com/raster/factor/stereo_cog_r1.tif&expression=0.57103%2Ab1%2B0.99672%2Ab2%2B0.96904%2Ab3%2B0.20021%2Ab4%2B0.27429%2Ab5%2B0.79971%2Ab6%2B0.72686%2Ab7%2B0.94084%2Ab8%2B0.18491%2Ab9%2B0.82333%2Ab10%2B0.2628%2Ab11%2B0.93301%2Ab12%2B0.38127%2Ab13%2B0.24539%2Ab14%2B0.10738%2Ab15%2B0.67462%2Ab16%2B0.27597%2Ab17%2B0.27698%2Ab18%2B0.50362%2Ab19%2B0.8538%2Ab20%2B0.27273%2Ab21%2B0.74617%2Ab22%2B0.4796%2Ab23%2B0.22111%2Ab24%2B0.65345%2Ab25%2B0.11374%2Ab26%2B0.09267%2Ab27%2B0.98799%2Ab28%2B0.24427%2Ab29%2B0.18995%2Ab30%2B0.15173%2Ab31%2B0.99163%2Ab32%2B0.59891%2Ab33%2B0.88331%2Ab34%2B0.97323%2Ab35%2B0.89888%2Ab36;0.04474%2Ab1%2B0.60977%2Ab2%2B0.43399%2Ab3%2B0.67842%2Ab4%2B0.39043%2Ab5%2B0.17055%2Ab6%2B0.11867%2Ab7%2B0.35566%2Ab8%2B0.94484%2Ab9%2B0.91253%2Ab10%2B0.30639%2Ab11%2B0.81236%2Ab12%2B0.98909%2Ab13%2B0.23044%2Ab14%2B0.81381%2Ab15%2B0.98246%2Ab16%2B0.97092%2Ab17%2B0.46153%2Ab18%2B0.99879%2Ab19%2B0.2217%2Ab20%2B0.5304%2Ab21%2B0.95593%2Ab22%2B0.01583%2Ab23%2B0.15223%2Ab24%2B0.07902%2Ab25%2B0.90673%2Ab26%2B0.86554%2Ab27%2B0.51667%2Ab28%2B0.60937%2Ab29%2B0.07176%2Ab30%2B0.74472%2Ab31%2B0.68408%2Ab32%2B0.99638%2Ab33%2B0.86553%2Ab34%2B0.74682%2Ab35%2B0.27824%2Ab36;0.00529%2Ab1%2B0.17842%2Ab2%2B0.10294%2Ab3%2B0.96833%2Ab4%2B0.85393%2Ab5%2B0.0152%2Ab6%2B0.00695%2Ab7%2B0.07031%2Ab8%2B0.60713%2Ab9%2B0.20663%2Ab10%2B0.72968%2Ab11%2B0.22667%2Ab12%2B0.42386%2Ab13%2B0.59142%2Ab14%2B0.83484%2Ab15%2B0.2196%2Ab16%2B0.51653%2Ab17%2B0.93309%2Ab18%2B0.32701%2Ab19%2B0.02677%2Ab20%2B0.98461%2Ab21%2B0.20406%2Ab22%2B0.01055%2Ab23%2B0.42558%2Ab24%2B0.00408%2Ab25%2B0.69651%2Ab26%2B0.7623%2Ab27%2B0.13883%2Ab28%2B0.99697%2Ab29%2B0.23217%2Ab30%2B0.91416%2Ab31%2B0.20706%2Ab32%2B0.26038%2Ab33%2B0.21719%2Ab34%2B0.22536%2Ab35%2B0.04243%2Ab36

Request without any problem:
https://ou60yuvp0i.execute-api.us-east-1.amazonaws.com/cog/tiles/14/8193/8190.png?url=https://stereo-e16.s3.amazonaws.com/raster/factor/stereo_cog_r1.tif&expression=0.57103%2Ab1%2B0.99672%2Ab2%2B0.96904%2Ab3%2B0.20021%2Ab4%2B0.27429%2Ab5%2B0.79971%2Ab6%2B0.72686%2Ab7%2B0.94084%2Ab8%2B0.18491%2Ab9%2B0.82333%2Ab10%2B0.2628%2Ab11%2B0.93301%2Ab12%2B0.38127%2Ab13%2B0.24539%2Ab14%2B0.10738%2Ab15;0.04474%2Ab1%2B0.60977%2Ab2%2B0.43399%2Ab3%2B0.67842%2Ab4%2B0.39043%2Ab5%2B0.17055%2Ab6%2B0.11867%2Ab7%2B0.35566%2Ab8%2B0.94484%2Ab9%2B0.91253%2Ab10%2B0.30639%2Ab11%2B0.81236%2Ab12%2B0.98909%2Ab13%2B0.23044%2Ab14%2B0.81381%2Ab15;0.00529%2Ab1%2B0.17842%2Ab2%2B0.10294%2Ab3%2B0.96833%2Ab4%2B0.85393%2Ab5%2B0.0152%2Ab6%2B0.00695%2Ab7%2B0.07031%2Ab8%2B0.60713%2Ab9%2B0.20663%2Ab10%2B0.72968%2Ab11%2B0.22667%2Ab12%2B0.42386%2Ab13%2B0.59142%2Ab14%2B0.83484%2Ab15

[vincentsarago]

that's really interesting.

you expressions are

['0.57103*b1+0.99672*b2+0.96904*b3+0.20021*b4+0.27429*b5+0.79971*b6+0.72686*b7+0.94084*b8+0.18491*b9+0.82333*b10+0.2628*b11+0.93301*b12+0.38127*b13+0.24539*b14+0.10738*b15+0.67462*b16+0.27597*b17+0.27698*b18+0.50362*b19+0.8538*b20+0.27273*b21+0.74617*b22+0.4796*b23+0.22111*b24+0.65345*b25+0.11374*b26+0.09267*b27+0.98799*b28+0.24427*b29+0.18995*b30+0.15173*b31+0.99163*b32+0.59891*b33+0.88331*b34+0.97323*b35+0.89888*b36', '0.04474*b1+0.60977*b2+0.43399*b3+0.67842*b4+0.39043*b5+0.17055*b6+0.11867*b7+0.35566*b8+0.94484*b9+0.91253*b10+0.30639*b11+0.81236*b12+0.98909*b13+0.23044*b14+0.81381*b15+0.98246*b16+0.97092*b17+0.46153*b18+0.99879*b19+0.2217*b20+0.5304*b21+0.95593*b22+0.01583*b23+0.15223*b24+0.07902*b25+0.90673*b26+0.86554*b27+0.51667*b28+0.60937*b29+0.07176*b30+0.74472*b31+0.68408*b32+0.99638*b33+0.86553*b34+0.74682*b35+0.27824*b36', '0.00529*b1+0.17842*b2+0.10294*b3+0.96833*b4+0.85393*b5+0.0152*b6+0.00695*b7+0.07031*b8+0.60713*b9+0.20663*b10+0.72968*b11+0.22667*b12+0.42386*b13+0.59142*b14+0.83484*b15+0.2196*b16+0.51653*b17+0.93309*b18+0.32701*b19+0.02677*b20+0.98461*b21+0.20406*b22+0.01055*b23+0.42558*b24+0.00408*b25+0.69651*b26+0.7623*b27+0.13883*b28+0.99697*b29+0.23217*b30+0.91416*b31+0.20706*b32+0.26038*b33+0.21719*b34+0.22536*b35+0.04243*b36']

which needs all those bands (36)

['b35', 'b26', 'b27', 'b17', 'b10', 'b31', 'b25', 'b4', 'b34', 'b29', 'b36', 'b24', 'b6', 'b14', 'b22', 'b33', 'b3', 'b9', 'b30', 'b5', 'b1', 'b13', 'b19', 'b12', 'b11', 'b28', 'b23', 'b2', 'b20', 'b7', 'b8', 'b15', 'b18', 'b16', 'b21', 'b32']

And I think the error comes from https://github.com/pydata/numexpr/blob/master/numexpr/interpreter.cpp#L1024-L1027 and then is related to numpy/numpy#4398

As soon as you have less than 32 bands in your expression it should work 😓

This is really unfortunate and not a TiTiler issue

We could maybe optimized rio-tiler to only pass the needed bands per bloc in https://github.com/cogeotiff/rio-tiler/blob/main/rio_tiler/expression.py#L74 (replacing local_dict=dict(zip(bands, data))) but it's not a quick fix and will not resolve the real issue from Numpy/Numexpr.

While I admire that you are using expression for this (I never thought to see this kind of large expression 😅) I think using the custom algorithm might be better suited

Note: I'm going to move this to discussion because it's not a BUG in titiler

[yonghah]

It was numexpr! Thank you so much for the pointer. Yes. We have 36 variables and we need to find a way to visualize 36 dimensions into R,G,B space. So we applied some kind of dimensionality reduction and the coefficients are the weights for this transformation. This was very useful to explore the spatial relationship between the variables. I'll look into custom alogorithms for this. Thank you for your answer and this awesome package!