Improve data pool node

[fabio-miranda]

As of now, the data pool node is listing every single row of a dataframe. It needs to be changed so that it lists only the top 100 or so rows.

[makaveli2P]

Hi, I started exploring this issue and found that there is already code limiting the display to 100 rows:

curio/utk_curio/frontend/urban-workflows/src/components/DataPoolBox.tsx

Lines 714 to 715 in 5be75de

	{outputTable
	.slice(0, 100)

However, I wasn't able to replicate the issue where all rows are displayed of a dataframe. Is my understanding of this issue correct? If not, could you provide more information/explanation about this behavior and what is expected?

[fabio-miranda]

Good point, I’d overlooked that :)
But even if it's only listing 100 rows, still all the data is being sent to the frontend. It is a convoluted flow of data between sandbox, backend, and frontend. I have tried to simplify this flow with this commit by keeping the data in the backend and saving binary files in the .curio folder. So what is being passed between nodes is the path of these binary files rather than the data itself. However, the data pool node is special, as it serves two purposes: (1) visualize the data as a table, and (2) act as an intermediate for the interaction of visualization nodes. Because of that, it is the only case (other than the visualization nodes of course) that requires the entire data.

There should be a better way to handle this special data pool case.

[makaveli2P]

I see, so the real issue is full dataset transfer. Even though only 100 rows are displayed, the entire dataframe is being sent from backend to frontend, which is an architectural bottleneck and not a display issue with the number of rows. Is that correct?

I'll explore how a change can be implemented without breaking the existing functionality.