Table Visualization Requirements

[jdunkerley]

Here's my summary of the grid features we want and the ag-grid investigation.

Wanted features for the read-only grid:

• Dense data display: old table had too low a data density. Large values must be truncated to fit in the column and shown in a tooltip on hover. The current ag-grid doesn't hover.
  • Agreed that nested data display should be removed and just shown as a summary.
• Support for all data types: need to render the Enso objects and the primitive types. This is a change in the JSON format sent - Enso should include a _DISPLAY_VALUE_ field with the string representation of the value. Tooltip would ideally show the full structure.
• Any scale: loading more data than 1,000 rows (our default page size). Community supports infinite scroll; Enterprise supports the more varied models. It needs to support high numbers of columns as well as rows.
• Resize Viz popup: the ability to make the shown viz any size the user wants and be persisted with the node even if open and closed.
• Column resizing: want to be able to resize columns to fit on screen. The data drive automatic sizing.
• Column reordering and selecting: the ability to look at a subset of the columns and move them around. The community edition has basic support for this. Enterprise adds the nicer column picker.
• Sorting and Filtering: the ability to sort and filter data. ag-grid community can only work for the visible set - current implementation only enables if the full set is shown. This would need to be driven by engine call-backs - this is possible within ag-grid enterprise edition but would need to be investigated.
• Clipboard support: copying values from the grid to the clipboard. This should either be the whole grid or the selection. It should allow choosing to include the column headers or not. The result should be pasteable into Excel or Google Sheets. ag-grid Community doesn't enable clipboard support; only Enterprise does.
• Keyboard navigation: the ability to move around within the grid.

Nice to have features for the grid:

• Value Inspector: not technically part of the grid, but the ability to drill into a single value such as a large text value and explore.
• Ability to paste values as a table node: not technically part of the grid, but if we could paste a table of values into the IDE and get a simple look-up table, this would make creating such things much easier.
• Ability to export the grid: saving it quickly to a file (CSV, Excel). The community edition supports only CSV. Enterprise supports a wider range of formats. This would be replaced to callbacks to the engine to export all data - again, the Enterprise server-side data model supports this.
• Ability to add nodes back into the workflow: nice to add a sort or a filter back into the workflow as connected nodes. Ideally, this would be a general part of the visualization API so we can use the visualizations to drive data discovery.
• Data quality indicator: custom rendering on the header to show the data quality. This would be a bespoke extension to ag-grid, creating a new header renderer. The Enso side would need to provide the data quality information to the grid in JSON.
• State serialization: restore hidden columns, column order, custom width, filters, sorts, and row selection - ideally sticking across viz modes.

I'm not considering the editable grid at the moment, but we would want the ability to edit and create table literals as soon as feasible easily.

[jdunkerley]

Attached below is the test project I was using for ag-grid as it's a good test set of all our table viz modes.

Table_Testing.zip

[jdunkerley]

The current implementation need to have:

• resizing sorted when shrinking back from full screen. Current column sizing makes this problematic
• the header row and the viz selector do not play nicely together and probably adding a status row with row count would be a nice way to get around this issue.
• limited to first 1000 rows as based on the old table viz, for community edition would need to wire up the infinite scroll to work with lazy data fetching.
• tool tip support for larger values.

The next steps would be:

• a data quality indicator PoC (ideally showing %NULL first and later other issues). This would need changes to the data serialization within the library side.
• the data serialization should be improved for objects so that a display format is sent. This would include not nesting downwards in the full structure.

There are possibilities to work around the limitations of the community edition for the time being but we should decide (and investigate options) of using enterprise as this would mean no need to implement things like clipboard, column selector, and export. This would only be feasible within the paid for product.

[wdanilo]

This is amazing, @jdunkerley!