Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csv: limit how much data is downloaded for the preview #113

Open
slint opened this issue Apr 2, 2020 · 2 comments
Open

csv: limit how much data is downloaded for the preview #113

slint opened this issue Apr 2, 2020 · 2 comments

Comments

@slint
Copy link
Member

slint commented Apr 2, 2020

In cases where the delimiter is wrongly detected or possibly, the rows are too big, the previewer tries to fetch too much data from the file, which of course is terrible if someone is on a mobile data connection. To solve this a (configurable) data limit has to be applied to the JavaScript that fetches the initial rows from the CSV file.

See for example the previewer trying to work with a 1.2GB CSV file: https://zenodo.org/record/3731542

@tmorrell
Copy link
Contributor

Should have been resolved with #171

@slint
Copy link
Member Author

slint commented Aug 29, 2023

Should have been resolved with #171

Correct! But it was done in the maint-1.0.0alpha branch, so I opened a "forward-port" (?) PR at #185.

The solution differs from the one I described in the initial issue since it applies a strict threshold as a validation check for previewing the file (but is good enough for now).

The best would be to implement this on the client side, but we're already in discussions about if/how we keep the current D3-based previewer (in case there's a better one out in the wild :) )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants