The serialized value is too large

[beljand]

When trying to persist DB, I often get an error saying:

DOMException: The serialised value is too large (size=133577246 bytes, max=133169152 bytes)

I've forked ForerunnerDB and updated to latest LocalForage (1.5.0), but the issue persist. The DB size on disk is 22-25MB. The collection that fails to persist has 9000-10000 items.

[beljand]

In fact JSON is larger than expected, probably larger than ~130MB, IDB size on disk is compressed.
Current assumption is that this is a Browser limitation.

Is there any other solution than splicing the collection into many collections (Collection Group), let's say 500 items per collection and then persist only one of the collections in the Group?

[tonyjoanes]

I'm just dealing with the same issue using localForage with Chrome. I think I will try and compress the data with something like lz-string and see if I can get a bit further

https://github.com/pieroxy/lz-string

[urossmerdelj]

Actually FDB already supports compression out of the box.
https://github.com/Irrelon/ForerunnerDB#data-compression-and-encryption

We ended up implementing a solution that splits data into multiple collections and grouping them in a collectionGroup (so we can query them all). Issue with grouping is that when inserting into a group the entity is inserted into all collections, so we use it for read only purposes and insert into a new collection when the last one reaches a certain size/count.

[Irrelon]

@Ulkuurz That is a very interesting and novel way to solve the problem! Kudos!

[tonyjoanes]

Ah yes that's what I've had to do as well, splitting the data up has helped a lot!

[Irrelon]

I'm rewriting ForerunnerDB from scratch in ES6 with a view to removing all of these annoyances. When I first coded FDB it was as an in-memory store for single page applications with absolutely no persistent storage layer at all so it wasn't designed with that in mind - I figured people would cache the HTTP calls that generated the data in the first place and replay them if they needed to. The rewrite will be persistence-first coding :)

[urossmerdelj]

That's great news, looking forward to that!

[Fadi288]

so we are no in 2020 and still my max size per objectstore is 133169152 ?? :(
still tehre is no workaround to solve this without having to split the wole image i wanna save as an array into many objectstores?

[Irrelon]

@Fadi288 unless browsers increase that limitation then no.