[Bug] confusing limit on library component count

[bradenmacdonald]

1. Improve error message when creating more than 1,000 components in a course. Currently it gives a generic error. It should say specifically that creating a component failed because the library has reached the maximum number of components supported.
2. Raise the limit in the platform from 1,000 to 100,000.

[bradenmacdonald]

@jmakowski1123 @ormsbee Please review this ticket.

[ormsbee]

@bradenmacdonald: How far do you think we could safely crank the limit on the search side of things? I'm tempted to just turn it up to 100K and see what happens. 😛

[bradenmacdonald]

@ormsbee I can't say from experience but I would hope it works fine even at 100K.

[pomegranited]

@jmakowski1123 @lizc577 @sdaitzman @marcotuts This is ready for AC testing on the sandbox

[ChrisChV]

@jmakowski1123 @lizc577 @sdaitzman @marcotuts Friendly reminder that this is ready for AC testing on the sandbox

[jmakowski1123]

Not sure how to test this? Is there a file with 100k components I could try to import?

[pomegranited]

Oh my apologies @jmakowski1123 , this is a hard one to test.

@bradenmacdonald is it worth taking a small amount of budget from the new project to create a huge library to test this issue, (and see how Meilisearch performs with the new limit)?

[bradenmacdonald]

@pomegranited I believe various people have already done that in the process of working on these issues - it should be relatively simple to paste a script into the console that mass-creates library blocks, and run that on the sandbox. In any case I agree it's a good idea to have a large library for testing on the sandbox.

[pomegranited]

I've got a script running to populate lib:OpenCraft:NL with 100,000 components -- will let you know when it's done.

[pomegranited]

@bradenmacdonald I created 100k components under lib:OpenCraft:NL, but Meilisearch was only showing 99,999. So I tried re-running reindex_studio to fix it, but it ran for 6h and didn't finish indexing that library.

This is going to impact re-deployments of tagging-preview :(

[bradenmacdonald]

@pomegranited I believe the "total number of hits" is only an estimate so I'm actually surprised that it's only off by one. There may be no issue.

Does running reindex_studio in the new incremental mode help?

[bradenmacdonald]

Also, for future times when we need a lot of components, please put a number into the title and/or body of the component so that we can tell them apart, e.g. "Text 1", "Text 2", "Text 89,234", etc. Makes it much easier to test and ensure search/pagination is working.

[pomegranited]

@bradenmacdonald

Also, for future times when we need a lot of components, please put a number into the title and/or body of the component so that we can tell them apart, e.g. "Text 1", "Text 2", "Text 89,234", etc

Ah yeah, that would be better.. I ran a script to update them (by creating a new ComponentVersion), and will re-index to get these changes into Meilisearch.

Does running reindex_studio in the new incremental mode help?

Oo hopefully.. trying that now.

[pomegranited]

Does running reindex_studio in the new incremental mode help?

@bradenmacdonald It did -- it still took a long time to run, but it completed. And now we've got an accurate count of blocks on lib:OpenCraft:NL.

I ran a script to update them (by creating a new ComponentVersion), and will re-index to get these changes into Meilisearch.

I stuffed this up (set the ComponentVersion title, but not the OLX display_name) and so am re-running that script now, so editing on the tagging-preview server might be a little sluggish.

But! This issue is ready for AC testing (CC @jmakowski1123 @lizc577 @sdaitzman @marcotuts):

1. Visit lib:OpenCraft:NL, which contains 100k components.
2. Try to add a new block to this library.
   Block creation will fail, and you should see this text in the toast:

   There was an error creating the content: Library cannot have more than 100000 Components