Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to download large legal hold files #44

Open
fmartingr opened this issue May 15, 2024 · 8 comments · Fixed by #50
Open

Unable to download large legal hold files #44

fmartingr opened this issue May 15, 2024 · 8 comments · Fixed by #50
Assignees
Milestone

Comments

@fmartingr
Copy link
Contributor

On my load tests environment I was unable to download large legal hold files with 500k+ posts.

I'm unsure if this is caused by the environment itself or by the plugin, since there was a proxy server in the middle of the connection and I already had to increase the proxy_timeout of the proxy server in order to download smaller files. This requires investigating locally (without proxy server) to test if there's a problem with the plugin, and if not, determine what proxy servers need in order for this to work.

@fmartingr fmartingr self-assigned this May 23, 2024
@fmartingr
Copy link
Contributor Author

Legal hold download is performed on the fly by creating a zip file and streaming it to the user. Since this sent to the user via HTTP, for large legal holds we could reach the server timeout set up into System Console > Web server > Write timeout and other timeouts present on reverse proxies in front of Mattermost.

@grundleborg
Copy link
Collaborator

If tweaking timeouts isn't enough to resolve this, we may need to add the ability to assemble the zip file in the background and allow the administrator to come back and download it later when it's done.

@fmartingr
Copy link
Contributor Author

If tweaking timeouts isn't enough to resolve this, we may need to add the ability to assemble the zip file in the background and allow the administrator to come back and download it later when it's done.

Yeah, I was thinking that it may be possible to assemble and upload it to S3 as well, using the server as a proxy between S3 and the user downloading the file.

@grundleborg
Copy link
Collaborator

Had a conversation with @wiggin77 about this just now. One possibility that came up was putting this particular API endpoint outside any timeout enforced in Mattermost Server itself.

@fmartingr
Copy link
Contributor Author

Had a conversation with @wiggin77 about this just now. One possibility that came up was putting this particular API endpoint outside any timeout enforced in Mattermost Server itself.

Like exposing a different http server for this plugin? The calls are bundled in the ServerHTTP method that gets into Mattermost's http.Server, so the solution here would be exposing a new server? That would mean binding into a different port. I'm not sure we can modify the global Read/Write timeouts for a server on a route basis, will investigate.

@fmartingr
Copy link
Contributor Author

Had a conversation with @wiggin77 about this just now. One possibility that came up was putting this particular API endpoint outside any timeout enforced in Mattermost Server itself.

Like exposing a different http server for this plugin? The calls are bundled in the ServerHTTP method that gets into Mattermost's http.Server, so the solution here would be exposing a new server? That would mean binding into a different port. I'm not sure we can modify the global Read/Write timeouts for a server on a route basis, will investigate.

Hey @mickmister, do you happen to know how could I achieve something like that? Does the Plugin API support anything like it?

@mickmister
Copy link
Contributor

@fmartingr Nothing directly supported by the plugin framework, but a plugin technically has the capability of hosting their own server/port on the same host as the MM server. It would depend on the hosting situation of the customer what may be the options there, but it should be possible to configure the plugin to just host its own server for this specific endpoint.

@fmartingr
Copy link
Contributor Author

@fmartingr Nothing directly supported by the plugin framework, but a plugin technically has the capability of hosting their own server/port on the same host as the MM server. It would depend on the hosting situation of the customer what may be the options there, but it should be possible to configure the plugin to just host its own server for this specific endpoint.

Yeah I don't think that's a viable option, it would involve more complexity in the plugin configuration by adding networking requirements, let's try another route.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants