Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load testing!!! #23

Open
2 tasks done
jywarren opened this issue May 2, 2019 · 48 comments
Open
2 tasks done

Load testing!!! #23

jywarren opened this issue May 2, 2019 · 48 comments

Comments

@jywarren
Copy link
Member

jywarren commented May 2, 2019 •
edited
Loading

We'll find a bigger one too!!!
@jywarren jywarren mentioned this issue May 2, 2019
Closed
@jywarren
Copy link
Member Author

jywarren commented May 2, 2019 •
edited
Loading

https://mapknitter.org/profile/eustatic and https://mapknitter.org/profile/cfastie probably have some GIANT ones.

image

@icarito
Copy link
Member

icarito commented May 4, 2019

Please point export.mapknitter.org to this IP: 104.196.68.120
It should work with large images now!
🤞

@icarito
Copy link
Member

icarito commented May 4, 2019

imagen
Cluster is churning away!
🕶️

@icarito
Copy link
Member

icarito commented May 4, 2019

🚂

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Hmm, maybe I did it wrong? This site can’t be reached export.mapknitter.org took too long to respond.

I'll try directly at 104.196.68.120

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Hmm, the log isn't showing any requests reaching it... strange.

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Been trying https://104.196.68.120/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30 and https://export.mapknitter.org/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

aha!!! http. duh.

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

http://export.mapknitter.org/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30 works! Great.

Hmm. Despite my changes in #27 it's still only displaying a string, not a JSON object with the location of the JPG file: http://export.mapknitter.org/id/1556986755/status.json

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

OK, outputting geotiff and zip, and echoing more of the status and paths to console: #28

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

http://export.mapknitter.org/id/1556986755/status.json seems still stuck on generating jpg; maybe we'll learn more from the logs after #28

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019 •
edited
Loading

Strange, i'm not seeing any of the kinds of log output we had seen previously, even last night, like:


2019-05-04T04:40:20.413119293Z- [-33400560.850078963, 11671074.30096948]
2019-05-04T04:40:20.413170584Zwarping 11 of 19
2019-05-04T04:40:20.573044610Zconvert -contrast-stretch 0 public/warps/193358-working/w193358-a_step_11_IMG_5313.JPG -crop 4287x4287+0+0! -flatten -distort Perspective '0,0 0,383 4287,0 99,0 4287,2821 314,122 0,2821 237,506' -flatten -crop 506x506+0+0! +repage public/warps/4621/w193358.png

Wonder why? I'm looking in the logs for Cloud Build... maybe that's no longer the right place...

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

#28 merged, deploying revision once it's ready...

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Aha, i bet i'm in the wrong cloud system. Kubernetes maybe?

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Looks like it. I'll try deploying it again.

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

OK, i'm a bit out of my depth now. But the latest main branch of this repo is ready to go!

@icarito
Copy link
Member

icarito commented May 4, 2019

I'm deploying from kubernetes

@icarito
Copy link
Member

icarito commented May 4, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

I wonder if it's this line ... Is file.open the right way to upload the image?

mapknitter-exporter-sinatra/app.rb

Line 172 in bdfb816

:body => File.open(path),

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Maybe File.open().read instead? Hmm.

@jywarren
Copy link
Member Author

jywarren commented May 4, 2019

Hmm, no, this says File.open is correct: http://fog.io/storage/

@icarito
Copy link
Member

icarito commented May 4, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented May 5, 2019

#29 once merged is ready to be deployed too!

@icarito
Copy link
Member

icarito commented May 5, 2019

I think we are running into fog/fog-google#416
I'll try their workaround!

@jywarren
Copy link
Member Author

jywarren commented May 5, 2019 •
edited
Loading

🎉

This big map fully exported (cropping bug tracking here: #32): https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json

~30 images of ~2mb each!

URL was this but it now says:

{"error":{"errors":[{"domain":"global","reason":"lockedDomainExpired","message":"Locked Domain Expired"}],"code":401,"message":"Locked Domain Expired"}}

so, i think these links aren't permanent, which is good to know.

@jywarren
Copy link
Member Author

jywarren commented May 8, 2019 •
edited
Loading

Lemon fair burn has crop issues too but seems OK otherwise, hard to tell:

image

image

And lincoln peak vineyard also completed, with cropping issues and some missing images:

image

image

However, it's GREAT to know that it can crank through 48 images of ~3mb each!!!

@jywarren jywarren mentioned this issue May 8, 2019
Closed
@jywarren jywarren mentioned this issue May 21, 2019
Open
@jywarren
Copy link
Member Author

Noting for optimization, from https://gis.stackexchange.com/a/44739:

While I don't know why GDAL provides this overlap in functionality, be sure to set the cache for gdalwarp to make it really fast:

# assuming 3G of cache here:
gdalwarp --config GDAL_CACHEMAX 3000 -wm 3000 $(list_of_tiffs) merged.tiff

@icarito
Copy link
Member

icarito commented May 23, 2019 via email

@jywarren
Copy link
Member Author

jywarren commented May 23, 2019 via email

@jywarren jywarren mentioned this issue May 23, 2019
Closed
@jywarren
Copy link
Member Author

image

image

https://mapknitter.org/maps/otter-view-park-west -- @icarito this looks pretty darn good!

I see a couple edges which are not fuzzy, and so that means they were cut off partway through their composite processes. I'm wondering how it happened, but it could be either a crash partway through, OR it could be that their new bounding boxes are not properly calculated and they're just cropped wrong. I'll have to check the intermediate images to be sure. I'm closing the "missing images" and "cropping" issues and moving back here for more load testing.

@jywarren
Copy link
Member Author

#39

@jywarren
Copy link
Member Author

1558733443_1558733443

@jywarren
Copy link
Member Author

Now testing https://mapknitter.org/maps/lincoln-peak-vineyard -- http://export.mapknitter.org/export?url=https://mapknitter.org/maps/lincoln-peak-vineyard/warpables.json&scale=30

http://export.mapknitter.org/id/1558800531/status.json

@jywarren
Copy link
Member Author

It's at warping 19 of 48 now.

@jywarren
Copy link
Member Author

Oh no, i started a new one. Maybe we should invent a way to cancel exports...

@jywarren
Copy link
Member Author

And now, i try http://export.mapknitter.org/working/1558800531/w132014.tif (the 17th or 18th image) and get:

This site can’t be reached The webpage at http://export.mapknitter.org/working/1558800531/w132014.tif might be temporarily down or it may have moved permanently to a new web address.

However a few moments later it loads and i download the tiff.

@jywarren
Copy link
Member Author

Looks like it might have been cut off:
image

@jywarren
Copy link
Member Author

This one was also slightly cut off:
image

@jywarren
Copy link
Member Author

It looks like at a stage earlier than the distortion and blending/cropping -- so in gdalwarp or gdal_translate, not in imagemagick convert. It's worth pulling logs for this.

@jywarren
Copy link
Member Author

OK, so, having tracked this, a fix should resolve #39

This was referenced May 27, 2019
Open
Open
@jywarren
Copy link
Member Author

jywarren commented May 28, 2019 •
edited
Loading

Just noting that this export: https://mapknitter-exports-warps.storage.googleapis.com/1559057443/status.json

Based on https://mapknitter.org/maps/usfws-restoration-bayou-bonfouca-southeast-survey (9 images)

Just output this at 9cm/px:

https://mapknitter-exports-warps.storage.googleapis.com/1559057443/1559057443.jpg

It's 8408x14912 pixels!!!! 🙌 in approx. 10 minutes.

@jywarren
Copy link
Member Author

@tech4GT Take a look at this last export we just generated! 8408x14912 pixels!!!

@jywarren
Copy link
Member Author

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1559057443/status.json","status":"complete","tms":"public/tms/1559057443/","geotiff":"public/warps/1559057443/1559057443.tif","zip":"public/tms/1559057443.zip","jpg":"public/warps/1559057443/1559057443.jpg","export_id":1559057443,"user_id":null,"size":"501.64MB","width":"8408","height":"14912","start_time":null,"run_time":null,"gem_version":"1.0.7","cm_per_pixel":9.0}

@tech4GT
Copy link
Member

tech4GT commented May 28, 2019

@jywarren Can you share the details on scaling with the applied scale also? I mean right now I am just generating a canvas by adding all the individual widths and heights, is there a better way to do it?

@jywarren
Copy link
Member Author

jywarren commented May 28, 2019 via email

@jywarren jywarren mentioned this issue Jun 10, 2019
Open
5 tasks
@jywarren
Copy link
Member Author

We'll soon be able to compare to publiclab/image-sequencer-app#19 -- the pure JavaScript implementation!

@jywarren jywarren mentioned this issue Jun 24, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jywarren @icarito @tech4GT