-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How do you write a range request? #59
Comments
checkout But that will only explain how to read a raster (COG). You have other issues to solve:
You don't need to write the range-request, rasterio/GDAL to do that for you ref: |
Thanks for the links, I hadn't seen those before. I'm trying to understand this in the most basic form possible (the naip-cogeo-mosaic link looks interesting, but I'd prefer not to get into another file format such as the mosaic.json described there). After looking through those links, I'm still not clear on how I would proceed here. I used the code above to grab my list of urls, and saved to "ca_urls.txt". A subset of a few here:
I see several options for what should go in place of "https://" such as "s3://" or "/vsis3/", but don't know what the difference is. Now I'm still not sure what to do with this. I looked at gdalbuildvrt, but couldn't find a configuration that actually generated a vrt. What is the overlap with the vrt and the mosaic.json project? They sound similar, so if I could just use the gdal vrt format instead that would be preferrable to minimize extra libraries. Would I then need to feed the vrt file into rio_tiler Reader, where it is treated like a regular file and I can then follow the example you linked (https://cogeotiff.github.io/rio-tiler/advanced/feature/) at which point my file will be downloaded and saved? Once I have my vrt file, should a command like this work? |
because the naip bucket is requester pays, you'll need to use the This should fix the
MosaicJSON is kinda like a VRT but
Yes
Rio-tiler is not meant to save output file, you'll better served with gdalwarp command line as you mentioned. |
Thanks for your help, it feels so close! I really appreciate it. I now have my list of urls in "ca_32113_urls.txt":
And my command
But am still getting "Warning 1: Can't open . Skipping it" |
try with :
|
That one didn't work either. |
🤦 my bad it should be
GDAL uses |
Ah, I tried that one already as well, didn't work unfortunately. |
it worked for me 🤷♂️ |
Ah, that's a good start! Did you run the same command I had listed above, or do you have the environment variables set locally? |
I'm on Mac OS, I have couple env set but it shouldn't make any difference for accessing the files
|
Do you have additional AWS environment variables set? I saw you omitted AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, so I'm guessing they are set as environment variables. |
yes they are in my env by default so I omitted those |
I tried a few more things, and it seems something was corrupted with the my input file list, "ca_32114_urls.txt". I had created that one by copy and pasting the text in from prior command line output, then saving it via Notepad++. I created a new file, With the new, uncorrupted
Thank you so much for your help with this! |
I'm trying to take advantage of COG format to write a python script to download only the data inside of a state border for example "california.shp". The data I want to download is hosted on AWS as requester pays, so it's important that I only download exactly what I need.
I have gone several pages in on google, and despite all the pages saying how great COG is because of the range requests, I just can't find a single real example showing how to do this.
So, given a shapefile, and an aws data COG data source (for example, naip-imagery), how would you do this? It would be great to have example scripts on this website (or anywhere, really).
This is the code I have so far. (Along with some creative suggestions from ChatGPT that didn't work out, I'm assuming it has just as little information to feed into the model as I did when trying to figure it out...)
My metadata response from AWS looks like this:
I'm hoping someone here is familiar enough with the format to point me in the right direction, it's disappointing that something so simple isn't described anywhere.
I'm not sure where to find:
Thanks! I really appreciate any help.
The text was updated successfully, but these errors were encountered: