Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quite slow when dealing with large sets #22

Open
cdown opened this issue Apr 10, 2017 · 21 comments
Open

Quite slow when dealing with large sets #22

cdown opened this issue Apr 10, 2017 · 21 comments
Labels
enhancement

Comments

@cdown
Copy link

cdown commented Apr 10, 2017

First of all, thanks for the great tool.

One thing that may be worth looking into is adding parallelism or some other kind of batching. Right now on a 100 mbit link, it takes a number of hours to download a ~10G set, and the link only becomes saturated in bursts. This seems to indicate that the work around downloading a photo using the API is a non-trivial amount of work that may benefit from being done in parallel.

I'm not sure if I'll have time to submit a pull request soon, so I'm submitting this ticket in lieu for now to see your thoughts on adding parallelism support to flickr-download. When I have more time I will get some more profiles and see if there are any low hanging fruit, as well.
@beaufour
Copy link
Owner

I'm happy that you like the tool!

I haven't thought about download speed at all to be honest, so I'm not that surprised. In theory, it shouldn't be that hard to add some parallelism to the downloads, but I'm using the flickr_api and I have never actually looked at what it does for downloading under the hood. I'll give it a peek one day.

@beaufour
Copy link
Owner

Seems like the Flickr API is not the snappiest. It's more than 200ms for most calls. And often it takes multiple seconds per call. That could certainly explain some of the slowness

@beaufour beaufour changed the title Quite slow when dealing with large sets on fast connections Quite slow when dealing with large sets Sep 26, 2022
@beaufour beaufour mentioned this issue Sep 26, 2022
Open
@beaufour
Copy link
Owner

One challenge here is that even though it looks like you can add extras arguments to the photosets.getPhotos API call to get all the information to download, the python-flickr-api library always calls photos.getInfo for each photo. I can't blame the library, as it just maps to the Flickr API.

So an optimization would be to add the following extras to the getPhotos API call and then use that to directly construct a Photo:original_format, o_dims, media, url_sq, url_t, url_s, url_m, url_o, url_l . That would save most of the API calls and really speed things up.

One could then also parallelize those downloads if one was really fancy :)

@eggplantedd
Copy link

eggplantedd commented Sep 26, 2022 •
edited
Loading

That sounds good. I think that would minimize the relevance/downtime of API errors.

I am of course available to test whenever 😃

@beaufour
Copy link
Owner

It's a bit of a major operation to tweak the API calls, which I've tracked in #64 . I likely won't get to that anytime soon. But in the current master, I've just added a --metadata_store flag. Combined with the --cache flag you can speed things up a lot on retries. The metadata store doesn't work retroactively on sets you've already downloaded I'm afraid, but all new sets can take advantage of this.

@eggplantedd
Copy link

I did a fresh run in a new folder with this new master and got to 1451 folders before I got a program-exiting 500 error (like before).

@beaufour
Copy link
Owner

This is not doing anything for the error 500, but if you use the new flags it should zip through the already downloaded files and restart actually downloading again immediately?

@eggplantedd
Copy link

eggplantedd commented Sep 27, 2022 •
edited
Loading

Will it? I must've mis-interpeted your comment thinking it wouldn't do that.

I used the arguments --metadata_store and --cache CACHE_FILE

Do I use those again when restarting? When I just did so, it started from the beginning skipping just as slow.

Should I have specified anything more for the cache file? I didn't see any file in the download directory, or in the flickr_download folder in %Appdata%.

@beaufour
Copy link
Owner

Are you sure you are using the latest version here from github? I just did a run with a fresh install below. As you can see the first run takes 9s and the second takes 0.198s. The last run is without caching and metadata enabled (ie the old way of skipping) and it takes 7.946s.

$ pip install https://github.com/beaufour/flickr-download/archive/refs/heads/master.zip
[pip output removed]
$ time flickr_download --cache cache.file --metadata --download 72157641626097033
INFO:root:Caching is enabled
INFO:root:Downloading Fasnet 2014
INFO:root:Saving: Fasnet 2014/Clown.jpg (https://www.flickr.com/photos/hhoch/13295370503/)
INFO:root:Saving: Fasnet 2014/Hopfennarr.jpg (https://www.flickr.com/photos/hhoch/13197784625/)
INFO:root:Saving: Fasnet 2014/Faselhannes.jpg (https://www.flickr.com/photos/hhoch/13164706264/)
INFO:root:Saving: Fasnet 2014/Schorrenweible.jpg (https://www.flickr.com/photos/hhoch/13011302384/)
INFO:root:Saving: Fasnet 2014/Kügele.jpg (https://www.flickr.com/photos/hhoch/12994078515/)
INFO:root:Saving: Fasnet 2014/Funkenmariechen.jpg (https://www.flickr.com/photos/hhoch/12974696174/)
INFO:root:Saving: Fasnet 2014/Rätsch.jpg (https://www.flickr.com/photos/hhoch/12934610845/)
INFO:root:Saving: Fasnet 2014/spanische Fliege _ Spanish Fly.jpg (https://www.flickr.com/photos/hhoch/12874357023/)
INFO:root:Saving: Fasnet 2014/Don Quijote.jpg (https://www.flickr.com/photos/hhoch/12853478335/)
INFO:root:Saving: Fasnet 2014/Afro.jpg (https://www.flickr.com/photos/hhoch/12831632914/)
INFO:root:Saving: Fasnet 2014/Queens.jpg (https://www.flickr.com/photos/hhoch/12822261443/)
INFO:root:Saving: Fasnet 2014/Dämonen _ demons.jpg (https://www.flickr.com/photos/hhoch/13325972784/)
INFO:root:Saving: Fasnet 2014/Keltenwächter vom Dickenwald.jpg (https://www.flickr.com/photos/hhoch/13359674385/)
INFO:root:Saving: Fasnet 2014/Die rote Spinne _ the red spider.jpg (https://www.flickr.com/photos/hhoch/13566149214/)
INFO:root:Saving: Fasnet 2014/Dämonen _ demons #02.jpg (https://www.flickr.com/photos/hhoch/13610444904/)
INFO:root:Saving: Fasnet 2014/Keltenwächter vom Dickenwald - green version.jpg (https://www.flickr.com/photos/hhoch/13642612284/)
INFO:root:Saving: Fasnet 2014/Federle.jpg (https://www.flickr.com/photos/hhoch/13741930325/)
flickr_download --cache cache.file --metadata --download 72157641626097033  1.09s user 0.18s system 13% cpu 9.407 total
$ time flickr_download --cache cache.file --metadata --download 72157641626097033
INFO:root:Caching is enabled
INFO:root:Downloading Fasnet 2014
INFO:root:Skipping download of already downloaded photo with ID: 13295370503
INFO:root:Skipping download of already downloaded photo with ID: 13197784625
INFO:root:Skipping download of already downloaded photo with ID: 13164706264
INFO:root:Skipping download of already downloaded photo with ID: 13011302384
INFO:root:Skipping download of already downloaded photo with ID: 12994078515
INFO:root:Skipping download of already downloaded photo with ID: 12974696174
INFO:root:Skipping download of already downloaded photo with ID: 12934610845
INFO:root:Skipping download of already downloaded photo with ID: 12874357023
INFO:root:Skipping download of already downloaded photo with ID: 12853478335
INFO:root:Skipping download of already downloaded photo with ID: 12831632914
INFO:root:Skipping download of already downloaded photo with ID: 12822261443
INFO:root:Skipping download of already downloaded photo with ID: 13325972784
INFO:root:Skipping download of already downloaded photo with ID: 13359674385
INFO:root:Skipping download of already downloaded photo with ID: 13566149214
INFO:root:Skipping download of already downloaded photo with ID: 13610444904
INFO:root:Skipping download of already downloaded photo with ID: 13642612284
INFO:root:Skipping download of already downloaded photo with ID: 13741930325
flickr_download --cache cache.file --metadata --download 72157641626097033  0.17s user 0.02s system 97% cpu 0.198 total
$ time flickr_download --download 72157641626097033
INFO:root:Downloading Fasnet 2014
INFO:root:Skipping Fasnet 2014/Clown.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Hopfennarr.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Faselhannes.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Schorrenweible.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Kügele.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Funkenmariechen.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Rätsch.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/spanische Fliege _ Spanish Fly.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Don Quijote.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Afro.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Queens.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Dämonen _ demons.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Keltenwächter vom Dickenwald.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Die rote Spinne _ the red spider.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Dämonen _ demons #02.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Keltenwächter vom Dickenwald - green version.jpg, as it exists already
INFO:root:Skipping Fasnet 2014/Federle.jpg, as it exists already
flickr_download --download 72157641626097033  0.79s user 0.10s system 11% cpu 7.946 total

@eggplantedd
Copy link

eggplantedd commented Oct 1, 2022 •
edited
Loading

So I did everything in your message above, including installing the master from pip, and it did indeed work.

I'm restarting the download process. I am not using the --download argument, but -u.

But I think there's an issue with its implementation. The cache file is only produced upon successful/normal exit of the program - it's not created as you're downloading.

That means in my case, with a program-exiting error in the middle of an incomplete download, I won't have a cache file I can use.

Evidence: I watched how the cache file was only generated with your example after the download had successfully completed. I then closed CMD in the middle of downloading your example photoset, and the cache file wasn't produced. I'm guessing the same will occur when I hit a 500 error. I will let you know when this happens.

@beaufour
Copy link
Owner

beaufour commented Oct 1, 2022 via email

@beaufour
Copy link
Owner

beaufour commented Oct 3, 2022

I just looked through the code and did some tests, and the metadata store should be written fine on errors. The API cache was not, which I've fixed in master now.

@eggplantedd
Copy link

eggplantedd commented Oct 4, 2022 •
edited
Loading

Thanks- I triple-checked I installed the right master and left the program on last night.

However, I didn't hit any errors for hours, so I had to manually exit the program to sleep. No cache file created- as expected, but could an argument be added to produce cache file upon manual exit? Would be final step to deal with downloads over multiple days.

So when I come back today, it has to use the slower initial checking process. Ok- but I hit a program-exiting error, and indeed a 74.4MB cache file is produced.

Yet when I restart the program with the same arguments, it doesn't use it. It just restarts the old process.

I am wondering if it produces cache it can't do anything with, or is programmed not to do anything with. The program may need a download to have occurred in the session cache, and so can't work in the 'blind' initial checking process.

My desire here is the cache from the initial checking process tells the program what files to zip through, so it can then get to the files it has to initially check.

My apologies for this back and forth, hoping these aren't too fundamental changes.

@eggplantedd
Copy link

eggplantedd commented Oct 4, 2022 •
edited
Loading

Am I doing something wrong? I got it to fail during the download stage, 500 error, it produced a cache file. I even close the cmd, then re-open it with same arguments, and it doesn't use it.

flickr_download --cache cache_file --metadata_store -q Original -u alookback.net

The file is cache_file. A chunk inside looks like

connection”Œ
Connection”Œ
keep-alive”†”Œ�vary”Œ�Vary”Œ�Accept-Encoding”†”Œ�date”Œ�Date”Œ�Tue, 04 Oct 2022 13:11:09 GMT”†”Œ�server”Œ�server”Œ�Apache/2.4.54 (Ubuntu)”†”Œ
set-cookie”Œ
set-cookie”Xè�  ccc=%7B%22needsConsent%22%3Atrue%2C%22managed%22%3A0%2C%22changed%22%3A0%2C%22info%22%3A%7B%22cookieBlock%22%3A%7B%22level%22%3A0%2C%22blockRan%22%3A0%7D%7D%7D; expires=Thu, 03-Nov-2022 13:11:09 GMT; Max-Age=2592000; path=/; domain=.flickr.com, ccc=%7B%22needsConsent%22%3Atrue%2C%22managed%22%3A0%2C%22changed%22%3A0%2C%22info%22%3A%7B%22cookieBlock%22%3A%7B%22level%22%3A0%2C%22blockRan%22%3A1%7D%7D%7D; expires=Thu, 03-Nov-2022 13:11:09 GMT; Max-Age=2592000; path=/; domain=.flickr.com”†”Œ�x-content-type-options”Œ�x-content-type-options”Œ�nosniff”†”Œ�x-frame-options”Œ�x-frame-options”Œ
SAMEORIGIN”†”Œ�content-encoding”Œ�Content-Encoding”Œ�gzip”†”Œ�x-cache”Œ�X-Cache”Œ�Miss from cloudfront”†”Œ�via”Œ�Via”Œ@1.1 524a19c44176623513fff0bfa6aba1f0.cloudfront.net (CloudFront)”†”Œx-amz-cf-pop”ŒX-Amz-Cf-Pop”Œ�MAN50-C2”†”Œ�x-amz-cf-id”Œ�X-Amz-Cf-Id”Œ8g3gtLycG9tIci8aBGbZObtRD2_c7nQxCN7PrfUYQk5kPwTGU56e6ww==”†”usbhGŒ%https://api.flickr.com/services/rest/”hI]”hKhLhMŒ�OK”hOhR)�”}”(hUhX)�”}”(h[ˆh\‰h]Nh^‰h_‰h`ˆha‰hbK hc‰hd‰hehhhi)hjNhkJê0<cubhl}”Œ�.flickr.com”}”hp}”Œ�ccc”ht)�”}”(hwK hxŒ�ccc”hzŒ›%7B%22needsConsent%22%3Atrue%2C%22managed%22%3A0%2C%22changed%22%3A0%2C%22info%22%3A%7B%22cookieBlock%22%3A%7B%22level%22%3A0%2C%22blockRan%22%3A1%7D%7D%7D”h|Nh}‰h~Œ�.flickr.com”h€ˆh�ˆh‚hphƒˆh„‰h…Jê½cch†‰h‡NhˆNh‰‰hŠ}”ubssshkJê0<cubhŒh�K K JÈ�� ‡”R”h’h”)�”}”(h—Œ�POST”hGj�G� hh�)�”}”h�h�)R”(Œ
user-agent”h�Œ�python-requests/2.27.1”†”Œ�accept-encoding”h¡h¢†”Œ�accept”h¥h¦†”Œ
connection”h©hª†”Œ�content-length”h­Œ�120”†”Œcontent-type”h±h²†”usbhlhR)�”}”(hUhX)�”}”(h[ˆh\‰h]Nh^‰h_‰h`ˆha‰hbK hc‰hd‰hehhhi)hjNhkJé0<cubhl}”hkJé0<cubh¹Œxmethod=flickr.photos.getSizes&photo_id=51144883146&api_key=XXXXXX&format=json&nojsoncallback=1”h»}”h½]”sh¿NububŒwmethod=flickr.photos.getInfo&photo_id=51145106088&api_key=XXXXXXX&format=json&nojsoncallback=1”h�)�”}”(h	BQ�  {"photo":{"id":"51145106088","secret":"55aa62ddee","server":"65535","farm":66,"dateuploaded":"1619637633","isfavorite":0,"license":"0","safety_level":"0","rotation":0,"originalsecret":"78924d57be","originalformat":"jpg","owner":{"nsid":"38552878@N02","username":"alookback.net","realname":"","location":"","iconserver":"0","iconfarm":0,"path_alias":null,"gift":{"gift_eligible":true,"eligible_durations":["year","month","week"],"new_flow":true}},"title":{"_content":"Phoenix 2 sm 2"},"description":{"_content":""},"visibility":{"ispublic":1,"isfriend":0,"isfamily":0},"dates":{"posted":"1619637633","taken":"2021-04-28 12:17:19","takengranularity":0,"takenunknown":"1","lastupdate":"1619637661"},"views":"102","editability":{"cancomment":0,"canaddmeta":0},"publiceditability":{"cancomment":1,"canaddmeta":0},"usage":{"candownload":1,"canblog":0,"canprint":0,"canshare":1},"comments":{"_content":"0"},"notes":{"note":[]},"people":{"haspeople":0},"tags":{"tag":[]},"urls":{"url":[{"type":"photopage","_content":"https:\/\/www.flickr.com\/photos\/38552878@N02\/51145106088\/"}]},"media":"photo"},"stat":"ok"}”h�KÈhh�)�”}”h�h�)R”(Œcontent-type”ŒContent-Type”Œ�application/json”†”Œ�transfer-encoding”Œ�Transfer-Encoding”Œ�chunked”†”Œ

Guessing that

connection”Œ
Connection”Œ

seperates out each photo to download.

@beaufour
Copy link
Owner

beaufour commented Oct 4, 2022

I don't quite understand what is going on here. Could you try to download the latest master and then run

> flickr_download --cache cache.file --metadata --download 72157641626097033 --verbose

Twice and share the output here?

@eggplantedd
Copy link

eggplantedd commented Oct 5, 2022 •
edited
Loading

I tested that as working before, including using the cache file. I couldn't encounter a program-exiting error with that singular photoset so I'm not sure it can apply to my use case ☹️

So I ran --verbose on the photo list + cache I described in my last post, and then on the one you just described. This should allow comparison between successful/unsuccessful use of cache.

--verbose for 72157641626097033:

G:\alookback2>flickr_download --cache cache.file --metadata --do
wnload 72157641626097033 --verbose
INFO:root:Caching is enabled
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getInfo' with arguments: {'photoset_id': '72157641626097033
'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 738
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getInfo&photoset_id=72157641626097033&api_key
=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getPhotos' with arguments: {'photoset_id': '721576416260970
33'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getPhotos&photoset_id=72157641626097033&api_k
ey=XXXXXX&format=json&nojsoncallback=1
INFO:root:Downloading Fasnet 2014
INFO:root:Skipping download of already downloaded photo with ID: 13295370503
INFO:root:Skipping download of already downloaded photo with ID: 13197784625
INFO:root:Skipping download of already downloaded photo with ID: 13164706264
INFO:root:Skipping download of already downloaded photo with ID: 13011302384
INFO:root:Skipping download of already downloaded photo with ID: 12994078515
INFO:root:Skipping download of already downloaded photo with ID: 12974696174
INFO:root:Skipping download of already downloaded photo with ID: 12934610845
INFO:root:Skipping download of already downloaded photo with ID: 12874357023
INFO:root:Skipping download of already downloaded photo with ID: 12853478335
INFO:root:Skipping download of already downloaded photo with ID: 12831632914
INFO:root:Skipping download of already downloaded photo with ID: 12822261443
INFO:root:Skipping download of already downloaded photo with ID: 13325972784
INFO:root:Skipping download of already downloaded photo with ID: 13359674385
INFO:root:Skipping download of already downloaded photo with ID: 13566149214
INFO:root:Skipping download of already downloaded photo with ID: 13610444904
INFO:root:Skipping download of already downloaded photo with ID: 13642612284
INFO:root:Skipping download of already downloaded photo with ID: 13741930325
DEBUG:root:Cache saved to cache.file

G:\alookback2>

--verbose for -u alookback.net (first few images):

G:\alookback2>flickr_download --cache cache_file --metadata_stor
e -q Original -u alookback.net --verbose
DEBUG:root:Cache loaded from cache_file
INFO:root:Caching is enabled
DEBUG:flickr_api.reflection:Calling method 'flickr.people.findByUsername' with arguments: {'username': 'alookback.net',
'auth_handler': None}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 104
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.people.findByUsername&username=alookback.net&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getList' with arguments: {'user_id': '38552878@N02'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getList&user_id=38552878%40N02&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getInfo' with arguments: {'photoset_id': '72157719309850907'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 448
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getInfo&photoset_id=72157719309850907&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getPhotos' with arguments: {'photoset_id': '72157719309850907'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getPhotos&photoset_id=72157719309850907&api_key=XXXXXX&format=json&nojsoncallback=1
INFO:root:Downloading Dealer Ads 0671
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '51212763705'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=51212763705&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '51212763705'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=51212763705&api_key=XXXXXX&format=json&nojsoncallback=1
INFO:root:Skipping Dealer Ads 0671\Calumet 1 sm 2 (Original).jpg, as it exists already
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '51212763510'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=51212763510&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '51212763510'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=51212763510&api_key=XXXXXX&format=json&nojsoncallback=1
INFO:root:Skipping Dealer Ads 0671\Cambridge 1 sm 2 (Original).jpg, as it exists already
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '51212763465'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=51212763465&api_key=XXXXXX&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '51212763465'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=51212763465&api_key=XXXXXX&format=json&nojsoncallback=1
INFO:root:Skipping Dealer Ads 0671\Cambridge 2 sm 2 (Original).jpg, as it exists already

It doesn't seem to start the using the cache process.

Here is the cache_file of alookback.net with my API key removed:

https://drive.google.com/file/d/1q9MXZRWgWLn6E3cAeH3PklBconuJaClO/view?usp=sharing

I have no trouble finding photo_id 51212763705, 51212763510 etc inside- so no idea why the program won't recognise it.

pip install log:

G:\alookback2>pip show flickr_download
Name: flickr-download
Version: 0.3.1
Summary: Download photos from Flickr
Home-page: https://github.com/beaufour/flickr-download
Author: Allan Beaufour
Author-email: [email protected]
License: Apache-2.0
Location: appdata\local\programs\python\python310\lib\site-packages
Requires: flickr-api, pathvalidate, python-dateutil, PyYAML
Required-by:

G:\alookback2>pip install https://github.com/beaufour/flickr-download/archive/refs/heads/master.zip
Collecting https://github.com/beaufour/flickr-download/archive/refs/heads/master.zip
  Downloading https://github.com/beaufour/flickr-download/archive/refs/heads/master.zip
     - 31.9 kB ? 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: PyYAML<6.0,>=5.3 in appdata\local\programs\python\python310\li
b\site-packages (from flickr-download==0.3.1) (5.4.1)
Requirement already satisfied: flickr-api<0.8.0,>=0.7.4 in appdata\local\programs\python\pyth
on310\lib\site-packages (from flickr-download==0.3.1) (0.7.4)
Requirement already satisfied: pathvalidate<3.0.0,>=2.5.2 in appdata\local\programs\python\py
thon310\lib\site-packages (from flickr-download==0.3.1) (2.5.2)
Requirement already satisfied: python-dateutil==2.8.1 in appdata\local\programs\python\python
310\lib\site-packages\python_dateutil-2.8.1-py3.10.egg (from flickr-download==0.3.1) (2.8.1)
Requirement already satisfied: six>=1.5 in appdata\local\programs\python\python310\lib\site-p
ackages (from python-dateutil==2.8.1->flickr-download==0.3.1) (1.16.0)
Requirement already satisfied: oauth2 in appdata\local\programs\python\python310\lib\site-pac
kages\oauth2-1.9.0.post1-py3.10.egg (from flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (1.9.0.post1)
Requirement already satisfied: requests in appdata\local\programs\python\python310\lib\site-p
ackages (from flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (2.27.1)
Requirement already satisfied: httplib2 in appdata\local\programs\python\python310\lib\site-p
ackages\httplib2-0.20.4-py3.10.egg (from oauth2->flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (0.20.4)
Requirement already satisfied: certifi>=2017.4.17 in appdata\local\programs\python\python310\
lib\site-packages (from requests->flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in appdata\local\programs\python\python310\lib\si
te-packages (from requests->flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in appdata\local\programs\python\pyt
hon310\lib\site-packages (from requests->flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (2.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in appdata\local\programs\python\python3
10\lib\site-packages (from requests->flickr-api<0.8.0,>=0.7.4->flickr-download==0.3.1) (1.26.8)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in appdata
\local\programs\python\python310\lib\site-packages (from httplib2->oauth2->flickr-api<0.8.0,>=0.7.4->flickr-download==0.
3.1) (3.0.7)

G:\alookback2>

@beaufour
Copy link
Owner

beaufour commented Oct 5, 2022

You are not running the latest version from master (from yesterday), as that would have printed the full path to the cache file. So please update with the latest version from master. You can either copy the flick_download.py file in to the correct location, or pip install directly from github (pip install https://github.com/beaufour/flickr-download/archive/refs/heads/master.zip).

If you run that twice with the arguments I have above I can see if it successfully uses the cache file at all. Then we can tackle the API errors as a second step if needed.

@eggplantedd
Copy link

eggplantedd commented Oct 5, 2022 •
edited
Loading

Edit: Yes, I see in my previous message it must've been the old version, because it didn't show the cache save location.

Here is what you asked for. I deleted the previously downloaded files/cache to be sure. Also deleted the API key after use. As far as I can see it is printing the full path to the cache file - G:\alookback2\cache.file

G:\alookback2>flickr_download --cache cache.file --metadata --do
wnload 72157641626097033 --verbose
INFO:root:Caching is enabled
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getInfo' with arguments: {'photoset_id': '72157641626097033
'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 738
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getInfo&photoset_id=72157641626097033&api_key
=f1114b4af84fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getPhotos' with arguments: {'photoset_id': '721576416260970
33'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photosets.getPhotos&photoset_id=72157641626097033&api_k
ey=f1114b4af84fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Downloading Fasnet 2014
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13295370503'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13295370503&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13295370503'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13295370503&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Clown.jpg (https://www.flickr.com/photos/hhoch/13295370503/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13197784625'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13197784625&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13197784625'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13197784625&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Hopfennarr.jpg (https://www.flickr.com/photos/hhoch/13197784625/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13164706264'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13164706264&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13164706264'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13164706264&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Faselhannes.jpg (https://www.flickr.com/photos/hhoch/13164706264/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13011302384'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13011302384&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13011302384'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13011302384&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Schorrenweible.jpg (https://www.flickr.com/photos/hhoch/13011302384/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12994078515'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12994078515&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12994078515'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12994078515&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Kügele.jpg (https://www.flickr.com/photos/hhoch/12994078515/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12974696174'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12974696174&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12974696174'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12974696174&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Funkenmariechen.jpg (https://www.flickr.com/photos/hhoch/12974696174/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12934610845'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12934610845&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12934610845'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12934610845&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Rätsch.jpg (https://www.flickr.com/photos/hhoch/12934610845/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12874357023'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12874357023&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12874357023'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12874357023&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\spanische Fliege _ Spanish Fly.jpg (https://www.flickr.com/photos/hhoch/12874357023/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12853478335'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12853478335&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12853478335'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12853478335&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Don Quijote.jpg (https://www.flickr.com/photos/hhoch/12853478335/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12831632914'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12831632914&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12831632914'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12831632914&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Afro.jpg (https://www.flickr.com/photos/hhoch/12831632914/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '12822261443'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=12822261443&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '12822261443'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=12822261443&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Queens.jpg (https://www.flickr.com/photos/hhoch/12822261443/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13325972784'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13325972784&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13325972784'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13325972784&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Dämonen _ demons.jpg (https://www.flickr.com/photos/hhoch/13325972784/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13359674385'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13359674385&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13359674385'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13359674385&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Keltenwächter vom Dickenwald.jpg (https://www.flickr.com/photos/hhoch/13359674385/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13566149214'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13566149214&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13566149214'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13566149214&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Die rote Spinne _ the red spider.jpg (https://www.flickr.com/photos/hhoch/13566149214/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13610444904'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13610444904&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13610444904'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13610444904&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Dämonen _ demons #02.jpg (https://www.flickr.com/photos/hhoch/13610444904/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13642612284'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13642612284&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13642612284'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13642612284&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Keltenwächter vom Dickenwald - green version.jpg (https://www.flickr.com/photos/hhoch/1364
2612284/)
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getInfo' with arguments: {'photo_id': '13741930325'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getInfo&photo_id=13741930325&api_key=f1114b4af84
fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photos.getSizes' with arguments: {'photo_id': '13741930325'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.flickr.com:443
DEBUG:urllib3.connectionpool:https://api.flickr.com:443 "POST /services/rest/ HTTP/1.1" 200 None
DEBUG:flickr_api.method_call:NO HIT for cache key: method=flickr.photos.getSizes&photo_id=13741930325&api_key=f1114b4af8
4fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Saving: Fasnet 2014\Federle.jpg (https://www.flickr.com/photos/hhoch/13741930325/)
DEBUG:root:Cache saved to G:\alookback2\cache.file

G:\alookback2>flickr_download -k f1114b4af84fa7874f7a6263cd7f5a32 -s b21632e0a480bb7f --cache cache.file --metadata --do
wnload 72157641626097033 --verbose
DEBUG:root:Cache loaded from: G:\alookback2\cache.file
INFO:root:Caching is enabled
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getInfo' with arguments: {'photoset_id': '72157641626097033
'}
DEBUG:flickr_api.method_call:   HIT for cache key: method=flickr.photosets.getInfo&photoset_id=72157641626097033&api_key
=f1114b4af84fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
DEBUG:flickr_api.reflection:Calling method 'flickr.photosets.getPhotos' with arguments: {'photoset_id': '721576416260970
33'}
DEBUG:flickr_api.method_call:   HIT for cache key: method=flickr.photosets.getPhotos&photoset_id=72157641626097033&api_k
ey=f1114b4af84fa7874f7a6263cd7f5a32&format=json&nojsoncallback=1
INFO:root:Downloading Fasnet 2014
INFO:root:Skipping download of already downloaded photo with ID: 13295370503
INFO:root:Skipping download of already downloaded photo with ID: 13197784625
INFO:root:Skipping download of already downloaded photo with ID: 13164706264
INFO:root:Skipping download of already downloaded photo with ID: 13011302384
INFO:root:Skipping download of already downloaded photo with ID: 12994078515
INFO:root:Skipping download of already downloaded photo with ID: 12974696174
INFO:root:Skipping download of already downloaded photo with ID: 12934610845
INFO:root:Skipping download of already downloaded photo with ID: 12874357023
INFO:root:Skipping download of already downloaded photo with ID: 12853478335
INFO:root:Skipping download of already downloaded photo with ID: 12831632914
INFO:root:Skipping download of already downloaded photo with ID: 12822261443
INFO:root:Skipping download of already downloaded photo with ID: 13325972784
INFO:root:Skipping download of already downloaded photo with ID: 13359674385
INFO:root:Skipping download of already downloaded photo with ID: 13566149214
INFO:root:Skipping download of already downloaded photo with ID: 13610444904
INFO:root:Skipping download of already downloaded photo with ID: 13642612284
INFO:root:Skipping download of already downloaded photo with ID: 13741930325
DEBUG:root:Cache saved to G:\alookback2\cache.file

G:\alookback2>

@beaufour
Copy link
Owner

beaufour commented Oct 6, 2022

Thanks! That seems to run exactly like advertised, right? It both has cache hits and also quickly skips the downloaded files. I also just tested it myself with a private photo set and it works fine.

I've also artificially triggered API errors, and it works fine. I've also killed the script and it works.

So I'm really puzzled to why it's not working for you? Were you just running an old version. Is it working now?

@eggplantedd
Copy link

Sorry- I had a catastrophic hard drive failure that was housing the files, so had to be dealt with.

I just restarted the download process with the same arguments, got a 500 error and a cache file produced, when I restarted it- success! It used the cache file to quickly/instanteously resume.

However...

Once I hit a second 500 error, the cache file did indeed write again (larger this time), but running program again with same arguments- it didn't use the cache file, and restarts the slow, old download process.

Is this a situation where the program isn't designed to handle beyond one error being raised? I'm not sure.

@beaufour
Copy link
Owner

Ugh, sorry to hear about your hard drive! I hope you got it back to working.

Ah, the cache times out after 3600 seconds.... which means it's gone when you restart after a long download. But the metadata store should help (ie --metadata) . It shouldn't need to call the API for each already downloaded photo. It should do one API call to get the photo set information and then skip right to the first new file. I don't fully understand why that's not working for you. Are you 100% sure you are running the latest version? You can just use the released version (through pip install)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@beaufour @cdown @eggplantedd