Pusher's memory usage is bad when started with lots of old data that needs uploading

[pboothe]

Pusher's memory use pattern turns out to be bad for systems with lots of old data. It creates one tarfile for every day/datatype combination with un-uploaded data. This means that using it on a directory with lots and lots of unuploaded days causes a memory spike as all those days are loaded into independent tarfiles and uploaded independently.

I think the right thing is probably for main to be:

for each datatype {
  in a goroutine {
    upload all existing data serially  // <-- THIS IS THE NEW PART
    start listening for new data
  }
}

That way, starting pusher with lots of old data won't lead to a memory spike - that data will just get uploaded serially.