Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ability to specify a data-url response (or just a JPG) instead of HTML wrapping a data-url #36

Open
jywarren opened this issue Dec 6, 2019 · 9 comments
Open

Ability to specify a data-url response (or just a JPG) instead of HTML wrapping a data-url #36

jywarren opened this issue Dec 6, 2019 · 9 comments
Labels
help wanted Extra attention is needed

Comments

@jywarren
Copy link
Member

jywarren commented Dec 6, 2019

We should make a parameter called response= which can be set to response=html or response=dataurl maybe?
@jywarren jywarren mentioned this issue Dec 6, 2019
Open
5 tasks
@jywarren
Copy link
Member Author

jywarren commented Dec 6, 2019

image-sequencer-app/src/api/v2/index.js

Lines 71 to 79 in dca8158

app.get("/status", (req, res) => {
mapknitterBucket.getFiles().then((files) => {
let index = files[0].map(el => el.metadata.name).indexOf(`export${req.query.pid}.html`);
if (index != -1)
res.redirect(files[0][index].metadata.mediaLink);
else
res.send("Still working on it");
});
});

@jywarren
Copy link
Member Author

jywarren commented Dec 6, 2019

Actually we should just redirect to a JSON status file that's actually stored in Google Cloud, which we are periodically updating as the export progresses.

@jywarren jywarren added the help wanted Extra attention is needed label Dec 6, 2019
@jywarren
Copy link
Member Author

jywarren commented Dec 6, 2019

See how here, we create a JSON file with the status, and repeatedly save it to Google Cloud storage as the process continues. And we report the location of that file back to the user immediately, so they can fetch it as often as they like without running any process on the exporter itself:

https://github.com/publiclab/mapknitter-exporter-sinatra/blob/main/app.rb#L119-L187

@tech4GT
Copy link
Member

tech4GT commented Dec 21, 2019 •
edited
Loading

Okay! This makes sense! So we just make and store a json file in the google cloud and update it periodically! Hmm, I wonder if this file could be stored locally? Since this file can be deleted after the export completes I'm not entirely sure if keeping it in the cloud would be a good idea, don't you think?
cc @jywarren

@jywarren jywarren mentioned this issue Dec 24, 2019
Merged
@jywarren
Copy link
Member Author

jywarren commented Jan 2, 2020 via email

@tech4GT
Copy link
Member

tech4GT commented Jan 6, 2020 •
edited
Loading

We actually want a permanent record of the status -- that's how the Ruby exporter works. Thank you!!!
…
On Sat, Dec 21, 2019 at 1:23 AM Varun Gupta @.***> wrote: Okay! This makes sense! So we just make and store a json file in the google cloud and update it periodically! Hmm, I wonder if this file could be stored locally? Since this file can be deleted after the export completes I'm not entirely sure if keeping it in the cloud would be a good idea, don't you think? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=AAAF6J6U65XMCJYFU2XPPA3QZWY6PA5CNFSM4JXCVBGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHOWFAQ#issuecomment-568156802>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J37ZKAJOGZNIWHPQP3QZWY6PANCNFSM4JXCVBGA .

Oh! Okay! I'll get started on this then! 😃

@jywarren
Copy link
Member Author

Noting also possibility of uploading the JPG to Google Cloud Storage as well; i think it would involve changing these lines:

image-sequencer-app/src/api/v2/util/process-multisequencer.js

Lines 66 to 79 in 744b521

var html = `<html>`
html += `<img width="100%" src= "${out}">`
html += `</html>`
fs.writeFileSync(path.join(__dirname, `../../../../temp/export${pid}.html`), html);
mapknitterBucket.upload(path.join(__dirname, `../../../../temp/export${pid}.html`), {
gzip: true
}).then(() => {
mapknitterBucket
.file(`export${pid}.html`)
.makePublic();
fs.unlinkSync(path.join(__dirname, `../../../../temp/export${pid}.html`));
mapknitterBucket.file(`export${pid}.html`)
.getMetadata()
.then((data) => changeStatus("uploaded", data[0].mediaLink));

I guess it would be, instead, something like (ref):

       var data = out.replace(/^data:image\/\w+;base64,/, ""); // strip off the data: url prefix to get just the base64-encoded bytes
       var buf = new Buffer(data, 'base64');
       var filename = `export${pid}.jpg`
       var imgPath = path.join(__dirname, `../../../../temp/${filename}`);
       fs.writeFileSync(imgPath, buf);
       mapknitterBucket.upload(imgPath, {
            gzip: true
        }).then(() => {
            mapknitterBucket
                .file(filename)
                .makePublic();
            fs.unlinkSync(path.join(__dirname, imgPath));
            mapknitterBucket.file(filename)
                .getMetadata()
                .then((data) => changeStatus("uploaded", data[0].mediaLink));

@jywarren jywarren mentioned this issue Apr 10, 2020
Merged
@jywarren
Copy link
Member Author

OK i made a PR here: #36

But we will also need to change the status.json output file to now point at the correct JPG file.

@jywarren
Copy link
Member Author

Ah no, that will happen here:

image-sequencer-app/src/api/v2/util/process-multisequencer.js

Line 80 in a3060fb

.then((data) => changeStatus("uploaded", data[0].mediaLink));

So this might just work...?

@jywarren jywarren changed the title Ability to specify a data-url response instead of HTML wrapping a data-url Ability to specify a data-url response (or just a JPG) instead of HTML wrapping a data-url Apr 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jywarren @tech4GT