[schema] Only download and process 3rd party images as they are queried

[grantglidewell]

Summary

Compile an 'expected schema' from queries that are created by the user in components and in gatsby-node.js. This can be used to reduce build times, address image processing issues, and implement better preview experience. Use it in gatsby-image to ensure the image being processed could be used in the app. For hot reloading and preview use the schema to update static pages generated in gatsby-node.js that do not respond the same way a page with a staticQuery does.

Basic example

Using the same mechanism that runs the static queries, collect them along with queries in gatsby-node to create an expected schema. This information can be used in gatsby-image to evaluate whether or not an image should be processed. So instead of so onCreateNode can exit early if the image is not going to be used in the app. If nodes that are used in templates are updated gatsby can re-render the static page since it is now aware that these a rendered with dumb templates with no query of their own.

the createNode flow might work something like:

if(nodeIdExists && expectedScema[nodeId].internal.hasDumbTemplate) {
createNode(node)
reRenderTemplate()
}

image could make a similar comparison

if(typeof Node === 'image' && expectedSchema.has.this.node || expectedSchema.has.parent.of.this.node) {
  doImageProcessingNow()
}

Motivation

Preview (Drupal) behavior is broken if a page is created through gatsby-node and does not have its own static query in the template.

[KyleAMathews]

I'm a bit confused still — what problem is this trying to solve?

[grantglidewell]

I'm a bit confused still — what problem is this trying to solve?

Live preview for Drupal doesnt re-render pages that are created with gatsby-node and passed context, only pages that use their own queries or a static query. Gatsby shouldnt care how you generate pages, it should always re-render them if the data they use in the schema changes. However that isnt the case, and this is an approach that not only can fix that, but help other performance issues, like the gatsby-image issues that I mentioned.

[KyleAMathews]

Oh interesting — yeah, it's pretty hard to know where data passed as context to createPage comes from (since it literally could be anywhere).

We can re-render pages with their own graphql queries because we know where data comes from.

There's an open issue for this #11691

[KyleAMathews]

On gatsby-image — I'm not sure what problem you're seeing? We already do caching for gatsby-image so image thumbnails shouldn't be recreated after they've been recreated once. Could you provide more information on what you're seeing there?

[grantglidewell]

We already do caching for gatsby-image so image thumbnails shouldn't be recreated after they've been recreated once.
They should never be processed if they wont be referenced in the app to begin with, its wasted effort up front.

example: I pull in a Drupal site and on node creation each image is processed (iirc this is how gatsby-image works), wether or not i use it in the app. That could be (and has been) a huge problem if a Drupal back end wasnt initially designed to be headless. So in a site that I use like 30 images, 3000+ are processed ballooning build times (so I dont use gatsby-image). I know the solution to this could be limiting JSON API and also using filters in the source plugin, but I have run into situations where that isnt enough.

Even using this in gatsby drupal source to know which assets to download and what we can ignore would help with build times there.

This solves a problem that, as people adopt Gatsby and migrate to headless from monolith (yuck) CMSs, will become more prevalent. early optimization? Maybe, but this is a feature where Ive only been able to identify three uses, but there are likely more, and the data is all there, waiting to be used. Possibly help with incremental builds? Possibly cache this 'schema' and diff it along with gatsby-node plugins to invalidate the cache?

[KyleAMathews]

Are images being processed or just downloaded? We do download images by default but they're not processed in any way unless one is queried.

It would be awesome to only download an image if it's queried. That'd be a great optimization. That should actually be possible with the new schema customization tools. /cc @freiksenet, @stefanprobst @pieh

[grantglidewell]

I thought for some reason it was onCreateNode that did a check if its a file and image, I guess that just creates the scaffolding for image sharp, wait, I may be mixing up image sharp here. But whatever the case I think you have the general idea now.

Yes, by default in the drupal source plugin all assets that are exposed are downloaded. I think to do with the with schema customization tools it would have to be manual? (I could be very wrong here). But using 'expected schema' it could be completely automated and not require the user to do any work, and get more free perf.

[grantglidewell]

I just did a quick test and the issue exists in a Contentful/Gatsby cloud stack as well. I will assume it's going to be widespread in any preview scenario.

The question is this, do we want to allow people to generate pages this way and still support preview? I would think so as limiting the use of createPage or any of the SSR APIs would kind of defeat the purpose/ease/philosophy of Gatsby.

Maybe this isnt the solution but requiring users to write their components/generate pages in a very specific way in order to get preview seems like requiring them to solve the problem for us (referencing the workaround in #11691). With this solution, a tool is introduced that could make a lot of other areas better/faster/stronger. I get that it isn't an easy lift though.

[KyleAMathews]

This seems pretty doable as gatsby-source-filesystem could download the file when loadNodeContent is called.

The trick would be how to create the File node and let transformers do their thing without downloading the remote file. You can get some info by doing HEAD request e.g. the content size. gatsby-transformer-sharp could use https://github.com/nodeca/probe-image-size to get image info. Other transformers would need the whole file.

So a transformer would need some sort of way to not download the whole file. Which actually could be as simple as just not calling loadNodeContent — gatsby-transformer-sharp would just look at the node (perhaps gatsby-source-filesystem could say that the file hasn't been downloaded yet? Or that the file is remote?) and then just use probe-image-size instead of loading the file directly.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[TylerBarnes]

Looks like this went stale and was closed, but this would be an awesome feature. Ok to re-open?

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!