feat: add retainBody option to the glob() loader

[ascorbic]

Changes

Adds a new retainBody option to the glob() loader to allow reducing the size of the data store.

Currently, the glob() loader stores the raw body of each content file in entry.body, in addition to the rendered HTML in entry.rendered.html.

The retainBody option defaults to true, but you can set it to false to prevent the raw body of content files from being stored in the data store. This significantly reduces the deployed size of the data store and helps avoid hitting size limits for sites with very large collections. This has caused issues for some very large sites, which have hit the maximum JSON string size.

The rendered body will still be available in the entry.rendered.html property for markdown files, and the entry.filePath property will still point to the original file. For MDX collections this will dramatically reduce the size of the collection, as there will no longer be any body retained in the store.

import { defineCollection } from 'astro:content';
import { glob } from 'astro/loaders';

const blog = defineCollection({
  loader: glob({
    pattern: '**/*.md',
    base: './src/content/blog',
    retainBody: false
  }),
});

When retainBody is false, entry.body will be undefined instead of containing the raw file contents.

Testing

Docs

[changeset-bot]

🦋 Changeset detected

Latest commit: fe1513f

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[codspeed-hq]

CodSpeed Performance Report

Merging #15022 will not alter performance

_{Comparing strip-entry-body (fe1513f) with main (8cdaa00)¹}

Summary

✅ 6 untouched

Footnotes

1. No successful run was found on main (0d2adac) during the generation of this report, so 8cdaa00 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[matthewp]

Would it make sense to have this enabled by default in Astro 6?

[matthewp]

Blocked as a minor change.

[ascorbic]

@matthewp yeah, I was going to suggest that

[delucis]

While thinking about content store size, it would be good to get #14315 over the line too. Even with an option like this, at some point things can get too big, so chunking feels like a good solution to that general problem.

[ascorbic]

@delucis yeah, it needs some manual testing and benchmarking though. I would like to be confident it doesn't cause performance regressions.

[pkg-pr-new]

npm i https://pkg.pr.new/astro@15022

npm i https://pkg.pr.new/@astrojs/svelte@15022

commit: fe1513f

[delucis]

@delucis yeah, it needs some manual testing and benchmarking though. I would like to be confident it doesn't cause performance regressions.

Thanks. Hopefully it works! While this option is a nice one for advanced optimizations, the “Hit limit => discover obscure option => double check you’re not actually using body anywhere” flow is not the nicest DX, so if Astro can make it rarer to hit those limits, that might be nice.

(Although we could also potentially one day consider defaulting to not storing the body and require people set retainBody: true in order to use it if we think it’s a rarer use case.)

[ascorbic]

Yes, as @matthewp said, I would suggest we make this the default in Astro 6

[matthewp]

Added this to the 5.17 milestone. Not yet sure when that will be.