RFC 100: Enhancing headless support in Wagtail core

[thibaudcolas]

View as an HTML document

Requires further work. This is based on my experience building a copy of the bakerydemo with Next.js: bakerydemo-nextjs.

[ahosgood]

I had to add extra code to get redirects to work in the API - would that be helpful to have included?

[thibaudcolas]

@ahosgood I would assume so? Redirects is a contrib module so not as "core" as some of the more fundamental aspects of the CMS, but certainly something that many sites would consider core CMS functionality.

[lb-]

I think it would be good to somehow include the pattern for JSON rendering using normal Page routing. Essentially each Page path can have a different request header and then return JSON instead of HTML.

It's quite an intuitive approach and aligns with how other CMSs (e.g. Adobe Experience Manager) can provide their APIs. It's not going to make sense for every installation but worth reviewing as part of this RFC.

In the abstract, this also could align with applications that may want to build out HTMX style applications, still returning HTML but providing a partial render of 'inner' HTML based on request headers.

See wagtail/wagtail#11752 & wagtail/wagtail#8374

[allcaps]

Can we include focal points?

Some frontends will handle the images. Instead of Wagtail providing various renditions, Wagtail would only supply the original image and leave the resizing to the frontend application. In this scenario it makes sense to include the focal point information in the API response.

[thibaudcolas]

Yep. I see we support accessing the focal point in templates so feels like appropriate feature parity.

[allcaps]

I've numbered my feedback. I hope it helps to keep discussion structured. There is no specific order.

I'm not sure if my points are in line with the goal of this RFC. I just mention them as I've experienced them as points to address in my headless projects.

1. All page URLs endpoint

In a frontend built step, all pages need to be build. Wagtail could use an endpoint listing all page URLs (no other fields needed, without pagination). One call to built them all.

2. Invalidate/purge

Because the frontend acts as a cache, it needs to be invalidated/purged. In Wagtail, it is necessary to track which object is used on which page so that when content is mutated, the correct frontend components/pages are invalidated/purged.

3. OpenAPI specification and/or JSON schema

I really like to have an OpenAPI specification or JSON schema available. This makes frontend development easier. Could Wagtail provide a schema for all its endpoints? Via introspection of Models and streamfields?

FYI: I my headless projects I roll my own DRF endpoints and DRF Spectacular for this.

4. Point Wagtail admin live button to the frontend

Override Page.get_url_parts to point the Wagtail admin live button to the frontend URL.

5. Override Page.serve to serve the API response

I'd like to override the Page serve method to serve the API response. This gives a one-to-one mapping with the frontend application and makes development much easier. I think many headless projects could benefit from this.

class APIMixin:
    def serve(self, request, *args, **kwargs):
        request.is_preview = getattr(request, "is_preview", False)

        serializer = ...

        # A DRF view gives us nice html or json output.
        # The default render class should be listed first.
        @api_view(("GET",))
        @renderer_classes([JSONRenderer, BrowsableAPIRenderer])
        def view(req):
            return Response(serializer(self, context={"request": request}).data)

        return view(request)

This is also a (partial) solution for the userbar. Example: it is easy to switch between https://nl.wagtail.space/ and https://cms.wagtail.space/

6. The way to expose Settings (and Snippets)

Site settings defining a menu, header, footer, contact details, or what not, should be exposed via REST API endpoints. Wagtail could define a best practice.

7. Translation of hardcoded strings

There are strings marked for translation in Django/Wagtail, and Django has the JavaScriptCatalog view. Many frontends come with their own translation solutions.

This might duplicate the way to lookup a specific translation, or lead to duplicated definitions.
If the decision is to use Django JavaScriptCatalog view, frontend translations might end-up in the backend.

8. Forms

Regular Django forms and Wagtail Form Builder forms.

Both need to re-implement rendering in the frontend.
Form validation has logic in backend (to clean/validate the submitted data) and frontend (for direct feedback).

Forms that validate input on-the-fly against the backend and use the error messages supplied by the backend have a single point of truth. To me, this works best.

9. RoutablePageMixins

The RoutablePageMixin mixin provides a convenient way for a page to respond on multiple sub-URLs with different views. The page is the starting point for fixed URL patterns.

I'm not sure how Wagtail API suggests to handle these. I guess point 5 is a possible solution?

[ahosgood]

Should/can we add the Wagtail accessibility checker in here? It's a really useful tool and while it works in the CMS, it might be a good idea to have the ability to add it to the frontend as well. Of course that also means there will be a requirement for validating logged in Wagtail users in the frontend and including some JS and CSS.

Maybe I'm overthinking it?

[laymonage]

I believe it's part of the "Content checks" and "Userbar" items. The accessibility checker is one of the content checks, and we currently piggyback on the userbar (the Wagtail button in the frontend) to implement it 🙂

[thibaudcolas]

Yes. I’ll update the RFC to make sure it’s more explicit.

[saevarom]

Headless was on the agenda for Wagtail Space NL but this only involved incorporating the areweheadlessyet.org website into the Wagtail documentation. The resulting PR is here: wagtail/wagtail#12039
The sprint topic discussion is here: https://paper.dropbox.com/doc/Sprint-topics-Wagtail-Space-NL-2024--CUVkcdc_H1K~k2BWz8SS9UUbAg-3rkJ5imATtyajKY2Ey1Cr#:uid=172696553243645599787735&h2=Headless-Wagtail

[dopry]

With a headless site, the backend only needs to store the redirects. Wagtail currently does this. The frontend needs to be able to query the redirects from the headless cms and apply them. I would say that wagtail provides what a headless cms needs to provide for handling redirects at this juncture. I have a next.js front end that query's redirects from wagtail at build time and it works just fine.

[ahosgood]

I'm not entirely sure it does 🤔

I had to use some custom code for finding a page by URL when that page had a redirect: https://github.com/nationalarchives/ds-wagtail/blob/develop/etna%2Fapi%2Furls.py#L162-L195

[dopry]

I don't go through the trouble of querying redirects page by page. I just grab all redirects and renderer them as pages that redirect or update the redirect list on my CDN at build time. You might also consider graphql/grapple instead of interacting with the API directly. It's much easier to get all your data per page in a single query that way vs making oodles of rest calls everytime you generate a page.

[thibaudcolas]

@dopry ah I think you’re right, I didn’t realise there was already a built-in endpoint to fetch the redirects.

@ahosgood if I understand the code you’re sharing, it returns a page that’s associated to a redirect and matches the provided html_path? That feels useful but more than I’d expect is needed to get to feature parity (which is what this RFC is about). In that scenario, it seems like the site’s front-end would return a 200 with the page data for the redirect path?

[ahosgood]

@dopry I'm not sure I've understood fully.

Our headless CMS is a separate service to the frontend and they are built and deployed independently so building Wagtail doesn't interact with the frontend code.

Also, redirects can be added at any time by updating a Wagtail slug so I don't think computing all redirects at build time is a possibility for us.

[ahosgood]

@thibaudcolas Essentially, yes.

We had two choices:

1. Return the page data even when requesting the "old" path (redirect just in the API)
2. Return the data needed for the frontend to be able to redirect

In the frontend I have made a simple switch for both of these options because I wasn't sure which was going to be best for us. https://github.com/nationalarchives/ds-frontend/blob/main/app/wagtail/routes.py#L231-L239

As you can see, I also had some (now commented out) code to return redirects to external sites as well which is also possible in Wagtail.

[dopry]

@ahosgood We generate the redirects when we build the frontend. We're using next.js on Netlify. In our nextjs config we have a function for the next.js redirects key that queries the redirects from grapple. When it get's published on netlify, their Next.JS integration pushes the redirects to their CDN. We also render the redirects as pages just in case for some unforseen reason the redirect isn't in the CDN it's a static page. We are querying redirects through grapple/graphql. @thibaudcolas I'm am not sure that the native wagtail API provides a list of redirects. They are available in grapple/graphql interface.

[ahosgood]

The API can expose redirects (https://docs.wagtail.org/en/stable/reference/contrib/redirects.html#api) but it sounds like your use case is to query all the redirects and then build your service using them whereas ours is not built statically.

Our frontend just queries the Wagtail API directly so I can add a redirect to Wagtail at any point and the frontend can use it straight away.

The non-headless Wagtail responds to a URI whilst taking redirects into consideration. The API does not (without having to make multiple calls) so I would argue that it is needed for parity.

[lb-]

Another long-standing issue with the current API is that we cannot easily generate an OpenAPI specification

See wagtail/wagtail#6209

I would say that we must have a documented or even out of the box way to generate specifications for our APIs if we want to truly say we have headless support.

[lb-]

Some questions to consider.

1. Would we want to consider a more open schema driven response structure ever? We kind of have this with the meta/data response objects but maybe go further with alignment to something like https://jsonapi.org/ or https://www.w3.org/TR/json-ld/ or similar.
2. What kind of story to we want to tell when it comes to emerging community practice such as HTMX/Turbo style applications? Or do we consider this not really 'headless'?

[dopry]

1. While API documentation is nice to have, I don't think it is essential for building a headless site. It shouldn't be something that blocks a consistent approach to headless previews and ensuring the API exposes all data (pages, snippets, redirects), and functions (crud, search, etc.) necessary for a headless client.

2. To me headless implies my CMS is no longer rendering/serving html. With AHAH (htmx, hotwire, et al) html is still being rendered at the server, so I wouldn't consider it headless. If you had a server responsible for just rendering that was sending hotwire/htmx to the client and not calling on wagtail directly, then that would be headless in my book. I feel like AHAH support should be it's own distinct thing, and it likely to be specific to the AHAH framework in play.

[allcaps]

@dopry The JSON and OpenAPI specifications are valuable not only for generating documentation but also for generating TypeScript types and even complete client applications. From my experience, using machine-generated types is a must. When the Wagtail data structures change, updating the frontend application is straightforward.

These schemas and types also enhance development with code editor autocompletion, which boosts productivity and reduces the likelihood of errors.

[dopry]

@allcaps it's a productivity boon, but not a requirement to implement a headless site. You can create a headless site without it. In my experience adding and removing properties from frontend data access layers is trivial. Figuring out what properties need to be added/removed/modified from the templates when that happens is the time sink, and schema documentation doesn't resolve that particular problem.