More portable data payload

[chrisvxd]

Problem

The current data model is hard to reason about, which makes it less portable and harder to build custom renderers on top of. Some concerns:

• zones are weird
• The Root component is treated differently to the other components
• DropZones rely on context or render props, which are React-only paradigms, making it hard to implement Multi-framework support #302 and negatively impacting performance

Example of a custom renderers:

• XML string (i.e. to produce React code)
• Non-React outputs

Current data model

{
  "content": [
    {
      "type": "Columns",
      "props": {
        "id": "Columns-1234"
      }
    }
  ],
  "root": { "props": { "title": "Page" } },
  "zones": {
    "Columns-1234:left": [
      {
        "type": "Heading",
        "props": { "title": "Hello, world" }
      }
    ]
  }
}

Proposal

react-from-json implements a predictable AST based on, but not tied to, the React tree. It would allow for custom renders and increased portability.

Proposed data model

{ 
  "root": {
    "type": "Root",
    "props": {
      "title": "Page",
      "children": [
        {
          "type": "Columns",
          "props": {
            "left": [
              {
                "type": "Heading",
                "props": { "title": "Hello, world" }
              }
            ]
          }
        }
      ]
    }
  },
  "content": [], // Deprecated, but still supported
  "zones": {}, // Part of DropZone API which will be deprecated, but still supported
}

New slot API

The children prop in the above will be powered by the slot field API, which implements a DropZone-like API but as a field. This makes data much more portable, as it's part of the same payload, whilst also taking advantage of the numerous field APIs (defaultProps, resolveData, etc).

Current proposal:

const config = {
  fields: {
    leftColumn: { type: "slot" }
  },
  defaultProps: {
    leftColumn: {} // Unlike DropZones, you can set defaultProps for slots
  },
  render: ({ leftColumn }) => (
    <div>{leftColumn()}</div>
  )
}

DropZone's will be deprecated but retained to avoid breaking changes, but user's will be directed to the slot API instead.

Considerations

• Implement migration using migrate() API
• Consider how content will be deprecated. We could 1) retain content as is indefinitely, 2) retain support for the user's existing payload (i.e. if content is passed in, content will be passed out until the user migrates the data using migrate()), or 3) force the user to migrate() their data (breaking change).
• Plugins modifying the Data payload will not immediately support slots or content if moved under root.props. Plugins are experimental, so no mitigation necessary.
• Consider making the react-from-json model a portable standard
• Consider how we might solve for detached roots, such as in Rendering Puck components outside of DropZone #196

Blockers

• Improve rendering performance #644, which is changing the underlying state management

[monospaced]

This makes sense to me 👍🏻

But I do think we should have a plan for mitigating before commiting to this

4. If the user modifies the output in any way, this will break

[BleedingDev]

I would check the output of BuilderIO. If it would be close to their, we could leverage their renderer components.

[jperasmus]

The proposed nested structure is a lot easier to work with.

I suggest you consider nesting the data payload one level deep and storing the current version of puck in the root level. Any other additional metadata could be persisted this way as well which will be beneficial if you need to run data migrations for existing pages.

{
  "puckVersion": "0.14.0",
  "content": {
    "type": "Root",
    "props": {
      "title": "Page",
      "children": [
        {
          "type": "Columns",
          "props": {
            "left": [
              {
                "type": "Heading",
                "props": { "title": "Hello, world" }
              }
            ]
          }
        }
      ]
    }
  }
}

[Tuyentran2k1213]

Hi @chrisvxd is this on working ??? this feature will be needed

[chrisvxd]

@Tuyentran2k1213 it's on our radar but we haven't started efforts yet! Hoping to look at it over the next few months.

[simon-virkesborsen]

@chrisvxd Would a good first step to be implementing this with a feature flag to avoid the breaking change at first?

[chrisvxd]

@simon-virkesborsen perhaps, but it's so core to the Puck implementation that it might cause a lot of overhead to retain two separate implementations.

One thing I'm considering is retaining the internal behaviour and just mapping the external data payload. That would enable the use of a feature flag.

[mkilpatrick]

In addition to the new format (love the puckVersion idea), could there be a non-puck field included like metadata to allow users to add additional info that we may require for our own services? Puck would ignore this field but allow additional custom attributes external systems could use.

This could be extended to the component config too.

As a concrete example, I'm looking to eventually add role-based controls where certain users can only control certain components/fields. I'm not sure if something like this is already on your roadmap, but with some custom metadata we could at least roll our own thing in the meantime. I'm sure there are countless other use cases for this too.

[shannonhochkins]

In addition to the new format (love the puckVersion idea), could there be a non-puck field included like metadata to allow users to add additional info that we may require for our own services? Puck would ignore this field but allow additional custom attributes external systems could use.

This could be extended to the component config too.

As a concrete example, I'm looking to eventually add role-based controls where certain users can only control certain components/fields. I'm not sure if something like this is already on your roadmap, but with some custom metadata we could at least roll our own thing in the meantime. I'm sure there are countless other use cases for this too.

I think this is a good start, there might also be internal config you might want to update too, like the dropzone properties or if a component should be draggable within a dropzone etc - might need to think about where this lives under this new proposed structure

[janbaykara]

Hi all. Firstly, big respect to the Puck project so far. This is all very cool.

We're exploring using Puck as a React-embedded editor on top of Wagtail, the Django-based CMS. Wagtail stores page content as Revisions, a JSON blob that serialises the Django model. It additionally has a concept of StreamField for block-based content storage.

From a purely self-motivated perspective, I'd wonder if there can be some convergence between these data structures and Puck's.

From the open source ecology perspective, @chrisvxd's idea of mapping Puck data structures to other ones strikes me as a versatile idea. Mapping packages, between Puck and third party data formats (e.g. puck-to-wagtail and wagtail-to-puck), would be a very cool addition to the ecosystem.

[mkilpatrick]

Wondering if this is currently in the works or when this might be completed. Once the format changes it will be hard to move existing data to the new format unless there's a migration plan. Either way, it'd be easier to wait for this before using Puck in a Production setting.

[chrisvxd]

@mkilpatrick we'll ensure there is a robust migration plan. Not yet in the works, but I'm hoping to begin working on it relatively soon (can't commit to timeline yet, as many client and personal commitments).

[mkilpatrick]

Something I was thinking about recently is separating the "view" from the "content." What I mean by this is having the used components and ordering be a key that's separate from the props data, opposed to having the latter nested in the current content key.

As a concrete use case, let's say an admin goes and creates the puckData final "layout" (everything dragged to build the page). Then a separate user is able to build on top of what that admin did. Maybe they can reorder the components, but more importantly, they can set the values of the component props. The admin comes back and makes adjustments to the layout, maybe adding more components. We wouldn't want to lose the edits made by the non-admin user and we'd have to implement some sort of merging between the two to get the final result. I think this would be easier if the prop data was recorded separately from the structural data.

I recall seeing another issue related to history diffs versus full copies. This idea is sort of in that same vein.