Collection Schema Presentation in Workflows

[kiahna-tucker]

Last Updated: Oct. 27, 2022

---

When creating or editing a task, the user should be given an easy means to view the schema of a collection eligible to be or already bound to the task. Aspects of a collection's schema may influence the user's decision to add or remove the entity from the task specification. Secondly, it improves the ease with which the user can troubleshoot build failures related to a bound collection schema, no longer needing to leave their active workflow to review its definition. And finally, equipping the user with a read-only view of eligible and bound collection schemas inside a task workflow can help reinforce the notion that creating/editing a task is a separate process than editing a collection.

NOTE: Before continuing discussion on this subject, it should be noted that schema inference-related design considerations should be viewed as a separate UI concern than that identified in this post. It is advised that the questions of should a user be able to edit a collection schema in the midst of a task-driven workflow and how can we enable that functionality while providing a seamless user experience and upholding flow concepts not be considered here. The aim of this discussion is to determine a satisfactory means of surfacing collection schemas for reference when creating or editing a task in the UI today.

Design

Core Elements

This section identifies the elements the read-only collection schema view should possess to be considered complete.

• A read-only editor to display the collection schema.

• Messaging that informs the user that the collection schema cannot be edited directly in the view or the UI at this time, but can be edited via the CLI. A link to documentation that explains how the user can edit their collection should be provided as well. This would be a great opportunity to create a flow CLI recipe (i.e., an outline of how to perform a commonplace operation via the CLI) and see how its received by our user-base, if one doesn't already exist.

• A means to surface the access token required for the CLI.

• A means to inform the UI that a collection schema has been changed and the view should surface the updated schema.

Impacts to Existing UX

This section identifies the changes to the existing UX (and design) that would be affected by the proposed idea.

• The specification editor section of the create and edit workflows would be reduced to an editor that contains the drafted task specification. The file selector would be removed since the only editable schema would be that of the task specification.

• The schemas of discovered collections would be presented in the proposed collection schema viewer, and thus be read-only. All collection schema editing would need to be performed via the CLI until schema inference is enabled in the UI. As aforementioned, messaging in the UI, access to the CLI authentication token within the active workflow, and step-by-step documentation on collection editing via the CLI aim to facilitate this process.

NOTE: If we are comfortable breaking the fidelity of the task workflow, an argument could be made to allow users to edit the schema of a newly discovered collection (i.e., a collection that has never been published). This distinction could be challenging to communicate to the user, however, so I would advise that all collections are treated similarly.

Sketches

A link to the Excalidraw document will be provided here once the sketch is moved into its own document and annotations created. A screenshot of an initial sketch can be found below.

NOTE: These sketches should be viewed as stencils to assist visual design-driven conversations. The styling of and explicit content within design elements are merely placeholders

Option 1: Direct integration into the resource config section of a create/edit form.

The placement of sketches corresponding to this design option would be found within the resource config accordion of a create/edit form.

Default View | Resource Configuration Tab

Schema-Viewer Tab | Tab Variant A | Access Token Variant 1

Schema-Viewer Tab | Tab Variant A | Access Token Variant 2

The design element represented by the blue text in the alert reading COPY ACCESS TOKEN would be a text button. When clicked, it would prompt the appearance of an element (e.g., popover component, modal, etc.) that would surface the CLI access token. A user would have the freedom to copy the token via the CTA (shown in purple) or manually. The approach Github has taken to surface a repository URL to clone inspired the idea and would likely be a familiar UX.

Schema-Viewer Tab | Tab Variant B | Access Token Variant 1

Schema-Viewer Tab | Tab Variant B | Access Token Variant 2

The design element represented by the blue text in the alert reading COPY ACCESS TOKEN would be a text button. When clicked, it would prompt the appearance of an element (e.g., popover component, modal, etc.) that would surface the CLI access token. A user would have the freedom to copy the token via the CTA (shown in purple) or manually. The approach Github has taken to surface a repository URL to clone inspired the idea and would likely be a familiar UX.

Questions

Directly in Scope

• At what relative frequency are users currently leveraging the specification editor in a create/edit workflow to edit a collection's schema? From what I have seen in LogRocket thus far, a very small percentage of users attempt this. For reference, only the schemas of collections discovered in a capture create/edit workflow can be edited in the specification editor today.

• Arguably, the shape the UI has taken makes a clear distinction between existing entities (i.e., published entities) and new entities. Is the process of editing a drafted collection and a published collection via the CLI the same? What degree of complexity does collection schema editing via the CLI have? A part of this proposal is to create a CLI command recipe for collection editing, so I am trying to gauge the likelihood of a user complete such a guide.

• Understanding that this functionality is temporary (since schema inference will unlock certain UX possibilities regarding collection schema editing), are we comfortable making this transition in the interim? Are there any points of tension you foresee this proposal introducing?

Out of Scope but Worthwhile

• How can we reinforce the streaming nature of the application in the create and edit workflows within the UI?

[dyaffe]

Overall I think the proposal is great and well written. I think I might be a fan of the first variant and would comment that the access token copier is something that would be great vs the current UI. We shouldn't need to show the whole token but rather show just a bit and have a copy button. For the other question:

When creating production data flows users do have to regularly update schema -- they don't when they're just trying something out and in the future we'll hope that they won't have to with the schema changes we'll be introducing.

The process is pretty much the same via the UI and the CLI for updating collections currently. I think long term we'll want to replace our in-ui schema editing with something more user friendly and clickable. Via the CLI, schema appears as YAML files which is pretty nice actually. I would imagine that advanced users would use the CLI command recipe for collection editing -- especially if we could make it easy.

I like this interim transition and can see how we could put schema inference into it -- we could almost have a drop down in the schema tab to select which schema you want to use. We have one we've inferred, potentially one from airbyte, maybe even a third one with only entities we're really certain about or ones that are available on every request, etc.

For your last question, I'd love to show sample documents that have come in recently to reinforce the streaming nature of the UI. That could be a third tab with "sample data".

[psFried]

I think this is an excellent framing of the conversation. And I think the basic proposal seems like a very reasonable and achievable step toward improving the UX. We'll eventually want the ability to edit newly discovered collections as part of the discover workflow in the UI. But IMO it's non-essential in the short term, and worth giving up in order to make it easier to display JSON schemas.

I have a few minor questions and quibbles that seem like good next steps to hash out in terms of working toward a concrete plan.

• IMO it's not necessary or desirable to duplicate the access token for flowctl. That's a mechanism that we'd like to eliminate soon, anyway.
• I do think it could be helpful to show the flowctl command that users can use to pull down the draft and edit it locally. Basically just templating this command and giving them a button to copy it: flowctl draft select --id ${draft_id} && flowctl draft develop

Aside:

I'd like to bring up what I see as a unique challenge with JSON schemas in Flow. JSON (and thus JSON schemas) is really a document, whereas sometimes users will want to think in terms of tabular data. For example, the capture source and materialization destination might both be SQL databases, and the user in that case may be inclined to think in terms of rows and columns. The fact that Flow uses JSON could be considered somewhat of an implementation detail, at least for some users. We'd also like to eventually build some kind of workflow (:wave: :wave:) around materialization field selection, which would also involve some presentation of the JSON schema as a source of fields that could be materialized.

I'm not trying to say that those are requirements from day one. I think a plain JSON or YAML representation of schemas will always be an important option to have, and is seems like the best place to start (basically like in your examples). That's all just trying to shed some light on future plans.