Control plane HTTP API

[psFried]

We'd like to introduce some control-plane HTTP APIs instead of needing to go through postgresT for everthing. This is partly driven by the goal of having an "officially supported" API for users, and so I'd like to:

• Align on what an "officially supported" API should look like
• Come up with a rough plan for how we get there
• See how much of that plan I can execute while adding these new handlers, so I don't have to re-do them later

Here's what I see as some achievable short-term goals:

• Clearly distinguish supported API endpoints from internal and/or beta endpoints
• Serve an OpenAPI spec for all supported APIs
• Be able to tell a sensible story about API compatibility
• Give us an easy way to consume these APIs from flowctl
• Any others I should be looking at?

Compatibility and paths

I'd previously brought up using an /api/v1/ path prefix, and nobody seemed upset about it, so I think we can say that any supported APIs should go under a prefix such as that, and anything else should be considered an internal-only API and not supported for end users.

In terms of compatibility, I'll propose that we start out with just an /api/v0/ prefix for now, and make breaking changes to it as desired until we feel like it's stable enough to bless it as /v1/. Once we do that, we'd commit to only making backward compatible changes to the v1 API. Adding new endpoints is always backward compatible, so we'd still have flexibility to change things without breaking existing clients.

OpenAPI

OpenAPI spec seems like an achievable short term goal, and having it can also make it easier to write client applications, which might be useful in flowctl. Although technically we could defer setting this up depending on our answer to the code organization question below. Personally, I'd rather just figure out the OpenAPI stuff up front, as long as it's not significantly more difficult than it seems.

I found the aide library, which seems like it can serve our OpenAPI needs and integrates with schemars, which we already use. So I'd like to take a stab at integrating that and see how it goes.¹

Accept headers, lol

The is maybe something that only I care about, but I've found HTTP content negotiation to be useful, and I don't want to close off the possibility of having APIs that return something other than just plain JSON (even LD-JSON should have a separate mime type). We don't currently do anything with the Accept header, so a client can send a request with Accept: application/xml and get back a successful JSON response. I'd like to just validate that if a request specifies an Accept header it permits us to return JSON. For now, all our APIs will only return JSON, so we're just ensuring that requests don't explicitly expect otherwise. Doing this now will ensure that it isn't a breaking change later, if we want to have responses of different mime types.

Error handling

I took a stab at making our error handling more ergonomic by adding a custom error type that has From impls for other common error types. Then handler functions can return Result<Json<SuccessResponse>, ApiError> instead of needing to use the wrap function. This also allows us to return different HTTP status codes as appropriate for different types of errors.

Code organization

I'm thinking specifically about how we share request and response types between the agent and flowctl. One option would be to have flowctl use openapi-generator to generate the API client types, just like a user would. Some questions I still have about this:

• How well would the generated client code would integrate with flowctl?
• What would be the impact on our build/dev process?

Another option would be to factor out a separate control_plane_api crate where we define the rust types used in our API, and then have both flowctl and agent depend on that. It's a little easier to answer the above questions with this approach, but I'm still not sure whether we might run into other snags.

As of now, I'm planning to explore both options at least a bit more, but I'd like to hear some other opinions or ideas if you have any.

Other?

Does anyone have strong feelings about response representations, or other stuff that we should standardize early on? Or alternatives to what I've outlined above?

Footnotes

1. Aide has a much nicer API that seems well integrated with axum. Unfortunately, it is not very actively maintained. utoipa is an alternative that we already use in the source-http-ingest connector. But utoipa doesn't support schemars, so we'd have to manually convert all the schemars schemas to utoipa ones. More importantly, the API requires defining your routes using their confusing macro. So I'm thinking to try aide first and see how it goes. ↩

[travjenkins]

anything else should be considered an internal-only API and not supported for end users.
I am not sure what exactly you mean here. Does this mean they will still be available via an HTTP endpoint but just under another prefix (or no prefix at all)?

Accept Headers
YES YES YES! I would love to start using these as we could do cool stuff having endpoints returning HTML/Markdown blurbs for error rendering.

Error handling
I have used/read/thought about OpenAPI in a hot minute. Do they handle the decision/fight over what HTTP response codes to send and when? Just have a lot of experience in places where this gets weirdly argument-y quickly.

[psFried]

Do they handle the decision/fight over what HTTP response codes to send and when?

The OpenAPI spec is unopinionated about response codes. For now, the generated spec just uses a default response schema for all errors, which is used when there's no response for a given http status code. In general, I try to avoid being too creative with the use of http status codes, and just stick to the guidelines from MDN. The only quirk is that we return a 404 when requesting live specs that the user doesn't have permission to read, so that we don't leak information about the existence of specs that the user is not supposed to know about.

[psFried]

I just pushed up a branch with a POC of using aide to generate the openapi docs. You can run it locally and check out the api and the generated docs.

• http://localhost:8675/api/v1/openapi.json has the openapi json spec
• http://localhost:8675/api/v1/ has the "scalar" generated docs page
• http://localhost:8675/api/v1/redoc has the "redoc" generated docs page

I like aide fairly well so far, so I'm thinking I'll see how it does with one or two other endpoints. aide comes with two different openapi frontends, and I've yet to form a preference, but we should probably just stick with one of them. Then next steps are to try out generating client code vs factoring out a shared crate.

[psFried]

I ended up going with Scalar for the generated docs page, because it includes the ability to make requests directly from the UI. Redoc only offers that with their paid offering.

[psFried]

I tried out using openapi-generator to generate a rust client crate. The results so far have been rather disappointing. The main issues I see with it are:

• It generates some_field: Option<Option<SomeField>> for fields that are not required and also permit the null type. This is a pretty common in a lot of our schemas, though it might be something we could address by updating our schemas. Example here (ignore the extra Box wrappers, as those can be easily disabled)
• It generates some absolutely disgusting enums when schemas use oneOf

Both of those issues might be solvable by updating our JSON schemas, though it seems like we'll get more bang for our buck by taking the types we already have and factoring them out into the models crate. So I think that's the path forward here.

[psFried]

In terms of compatibility, I'll propose that we start out with just an /api/v0/ prefix for now, and make breaking changes to it as desired until we feel like it's stable enough to bless it as /v1/.

This ended up feeling a bit silly. I've instead introduced the endpoint under /api/v1/ and we can make breaking changes there if we need to before we release it. This avoids needing to change paths, or maintain multiple routes for the same handler.

[danielloader]

Any traction on this? We're looking to start paying for the hosted cloud version of Estuary Flow but we would need to be able to see a control plane API (and as such a terraform provider) on the roadmap before we can scale it out past a few clickopsy style prototypes.

We need to be able to provision a ETL flow per database and we have a database per client tenant, and we have a control plane to build out this postgres per tenant model - but we would need to be able to on demand tack on a "dump to s3" type functionality for the client. We're evaluated the product and it looks to do what we want with parquet or CDC/kafka but not being able to drive all the configuration for these flows via an API/SDK and ultimately terraform is a bit limiting.

Thanks!

[dyaffe]

This is in progress right now -- we're building out a full graphql api and transitioning all of our internal services to use it. As soon as that happens, we'll make it publicly available to customers.