Replies: 5 comments 3 replies
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I just pushed up a branch with a POC of using
I like |
Beta Was this translation helpful? Give feedback.
-
|
I tried out using
Both of those issues might be solvable by updating our JSON schemas, though it seems like we'll get more bang for our buck by taking the types we already have and factoring them out into the |
Beta Was this translation helpful? Give feedback.
-
This ended up feeling a bit silly. I've instead introduced the endpoint under |
Beta Was this translation helpful? Give feedback.
-
|
Any traction on this? We're looking to start paying for the hosted cloud version of Estuary Flow but we would need to be able to see a control plane API (and as such a terraform provider) on the roadmap before we can scale it out past a few clickopsy style prototypes. We need to be able to provision a ETL flow per database and we have a database per client tenant, and we have a control plane to build out this postgres per tenant model - but we would need to be able to on demand tack on a "dump to s3" type functionality for the client. We're evaluated the product and it looks to do what we want with parquet or CDC/kafka but not being able to drive all the configuration for these flows via an API/SDK and ultimately terraform is a bit limiting. Thanks! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
We'd like to introduce some control-plane HTTP APIs instead of needing to go through postgresT for everthing. This is partly driven by the goal of having an "officially supported" API for users, and so I'd like to:
Here's what I see as some achievable short-term goals:
Compatibility and paths
I'd previously brought up using an
/api/v1/path prefix, and nobody seemed upset about it, so I think we can say that any supported APIs should go under a prefix such as that, and anything else should be considered an internal-only API and not supported for end users.In terms of compatibility, I'll propose that we start out with just an
/api/v0/prefix for now, and make breaking changes to it as desired until we feel like it's stable enough to bless it as/v1/. Once we do that, we'd commit to only making backward compatible changes to the v1 API. Adding new endpoints is always backward compatible, so we'd still have flexibility to change things without breaking existing clients.OpenAPI
OpenAPI spec seems like an achievable short term goal, and having it can also make it easier to write client applications, which might be useful in flowctl. Although technically we could defer setting this up depending on our answer to the code organization question below. Personally, I'd rather just figure out the OpenAPI stuff up front, as long as it's not significantly more difficult than it seems.
I found the aide library, which seems like it can serve our OpenAPI needs and integrates with
schemars, which we already use. So I'd like to take a stab at integrating that and see how it goes.1Accept headers, lol
The is maybe something that only I care about, but I've found HTTP content negotiation to be useful, and I don't want to close off the possibility of having APIs that return something other than just plain JSON (even LD-JSON should have a separate mime type). We don't currently do anything with the
Acceptheader, so a client can send a request withAccept: application/xmland get back a successful JSON response. I'd like to just validate that if a request specifies anAcceptheader it permits us to return JSON. For now, all our APIs will only return JSON, so we're just ensuring that requests don't explicitly expect otherwise. Doing this now will ensure that it isn't a breaking change later, if we want to have responses of different mime types.Error handling
I took a stab at making our error handling more ergonomic by adding a custom error type that has
Fromimpls for other common error types. Then handler functions can returnResult<Json<SuccessResponse>, ApiError>instead of needing to use thewrapfunction. This also allows us to return different HTTP status codes as appropriate for different types of errors.Code organization
I'm thinking specifically about how we share request and response types between the agent and flowctl. One option would be to have flowctl use
openapi-generatorto generate the API client types, just like a user would. Some questions I still have about this:Another option would be to factor out a separate
control_plane_apicrate where we define the rust types used in our API, and then have both flowctl and agent depend on that. It's a little easier to answer the above questions with this approach, but I'm still not sure whether we might run into other snags.As of now, I'm planning to explore both options at least a bit more, but I'd like to hear some other opinions or ideas if you have any.
Other?
Does anyone have strong feelings about response representations, or other stuff that we should standardize early on? Or alternatives to what I've outlined above?
Footnotes
Aide has a much nicer API that seems well integrated with
axum. Unfortunately, it is not very actively maintained.utoipais an alternative that we already use in thesource-http-ingestconnector. Bututoipadoesn't supportschemars, so we'd have to manually convert all theschemarsschemas toutoipaones. More importantly, the API requires defining your routes using their confusing macro. So I'm thinking to tryaidefirst and see how it goes. ↩Beta Was this translation helpful? Give feedback.
All reactions