- The Stream Service consists of API and Worker nodes.
- API nodes contain configuration endpoints and collect data via the Import endpoint.
- Worker nodes import the collected data into Storage tables in batches.
User documentation can be found at https://developers.keboola.com/integrate/push-data/.
- Entrypoint: cmd/stream/main.go
- The server utilizes code generated from the API design.
The API design can be found in api/stream/design.go and is implemented using Goa framework.
There is generate-stream-api make command available to generate the code from the design specs. (The command is
also run before other commands to run the API locally, build API image, release the API, etc.)
The command generates:
- Code to internal/pkg/service/stream/api/gen.
- OpenAPI specifications to internal/pkg/service/stream/api/openapi.
Endpoints behavior is implemented in internal/pkg/service/stream/api/service/service.go.
Endpoints code performs validation of user inputs and typically runs one or more operations. See internal/pkg/service/common/dependencies/dependencies.go for a detailed explanation of dependency injection and the command design pattern implementation.
TODO
The Service uses an etcd database to stream incoming data until they are imported to Storage.