Skip to content

Commit

Permalink
Adds bacalhau support
Browse files Browse the repository at this point in the history
Adds a destination node for bacalhau that is able to submit jobs to a
local requester node, using the orchestrator API.

Each instance of the destination node is configured with the location of
the jobstore, a local directory containing .yaml files describing jobs.
It should also be provided with the job name (one of the .yaml files)
so that it might be configured with the following

```
jobstore: /tmp/jobstore
job: process
```

and will load /tmp/jobstore/process.yaml as the job specification.  Once
loaded the job specification can be templated using handlebars syntax,
so that {{key}} will be replaced with values from the 'key' field in the
input message.
  • Loading branch information
rossjones committed Jan 12, 2024
1 parent 937a917 commit fc8df54
Show file tree
Hide file tree
Showing 17 changed files with 686 additions and 42 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,5 @@ Cargo.lock

# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb

.vscode/
155 changes: 128 additions & 27 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ members = [
"pipe/runtime",
"pipe/arrow_msg",
"pipe/section",
"pipe/section/section_impls/bacalhau",
"pipe/section/section_impls/hello_world",
"pipe/section/section_impls/sqlite_connector",
"pipe/section/section_impls/excel_connector",
Expand Down
9 changes: 9 additions & 0 deletions common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ pub enum Destination {
Sqlite_Connector(SqliteConnectorConfig),
Snowflake(SnowflakeConfig),
Sqlite_Physical_Replication(SqlitePhysicalReplicationDestinationConfig),
Bacalhau(BacalhauDestinationConfig),
Hello_World(HelloWorldDestinationConfig),
Kafka(KafkaDestinationConfig),
Postgres_Connector(PostgresConnectorDestinationConfig),
Expand Down Expand Up @@ -115,6 +116,14 @@ pub struct SqlitePhysicalReplicationSourceConfig {
// database path
}

#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct BacalhauDestinationConfig {
#[serde(flatten)]
pub common_attrs: CommonAttrs,
pub job: String,
pub jobstore: String,
}

#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct HelloWorldSourceConfig {
#[serde(flatten)]
Expand Down
Loading

0 comments on commit fc8df54

Please sign in to comment.