Add Local Processing Agents (#323)

Author: fivetran-aleksandrboldyrev

CHANGELOG.md

@@ -7,9 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased](https://github.com/fivetran/terraform-provider-fivetran/compare/v1.2.0...HEAD)
 
-## [1.2.0](https://github.com/fivetran/terraform-provider-fivetran/compare/v1.1.26...v1.2.0)
+## [1.2.0](https://github.com/fivetran/terraform-provider-fivetran/compare/v1.1.27...v1.2.0)
+
+## Added
+- New resource `fivetran_local_processing_agent` that allows to manage Local Processing Agents.
+- New data source `fivetran_local_processing_agent` that allows to retrieve details of the existing Local Processing Agent for a given identifier.
+- New data source `fivetran_local_processing_agents` that allows to retrieve the list of existing Local Processing Agents available for the current account.
 
 ## Updated
+Updates to support management of local processing agents:
+- Resource `fivetran_connector` updates:
+  - Added field `fivetran_connector.local_processing_agent_id`.
+
+- Resource `fivetran_destination` updates:
+  - Added field `fivetran_connector.local_processing_agent_id`.
+  - Added field `fivetran_connector.networking_method`.
+
+- Datasource `fivetran_connector` updates:
+  - Added field `fivetran_connector.local_processing_agent_id`.
+
+- Datasource `fivetran_destination` updates:
+  - Added field `fivetran_connector.local_processing_agent_id`.
+  - Added field `fivetran_connector.networking_method`.
+
 - Resource `fivetran_connector_schema_config` reworked, added new validation logic and new field `validation_level`.
     - `fivetran_connector_schema_config.validation_level` allows to setup desired level of schema validation before apply
         - NONE: no validation needed, the fastest way to apply schema, especially for a newly created connector - it will use new [Create a Connector Schema Config](https://fivetran.com/docs/rest-api/connectors#createaconnectorschemaconfig) endpoint.

docs/data-sources/connector.md

@@ -30,6 +30,7 @@ data "fivetran_connector" "connector" {
 - `destination_schema` (Block, Read-only) (see [below for nested schema](#nestedblock--destination_schema))
 - `failed_at` (String) The timestamp of the time the connector sync failed last time.
 - `group_id` (String) The unique identifier for the Group (Destination) within the Fivetran system.
+- `local_processing_agent_id` (String) The local processing agent ID that refers to the controller created for the group the connection belongs to. If the value is specified, the system will try to associate the connection with an existing agent.
 - `name` (String) The name used both as the connector's name within the Fivetran system and as the source schema's name within your destination.
 - `networking_method` (String) Possible values: Directly, PrivateLink, SshTunnel, ProxyAgent.
 - `pause_after_trial` (Boolean) Specifies whether the connector should be paused after the free trial period has ended.

docs/data-sources/destination.md

@@ -26,6 +26,8 @@ data "fivetran_destination" "dest" {
 - `config` (Block, Read-only) (see [below for nested schema](#nestedblock--config))
 - `daylight_saving_time_enabled` (Boolean) Shift my UTC offset with daylight savings time (US Only)
 - `group_id` (String) The unique identifier for the Group within the Fivetran system.
+- `local_processing_agent_id` (String) The local processing agent ID that refers to the controller created for the group the connection belongs to. If the value is specified, the system will try to associate the connection with an existing agent.
+- `networking_method` (String) Possible values: Directly, PrivateLink, SshTunnel, ProxyAgent.
 - `region` (String) Data processing location. This is where Fivetran will operate and run computation on data.
 - `service` (String) The destination type id within the Fivetran system.
 - `setup_status` (String) Destination setup status.

docs/data-sources/local_processing_agent.md

@@ -0,0 +1,38 @@
+---
+page_title: "Data Source: fivetran_local_processing_agent"
+---
+
+# Data Source: fivetran_local_processing_agent
+
+This data source returns a local processing agent object.
+
+## Example Usage
+
+```hcl
+data "fivetran_local_processing_agent" "local_processing_agent" {
+    id = "local_processing_agent_id"
+}
+```
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Required
+
+- `id` (String) The unique identifier for the local processing agent within your account.
+
+### Read-Only
+
+- `display_name` (String) The unique name for the local processing agent.
+- `group_id` (String) The unique identifier for the Group within the Fivetran system.
+- `registered_at` (String) The timestamp of the time the local processing agent was created in your account.
+- `usage` (Attributes Set) (see [below for nested schema](#nestedatt--usage))
+
+<a id="nestedatt--usage"></a>
+### Nested Schema for `usage`
+
+Read-Only:
+
+- `connection_id` (String) The unique identifier of the connection associated with the agent.
+- `schema` (String) The connection schema name.
+- `service` (String) The connection type.

docs/data-sources/local_processing_agents.md

@@ -0,0 +1,44 @@
+---
+page_title: "Data Source: fivetran_local_processing_agents"
+---
+
+# Data Source: fivetran_local_processing_agents
+
+This data source returns a list of all local processing agents within your Fivetran account.
+
+## Example Usage
+
+```hcl
+data "fivetran_local_processing_agents" "local_processing_agents" {
+}
+```
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Read-Only
+
+- `items` (Attributes Set) (see [below for nested schema](#nestedatt--items))
+
+<a id="nestedatt--items"></a>
+### Nested Schema for `items`
+
+Required:
+
+- `id` (String) The unique identifier for the local processing agent within your account.
+
+Read-Only:
+
+- `display_name` (String) The unique name for the local processing agent.
+- `group_id` (String) The unique identifier for the Group within the Fivetran system.
+- `registered_at` (String) The timestamp of the time the local processing agent was created in your account.
+- `usage` (Attributes Set) (see [below for nested schema](#nestedatt--items--usage))
+
+<a id="nestedatt--items--usage"></a>
+### Nested Schema for `items.usage`
+
+Read-Only:
+
+- `connection_id` (String) The unique identifier of the connection associated with the agent.
+- `schema` (String) The connection schema name.
+- `service` (String) The connection type.

docs/resources/connector.md

@@ -70,6 +70,7 @@ resource "fivetran_connector" "amplitude" {
 - `auth` (Block, Optional) (see [below for nested schema](#nestedblock--auth))
 - `config` (Block, Optional) (see [below for nested schema](#nestedblock--config))
 - `destination_schema` (Block, Optional) (see [below for nested schema](#nestedblock--destination_schema))
+- `local_processing_agent_id` (String) The local processing agent ID that refers to the controller created for the group the connection belongs to. If the value is specified, the system will try to associate the connection with an existing agent.
 - `networking_method` (String) Possible values: Directly, PrivateLink, SshTunnel, ProxyAgent.
 - `proxy_agent_id` (String) The proxy agent ID.
 - `run_setup_tests` (Boolean) Specifies whether the setup tests should be run automatically. The default value is TRUE.

docs/resources/destination.md

@@ -44,6 +44,8 @@ resource "fivetran_destination" "dest" {
 
 - `config` (Block, Optional) (see [below for nested schema](#nestedblock--config))
 - `daylight_saving_time_enabled` (Boolean) Shift my UTC offset with daylight savings time (US Only)
+- `local_processing_agent_id` (String) The local processing agent ID that refers to the controller created for the group the connection belongs to. If the value is specified, the system will try to associate the connection with an existing agent.
+- `networking_method` (String) Possible values: Directly, PrivateLink, SshTunnel, ProxyAgent.
 - `run_setup_tests` (Boolean) Specifies whether the setup tests should be run automatically. The default value is TRUE.
 - `timeouts` (Block, Optional) (see [below for nested schema](#nestedblock--timeouts))
 - `trust_certificates` (Boolean) Specifies whether we should trust the certificate automatically. The default value is FALSE. If a certificate is not trusted automatically, it has to be approved with [Certificates Management API Approve a destination certificate](https://fivetran.com/docs/rest-api/certificates#approveadestinationcertificate).

docs/resources/local_processing_agent.md

@@ -0,0 +1,51 @@
+---
+page_title: "Resource: fivetran_local_processing_agent"
+---
+
+# Resource: fivetran_local_processing_agent
+
+This resource allows you to create, update, and delete local processing agents.
+
+## Example Usage
+
+```hcl
+resource "fivetran_local_processing_agent" "test_lpa" {
+    provider = fivetran-provider
+
+    display_name = "display_name"
+    group_id = "group_id"
+}
+```
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Required
+
+- `display_name` (String) The unique name for the local processing agent.
+- `group_id` (String) The unique identifier for the Group within the Fivetran system.
+
+### Optional
+
+- `authentication_counter` (Number) Determines whether re-authentication needs to be performed.
+
+### Read-Only
+
+- `auth_json` (String) Base64-encoded content of the auth.json file.
+- `config_json` (String) Base64-encoded content of the config.json file.
+- `docker_compose_yaml` (String) Base64-encoded content of the compose file for the chosen containerization type.
+- `id` (String) The unique identifier for the local processing agent within your account.
+- `registered_at` (String) The timestamp of the time the local processing agent was created in your account.
+- `usage` (Attributes Set) (see [below for nested schema](#nestedatt--usage))
+
+<a id="nestedatt--usage"></a>
+### Nested Schema for `usage`
+
+Required:
+
+- `schema` (String) The connection schema name.
+- `service` (String) The connection type.
+
+Read-Only:
+
+- `connection_id` (String) The unique identifier of the connection associated with the agent.

fivetran/framework/core/core.go

@@ -86,7 +86,6 @@ type SchemaField struct {
 	Sensitive      bool
 
 	DefaultString string
-
 	Readonly    bool
 	Description string
 }

fivetran/framework/core/model/connector.go

@@ -35,6 +35,7 @@ type ConnectorDatasourceModel struct {
 
     ProxyAgentId            types.String `tfsdk:"proxy_agent_id"`
     NetworkingMethod        types.String `tfsdk:"networking_method"`
+    LocalProcessingAgentId  types.String `tfsdk:"local_processing_agent_id"`
 
 	Status types.Object `tfsdk:"status"`
 
@@ -123,6 +124,7 @@ type ConnectorResourceModel struct {
 
     ProxyAgentId            types.String `tfsdk:"proxy_agent_id"`
     NetworkingMethod        types.String `tfsdk:"networking_method"`
+    LocalProcessingAgentId  types.String `tfsdk:"local_processing_agent_id"`
 
 	Config   types.Object   `tfsdk:"config"`
 	Auth     types.Object   `tfsdk:"auth"`
@@ -193,6 +195,12 @@ func (d *ConnectorResourceModel) ReadFromContainer(c ConnectorModelContainer, fo
 	d.CreatedAt = types.StringValue(c.CreatedAt)
 	d.GroupId = types.StringValue(c.GroupId)
 	d.Service = types.StringValue(c.Service)
+    
+    if c.LocalProcessingAgentId != "" {
+        d.LocalProcessingAgentId = types.StringValue(c.LocalProcessingAgentId)
+    } else {
+        d.LocalProcessingAgentId = types.StringNull()
+    }
 
 	d.DestinationSchema = getDestinationSchemaValue(c.Service, c.Schema)
 
@@ -222,6 +230,12 @@ func (d *ConnectorDatasourceModel) ReadFromContainer(c ConnectorModelContainer)
 	d.CreatedAt = types.StringValue(c.CreatedAt)
 	d.GroupId = types.StringValue(c.GroupId)
 	d.Service = types.StringValue(c.Service)
+    
+    if c.LocalProcessingAgentId != "" {
+        d.LocalProcessingAgentId = types.StringValue(c.LocalProcessingAgentId)
+    } else {
+        d.LocalProcessingAgentId = types.StringNull()
+    }
 
 	d.DestinationSchema = getDestinationSchemaValue(c.Service, c.Schema)
 
@@ -255,6 +269,7 @@ type ConnectorModelContainer struct {
 
     ProxyAgentId            string
     NetworkingMethod        string
+    LocalProcessingAgentId  string
 
 	Config map[string]interface{}
 
@@ -280,6 +295,10 @@ func (c *ConnectorModelContainer) ReadFromResponseData(data connectors.DetailsRe
     if data.NetworkingMethod != "" {
         c.NetworkingMethod = data.NetworkingMethod
     }
+
+    if data.LocalProcessingAgentId != "" {
+        c.LocalProcessingAgentId = data.LocalProcessingAgentId
+    }
 }
 
 func getDestinatonSchemaForConfig(serviceId, nameAttr, tableAttr, prefixAttr attr.Value) (map[string]interface{}, error) {

fivetran/framework/core/model/destination.go

@@ -12,6 +12,8 @@ type destinationModel interface {
 	SetTimeZonOffset(string)
 	SetSetupStatus(string)
 	SetDaylightSavingTimeEnabled(bool)
+    SetLocalProcessingAgentId(string)
+    SetNetworkingMethod(string)
 	SetConfig(map[string]interface{})
 }
 
@@ -23,5 +25,7 @@ func readFromResponse(d destinationModel, resp destinations.DestinationDetailsBa
 	d.SetSetupStatus(resp.SetupStatus)
 	d.SetTimeZonOffset(resp.TimeZoneOffset)
 	d.SetDaylightSavingTimeEnabled(resp.DaylightSavingTimeEnabled)
+	d.SetLocalProcessingAgentId(resp.LocalProcessingAgentId)
+	d.SetNetworkingMethod(resp.NetworkingMethod)
 	d.SetConfig(config)
 }

fivetran/framework/core/model/destination_datasource_model.go

@@ -15,6 +15,8 @@ type DestinationDatasourceModel struct {
 	TimeZoneOffset				 types.String `tfsdk:"time_zone_offset"`
 	SetupStatus    				 types.String `tfsdk:"setup_status"`
 	DaylightSavingTimeEnabled	 types.Bool   `tfsdk:"daylight_saving_time_enabled"`
+    LocalProcessingAgentId       types.String `tfsdk:"local_processing_agent_id"`
+    NetworkingMethod             types.String `tfsdk:"networking_method"`
 	Config        				 types.Object `tfsdk:"config"`
 }
 
@@ -41,7 +43,18 @@ func (d *DestinationDatasourceModel) SetSetupStatus(value string) {
 func (d *DestinationDatasourceModel) SetDaylightSavingTimeEnabled(value bool) {
 	d.DaylightSavingTimeEnabled = types.BoolValue(value)
 }
-
+func (d *DestinationDatasourceModel) SetLocalProcessingAgentId(value string) {
+    if value != "" {
+        d.LocalProcessingAgentId = types.StringValue(value)
+    } else {
+        d.LocalProcessingAgentId = types.StringNull()
+    }
+}
+func (d *DestinationDatasourceModel) SetNetworkingMethod(value string) {
+    if value != "" {
+        d.NetworkingMethod = types.StringValue(value)
+    }
+}
 func (d *DestinationDatasourceModel) SetConfig(value map[string]interface{}) {
 	if d.Service.IsNull() || d.Service.IsUnknown() {
 		panic("Service type is null. Can't handle config without service type.")

fivetran/framework/core/model/destination_resource_model.go

@@ -18,6 +18,8 @@ type DestinationResourceModel struct {
 	DaylightSavingTimeEnabled	 types.Bool   	`tfsdk:"daylight_saving_time_enabled"`
 	Config  					 types.Object   `tfsdk:"config"`
 	Timeouts					 timeouts.Value `tfsdk:"timeouts"`
+    LocalProcessingAgentId       types.String   `tfsdk:"local_processing_agent_id"`
+    NetworkingMethod             types.String   `tfsdk:"networking_method"`
 
 	RunSetupTests    			 types.Bool 	`tfsdk:"run_setup_tests"`
 	TrustCertificates			 types.Bool 	`tfsdk:"trust_certificates"`
@@ -47,6 +49,18 @@ func (d *DestinationResourceModel) SetSetupStatus(value string) {
 func (d *DestinationResourceModel) SetDaylightSavingTimeEnabled(value bool) {
 	d.DaylightSavingTimeEnabled = types.BoolValue(value)
 }
+func (d *DestinationResourceModel) SetLocalProcessingAgentId(value string) {
+    if value != "" {
+        d.LocalProcessingAgentId = types.StringValue(value)
+    } else {
+        d.LocalProcessingAgentId = types.StringNull()
+    }
+}
+func (d *DestinationResourceModel) SetNetworkingMethod(value string) {
+    if value != "" {
+        d.NetworkingMethod = types.StringValue(value)
+    }
+}
 func (d *DestinationResourceModel) SetConfig(value map[string]interface{}) {
 	if d.Service.IsNull() || d.Service.IsUnknown() {
 		panic("Service type is null. Can't handle config without service type.")

fivetran/framework/core/model/local_processing_agent.go

@@ -0,0 +1,35 @@
+package model
+
+import (
+    "github.com/fivetran/go-fivetran/local_processing_agent"
+)
+
+type localProcessingAgentModel interface {
+    SetId(string)
+    SetDisplayName(string)
+    SetGroupId(string)
+    SetRegisteredAt(string)
+    SetConfigJson(string)
+    SetAuthJson(string)
+    SetDockerComposeYaml(string)
+    SetUsage([]localprocessingagent.LocalProcessingAgentUsageDetails)
+}
+
+func readLocalProcessingAgentFromResponse(d localProcessingAgentModel, resp localprocessingagent.LocalProcessingAgentDetailsResponse) {
+    d.SetId(resp.Data.Id)
+    d.SetDisplayName(resp.Data.DisplayName)
+    d.SetGroupId(resp.Data.GroupId)
+    d.SetRegisteredAt(resp.Data.RegisteredAt)
+    d.SetUsage(resp.Data.Usage)
+}
+
+func readLocalProcessingAgentFromCreateResponse(d localProcessingAgentModel, resp localprocessingagent.LocalProcessingAgentCreateResponse) {
+    d.SetId(resp.Data.Id)
+    d.SetDisplayName(resp.Data.DisplayName)
+    d.SetGroupId(resp.Data.GroupId)
+    d.SetRegisteredAt(resp.Data.RegisteredAt)
+    d.SetConfigJson(resp.Data.Files.ConfigJson)
+    d.SetAuthJson(resp.Data.Files.AuthJson)
+    d.SetDockerComposeYaml(resp.Data.Files.DockerComposeYaml)
+    d.SetUsage(nil)
+}