refactor(fivetran_sdk): restructuring of the repo to make it consiste…

…nt with Production release (#89)

* remove v1 and restructuring of v2

* proto import change

* script change

* doc change

* doc change

* Update audit.yml (#90)

---------

Co-authored-by: Satvik Patil <[email protected]>
Co-authored-by: Rishabh Ghosh <[email protected]>

.github/workflows/audit.yml

@@ -32,19 +32,19 @@ jobs:
       run: python -m pip install --upgrade pip setuptools wheel
 
     - name: Install Python connector requirements
-      working-directory: v1/examples/source_connector/python
+      working-directory: examples/source_connector/python
       run: pip install -r requirements.txt
 
     - name: Run Python connector build script
-      working-directory: v1/examples/source_connector/python
+      working-directory: examples/source_connector/python
       run: ./build.sh
 
     - name: Install Python destination requirements
-      working-directory: v1/examples/destination_connector/python
+      working-directory: examples/destination_connector/python
       run: pip install -r requirements.txt
 
     - name: Run Python destination build script
-      working-directory: v1/examples/destination_connector/python
+      working-directory: examples/destination_connector/python
       run: ./build.sh
 
     - name: Set up Go
@@ -58,45 +58,45 @@ jobs:
         version: "23.2"
 
     - name: Set up protobuf for Go
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
 
     - name: Set up protoc-gen-go
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
 
     - name: Set up Go protoc path
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: export PATH="$PATH:$(go env GOPATH)/bin"
 
     - name: Verify dependencies
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: go mod verify
 
     - name: Build Go connector
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: scripts/build.sh
 
     - name: Run go vet
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: go vet ./...
 
     - name: Install staticcheck
       run: go install honnef.co/go/tools/cmd/staticcheck@latest
 
     - name: Run staticcheck
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: staticcheck ./...
 
     - name: Install golint
       run: go install golang.org/x/lint/golint@latest
 
     - name: Run golint
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: golint ./...
 
     - name: Run tests
-      working-directory: v1/examples/source_connector/golang
+      working-directory: examples/source_connector/golang
       run: go test -race -vet=off ./...
 
     - name: Setup Gradle
@@ -105,17 +105,17 @@ jobs:
         gradle-version: 8.5
 
     - name: Run Java connector copyProtos
-      working-directory: v1/examples/source_connector/java
+      working-directory: examples/source_connector/java
       run: gradle copyProtos
 
     - name: Run Java connector test with Gradle Wrapper
-      working-directory: v1/examples/source_connector/java
+      working-directory: examples/source_connector/java
       run: gradle build
 
     - name: Run Java destination copyProtos
-      working-directory: v1/examples/destination_connector/java
+      working-directory: examples/destination_connector/java
       run: gradle copyProtos
 
     - name: Run Java destination test with Gradle Wrapper
-      working-directory: v1/examples/destination_connector/java
-      run: gradle build
+      working-directory: examples/destination_connector/java
+      run: gradle build

.gitignore

@@ -5,20 +5,13 @@
 **/build/
 **/golang/main
 bin/*
-v1/*/**/*.proto
-v1/**/*.pb.go
-v1/**/*pb2.py
-v1/**/*pb2.pyi
-v1/**/*pb2_grpc.py
 destination_run/
 connector_run/
 **/__pycache__/
-v1/**/node_modules/
-v1/**/package-lock.json
-v2/*/**/*.proto
-v2/**/*.pb.go
-v2/**/*pb2.py
-v2/**/*pb2.pyi
-v2/**/*pb2_grpc.py
-v2/**/node_modules/
-v2/**/package-lock.json
+examples/**/*.proto
+examples/**/*.pb.go
+examples/**/*pb2.py
+examples/**/*pb2.pyi
+examples/**/*pb2_grpc.py
+examples/**/node_modules/
+examples/**/package-lock.json

README.md

@@ -3,14 +3,13 @@ A way for partners to create source and destination connectors that run on [Five
 
 ## Repo Structure
 This repo consists of example source and destination connectors along with a local testing environment. Both connectors are written in a [gRPC supported language](https://grpc.io/docs/languages/) that can generate a statically linked binary. We recommend Java, Golang, or Rust. 
-* [V2 Examples](v2/examples)
-* [V1 Examples](v1/examples)
+* [V2 Examples](examples)
 * [Local Testing Tools](tools/)
 
 ## Development
 You can follow the [SDK Development Guide](development-guide.md) for guidance on how to develop your code. 
 
-> NOTE: If you're a new partner that just started working with the Connector SDK, use [V2 protos](v2). We will continue to support [V1 protos](v1) for partners that have used them to build their SDK connectors, and these will be removed once the migration to V2 is complete.
+> NOTE: If you're a new partner that just started working with the Connector SDK, use V2 release. We will continue to support V1 release for partners that have used them to build their SDK connectors, and these will be removed once the migration to V2 is complete.
 
 Once you have your code ready to run:
 1. Start up your connector running on port 50051 (for destination code, use port 50052).

v1/connector_sdk.proto → connector_sdk.proto

@@ -1,13 +1,13 @@
 syntax = "proto3";
 option optimize_for = SPEED;
 option java_multiple_files = true;
-option go_package = "fivetran.com/fivetran_sdk";
-package fivetran_sdk;
+option go_package = "fivetran.com/fivetran_sdk_v2";
+package fivetran_sdk.v2;
 
 import "common.proto";
 
-// Fivetran (grpc client) <> Connector (grpc server)
-service Connector {
+// Fivetran (grpc client) <> SourceConnector (grpc server)
+service SourceConnector {
     rpc ConfigurationForm (ConfigurationFormRequest) returns (ConfigurationFormResponse) {}
     rpc Test (TestRequest) returns (TestResponse) {}
     rpc Schema (SchemaRequest) returns (SchemaResponse) {}
@@ -65,28 +65,12 @@ message TableSelection {
 }
 
 message UpdateResponse {
-    oneof response {
-        LogEntry log_entry = 1;
-        Operation operation = 2;
-    }
-}
-
-enum LogLevel {
-    INFO = 0;
-    WARNING = 1;
-    SEVERE = 2;
-}
-
-message LogEntry {
-    LogLevel level = 1;
-    string message = 2;
-}
-
-message Operation {
-    oneof op {
+    oneof operation {
         Record record = 1;
         SchemaChange schema_change = 2;
         Checkpoint checkpoint = 3;
+        Warning warning = 4;
+        Task task = 5;
     }
 }
 
@@ -100,10 +84,10 @@ message SchemaChange {
 message Record {
     optional string schema_name = 1;
     string table_name = 2;
-    OpType type = 3;
+    RecordType type = 3;
     map<string, ValueType> data = 4;
 }
 
 message Checkpoint {
     string state_json = 1;
-}
+}

v2/destination_connector_sdk_v2.proto → destination_sdk.proto

@@ -5,7 +5,7 @@ option go_package = "fivetran.com/fivetran_sdk_v2";
 package fivetran_sdk.v2;
 
 import "google/protobuf/timestamp.proto";
-import "common_v2.proto";
+import "common.proto";
 
 // Fivetran (grpc client) <> DestinationConnector (grpc server)
 service DestinationConnector {

development-guide.md

@@ -21,7 +21,7 @@ The executable needs to do the following:
 
 * Partners should not add the proto files to their repos. Proto files should be pulled in from this repo at build time and added to `.gitignore` so they are excluded.
 * Always use proto files from latest release and update you're code if necessary. Older releases proto files can be considered deprecated and will be expired at later date.
-* New partners that just started working with the Connector SDK should use [V2 protos](v2). Only partners that have already used [V1 protos](v1) to build SDK connectors should continue using them. 
+* New partners that just started working with the Connector SDK should use V2 release. We will continue to support V1 release for partners that have used them to build their SDK connectors, and these will be removed once the migration to V2 is complete.
 
 
 ### Logging

v2/examples/source_connector/nodejs/src/index.js → examples/source_connector/nodejs/src/index.js

@@ -1,5 +1,5 @@
 const grpc = require("@grpc/grpc-js");
-const PROTO_PATH_CONNECTOR = "./src/protos/source_connector_sdk_v2.proto";
+const PROTO_PATH_CONNECTOR = "./src/protos/connector_sdk.proto";
 var protoLoader = require("@grpc/proto-loader");
 
 const options = {

v2/examples/source_connector/python/main.py → examples/source_connector/python/main.py

@@ -1,5 +1,3 @@
-from sys import api_version
-
 import grpc
 from concurrent import futures
 import json

tools/destination-connector-tester/README.md

@@ -2,7 +2,7 @@
 
 ## Pre-requisites
 - Docker Desktop >= 4.23.0 or [Rancher Desktop](https://rancherdesktop.io/) >= 1.12.1
-- gRPC server is running for the particular example (see [example readme's](/v2/examples/destination_connector/))
+- gRPC server is running for the particular example (see [example readme's](/examples/destination_connector/))
 
 ## How To Run
 
@@ -17,8 +17,6 @@
         docker pull us-docker.pkg.dev/build-286712/public-docker-us/sdktesters-v2/sdk-tester:<version> 
     ```
 
-> NOTE: If using V1 proto versions, use the latest docker image of the [public-docker-us/sdktesters/sdk-tester](https://console.cloud.google.com/artifacts/browse/build-286712/us/public-docker-us/sdktesters%2Fsdk-tester) artifact in Google Artifact Registry.
-
 2. Run a container using the image with the following command. Make sure to map a local directory for the tool by replacing `<local-data-folder>` placeholders in the command, and replace `<version>` with the version of the image you pulled.
 
 ```

tools/source-connector-tester/README.md

@@ -2,7 +2,7 @@
 
 ## Pre-requisites
 - Docker Desktop >= 4.23.0 or [Rancher Desktop](https://rancherdesktop.io/) >= 1.12.1
-- gRPC server is running for the particular example (see [example readme's](/v2/examples/source_connector/))
+- gRPC server is running for the particular example (see [example readme's](/examples/source_connector/))
 
 ## How To Run
 
@@ -17,8 +17,7 @@
     ```
         docker pull us-docker.pkg.dev/build-286712/public-docker-us/sdktesters-v2/sdk-tester:<version>   
     ```
-> NOTE: If using V1 proto versions, use the latest docker image of the [public-docker-us/sdktesters/sdk-tester](https://console.cloud.google.com/artifacts/browse/build-286712/us/public-docker-us/sdktesters%2Fsdk-tester) artifact in Google Artifact Registry.
-
+   
 2. Run a container using the image with the following command. Make sure to map a local directory for storing files that the tool generates by replacing `<local-data-folder>` in the command, and replace <version> with the version of the image you pulled.
 
 ```