forked from DTStack/chunjun
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[doc][sqlserver] add SQLServer en doc
- Loading branch information
Showing
3 changed files
with
728 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,119 @@ | ||
# SqlServer Lookup | ||
|
||
## 1. Introduction | ||
SqlServer Lookup table,full and asynchronous approaches are supported.<br /> | ||
Full cache: Load all dimensional table data into memory, and it is recommended that the amount of data is not used.<br /> | ||
Asynchronous caching: Query data using asynchronous means, and cache the queried data to memory using lru, which is recommended for large amounts of data. | ||
|
||
## 2. Support Version | ||
Microsoft SQL Server 2012 and above | ||
|
||
## 3. Plugin Name | ||
| SQL | sqlserver-x | | ||
| --- | --- | | ||
|
||
## 4. Parameter | ||
|
||
- **connector** | ||
- definition:connection plugin name | ||
- required:required | ||
- data type:String | ||
- value:sqlserver-x | ||
|
||
<br /> | ||
|
||
- **url** | ||
- definition:Use the open source jtds driver connection instead of Microsoft's official driver<br />jdbcUrl Reference documents:[jtds Reference documents](http://jtds.sourceforge.net/faq.html) | ||
- required:required | ||
- data type:String | ||
- default:none | ||
|
||
<br /> | ||
|
||
- **table-name** | ||
- definition:table name | ||
- required:required | ||
- data type:String | ||
- default:none | ||
|
||
<br /> | ||
|
||
- **schema** | ||
- definition:schema | ||
- required:optional | ||
- data type:String | ||
- default:none | ||
|
||
<br /> | ||
|
||
- **username** | ||
- definition:username | ||
- required:required | ||
- data type:String | ||
- default:none | ||
|
||
<br /> | ||
|
||
- **password** | ||
- definition:password | ||
- required:required | ||
- data type:String | ||
- default:none | ||
|
||
<br /> | ||
|
||
- **lookup.cache-type** | ||
- definition:Dimension table cache type(NONE、LRU、ALL),default is LRU | ||
- required:optional | ||
- data type:string | ||
- default:LRU | ||
|
||
<br /> | ||
|
||
- **lookup.cache-period** | ||
- definition:Interval for loading data when the cache type is all, default is 3600000ms(one hour) | ||
- required:optional | ||
- data type:string | ||
- default:3600000 | ||
|
||
<br /> | ||
|
||
- **lookup.cache.max-rows** | ||
- definition:the cache rows of lru lookup table ,default value is 10000 | ||
- required:optional | ||
- data type:string | ||
- default:10000 | ||
|
||
<br /> | ||
|
||
- **lookup.cache.ttl** | ||
- definition:Interval for loading data when the cache type is lru,default value is 60000ms | ||
- required:optional | ||
- data type:string | ||
- default:60000 | ||
|
||
<br /> | ||
|
||
- **lookup.fetch-size** | ||
- definition:the num of data fetched from the oracle table which is used as lookup all table at a time | ||
- required:optional | ||
- data type:string | ||
- default:1000 | ||
|
||
<br /> | ||
|
||
- **lookup.parallelism** | ||
- definition:the parallelism of the lookup table | ||
- required:optional | ||
- data type:string | ||
- default:none | ||
|
||
|
||
|
||
## 5. Data type | ||
|Supported data type | BIT、INT、SMALLINT、TINYINT、BIGINT、INT IDENTITY、REAL、FLOAT、DECIMAL、NUMERIC、CHAR、VARCHAR、VARCHAR(MAX)、TEXT、XML、NCHAR、NVARCHAR、NVARCHAR(MAX)、NTEXT、TIME、DATE、DATETIME、DATETIME2、SMALLDATETIME、DATETIMEOFFSET、TIMESTAMP、BINARY、VARBINARY、IMAGE、MONEY、SMALLMONEY、UNIQUEIDENTIFIER | | ||
| ---| ---| | ||
| Not supported at the moment | CURSOR、ROWVERSION、HIERARCHYID、SQL_VARIANT、SPATIAL GEOMETRY TYPE、SPATIAL GEOGRAPHY TYPE、TABLE | | ||
|
||
## 6. Profile Demo | ||
see `flinkx-examples` directory。 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,254 @@ | ||
# SqlServer Source | ||
|
||
## 1. Introduce | ||
SqlServer Sink support to write data to database SQLServer | ||
|
||
## 2. Version Support | ||
Microsoft SQL Server 2012 and above | ||
|
||
## 3、Connector name | ||
| Sync | sqlserverwriter、sqlserversink | | ||
| --- | --- | | ||
| SQL | sqlserver-x | | ||
|
||
## 4、Parameter description | ||
|
||
### 1.Sync | ||
|
||
- **connection** | ||
- Description:param for Database connection,including jdbcUrl、schema、table and so on | ||
- Required:required | ||
- Type:List | ||
- Default:none | ||
|
||
```json | ||
"connection": [{ | ||
"jdbcUrl": "jdbc:jtds:sqlserver://0.0.0.1:1433;DatabaseName=DTstack", | ||
"table": ["table"], | ||
"schema":"public" | ||
}] | ||
``` | ||
|
||
|
||
|
||
<br /> | ||
|
||
- **jdbcUrl** | ||
- Description:Use the open source jtds driver connection instead of Microsoft's official driver<br />jdbcUrlReference documents:[jtds Reference documents](http://jtds.sourceforge.net/faq.html) | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
|
||
|
||
<br /> | ||
|
||
- **schema** | ||
- Description:Database schema | ||
- Required:optional | ||
- Type:string | ||
- Default: none | ||
<br /> | ||
|
||
- **table** | ||
- Description: sqlserver table name, only support one table in a single work at the moment. | ||
- Required:required | ||
- Type:List | ||
- Default:none | ||
<br /> | ||
|
||
- **username** | ||
- Description: user name | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
- **password** | ||
- Description: password | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
- **column** | ||
- Description:the fields to be written to the destination table,which is separated by English commas.for example: "column": ["id","name","age"] | ||
- Required:required | ||
- Type:List | ||
- Default:none | ||
<br /> | ||
|
||
|
||
<br /> | ||
|
||
- **fullcolumn** | ||
- Description:All fields in the destination table ,which is separated by English commas.for example: "column": ["id","name","age","hobby"],if not configured, it will be obtained in the system table | ||
- Required:optional | ||
- Type:List | ||
- Default:none | ||
<br /> | ||
|
||
- **preSql** | ||
- Description:the sql executed before writing data into the destination table | ||
- Required:optional | ||
- Type:List | ||
- Default:none | ||
<br /> | ||
|
||
<br /> | ||
|
||
- **postSql** | ||
- Description:the sql executed after writing data into the destination table | ||
- Required:optional | ||
- Type:List | ||
- Default:none | ||
<br /> | ||
|
||
<br/> | ||
|
||
- **writeMode** | ||
- Description:the mode of writing data, insert into or merge into | ||
- Required:required | ||
- All options:insert/update | ||
- Type:String | ||
- Default:insert | ||
|
||
<br /> | ||
|
||
- **withNoLock** | ||
- Description:add the sql with(nolock) | ||
- Required:optional | ||
- Type:Boolean | ||
- Default:false | ||
|
||
<br /> | ||
|
||
- **updateKey** | ||
- Description:When the write mode is update, you need to specify the value of this parameter as the unique index field | ||
- attention: | ||
- If this parameter is empty and the write mode is update, the application will automatically obtain the unique index in the database; | ||
- If the data table does not have a unique index, but the required write mode is configured as update and, the application will write data in the way of insert; | ||
- Required:optional | ||
- Type:Map<String,List> | ||
- for example:"updateKey": {"key": ["id"]} | ||
- Default:none | ||
|
||
|
||
<br /> | ||
|
||
- **semantic** | ||
- Description:sink operator support phase two commit | ||
- attention: | ||
-If this parameter is blank, phase two commit is not enabled by default,which means sink operators do not support exactly-once semantics | ||
-Currently only supported exactly-once and at-least-once | ||
- Required:optional | ||
- Type:String | ||
- for example:"semantic": "exactly-once" | ||
- Default:at-least-once | ||
|
||
- **batchSize** | ||
- Description:The number of records submitted in batch at one time. This value can greatly reduce the number of network interactions between flinkx and the database and improve the overall throughput,Setting this value too large may cause the flinkx process to run oom | ||
- Required:optional | ||
- Type:int | ||
- Default:1024 | ||
|
||
|
||
|
||
### 2.SQL | ||
|
||
- **connector** | ||
- Description:connector type | ||
- Required:required | ||
- Type:String | ||
- value:sqlserver-x | ||
|
||
<br/> | ||
|
||
- **url** | ||
- Description:Use the open source jtds driver connection instead of Microsoft's official driver | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
|
||
<br /> | ||
|
||
- **table-name** | ||
- Description: table name | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
- **schema** | ||
- Description:schema | ||
- Required:optional | ||
- Type:String | ||
- Default:none | ||
|
||
<br/> | ||
|
||
- **username** | ||
- Description:username | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
- **password** | ||
- Description:password | ||
- Required:required | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
<br/> | ||
|
||
- **sink.buffer-flush.max-rows** | ||
- Description:Number of data pieces written in batch | ||
- Required:optional | ||
- Type:String | ||
- Default:1024 | ||
<br /> | ||
|
||
|
||
- **sink.buffer-flush.interval** | ||
- Description:Batch write interval,Unit: ms | ||
- Required:optional | ||
- Type:String | ||
- Default:10000 | ||
<br /> | ||
|
||
- **sink.all-replace** | ||
- Description: whether to replace all data in the database | ||
- Required:optional | ||
- Type:String | ||
- Default:false | ||
<br /> | ||
|
||
<br/> | ||
|
||
- **sink.semantic** | ||
- Description:sink operator support phase two commit | ||
- attention: | ||
-If this parameter is blank, phase two commit is not enabled by default,which means sink operators do not support exactly-once semantics; | ||
-Currently only supported exactly-once and at-least-once | ||
- Required:optional | ||
- Type:String | ||
- for example:"semantic": "exactly-once" | ||
- Default:at-least-once | ||
<br /> | ||
|
||
- **sink.parallelism** | ||
- Description:the parallelism of sink operator | ||
- Required:optional | ||
- Type:String | ||
- Default:none | ||
<br /> | ||
|
||
|
||
|
||
## 5. Supported data type | ||
|Supported data type | BIT、INT、SMALLINT、TINYINT、BIGINT、INT IDENTITY、REAL、FLOAT、DECIMAL、NUMERIC、CHAR、VARCHAR、VARCHAR(MAX)、TEXT、XML、NCHAR、NVARCHAR、NVARCHAR(MAX)、NTEXT、TIME、DATE、DATETIME、DATETIME2、SMALLDATETIME、DATETIMEOFFSET、TIMESTAMP、BINARY、VARBINARY、IMAGE、MONEY、SMALLMONEY、UNIQUEIDENTIFIER | | ||
| ---| ---| | ||
| Not supported at the moment | CURSOR、ROWVERSION、HIERARCHYID、SQL_VARIANT、SPATIAL GEOMETRY TYPE、SPATIAL GEOGRAPHY TYPE、TABLE | | ||
## 6. Demo | ||
see details in`flinkx-examples` directory. |
Oops, something went wrong.