Skip to content

Commit

Permalink
Merge pull request #347 from cookpad/nekketsuuu-import
Browse files Browse the repository at this point in the history
Deprecate SynchronizeDataSources
  • Loading branch information
nekketsuuu authored Apr 18, 2024
2 parents 79b165b + a0ab72f commit e40f570
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 29 deletions.
38 changes: 9 additions & 29 deletions app/batches/README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,12 @@
# Batch Jobs

For synchronization of database information from data source.
The behavior differs greatly depending on whether you use SynchronizeDataSources or not.
All batches except SynchronizeDataSources are new beta batches added to improve batch operations.

## SynchronizeDataSources

Synchronize all data from all data sources.
IgnoredTables and MskedData will work.
If the number of schemas and tables to be synchronized is large, it will take a long time to synchronize.
Until v0.8.1, the target schema was dependent on the search_path in PostgreSQL and Redshift. Starting from the next version, search_path will no longer be used. Synchronize all visible schemas by a user account of Dmemo access.


## Import Batches

This is a new batch that replaces SynchronizeDataSources. New batches have been created to reduce the time spent on fetching and to make it easier to manage the schemas to be fetched.
**This is a beta version and is currently in trial.**
IgnoredTables and MskedData work. Only schemas that have SchemaMemo and are linked in Dmemo will be imported.
If you don't want to include them, the Dmemo administrator can edit the DataSource and unlink it.
Since we have created batches for each granularity, we need to specify the target with an argument for each granularity.

### For Examples
```
$bundle exec rails runner 'ImportDataSourceDefinitions.run("DWH”)’
$bundle exec rails runner 'ImportTableDefinitions.run("DWH", “sample_schema", “target_table”)'
Schemas, tables, and columns can be imported using `ImportDataSourceDefinition` job. Please run the job periodically.
Note that only linked schemas are imported by the job. The settings can be configured by admins.

```console
$ bundle exec rails runner 'ImportDataSourceDefinitions.run("DWH”)'
$ # If you want to import just a specific table:
$ bundle exec rails runner 'ImportTableDefinitions.run("DWH", “sample_schema", “target_table”)'
$ # Or if you want to import all schemas:
$ bundle exec rails runner 'SynchronizeDefinitions.run'
```

## SynchronizeDefinitions

Run `ImportDataSourceDefinitions` for all data sources.
1 change: 1 addition & 0 deletions app/batches/synchronize_data_sources.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
class SynchronizeDataSources
def self.run
Rails.logger.warn 'SynchronizeDataSources is deprecated. Please consider using ImportDataSourceDefinitions instead.'
DataSource.all.find_each do |data_source|
import_data_source!(data_source)
end
Expand Down

0 comments on commit e40f570

Please sign in to comment.