[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468

emptyOVO · 2024-11-07T13:14:22Z

Motivation

add dolphinscheduler package in org.apache.inlong.manager.schedule
add Client and Engine for DS, and Utils for operating open-API of DS
add pojo class for DS interaction
add UT, provide image env for test cases, mock DS for inlong to test schedule ability

Modifications

Because DS officially does not provide SDK to call, only provides openAPI call mode, so all scheduling behavior is based on open-api requests

when inlong enabled the DS schedule mode, a project in DS will be initialized to handle processes
when inlong starts a offline schedule job for a inlong-group, it will create a workflow process definition in project, scheduled by info, with a script periodically sends requests to call back to inlong
when unregistered, it offline the process running on DS and delete it

Provides some disaster recovery logic

After inlong shutdown and restart in unexpected situations, the workflow process data will recovered and continue work, prevent duplicate data generation and data loss
The latest ScheduleInfo is enabled during repeated registration to avoid data redundancy
ConcurrentHashMap to store schedule data ensure thread safety during start, running or stop

How to use

Specify the url and token in the configuration file

# DolphinScheduler related config
inlong.schedule.dolphinscheduler.url=
inlong.schedule.dolphinscheduler.token=

inlong manager performs dependency injection at startup via the @value annotation

Configure the DolphinScheduleEngine when it is initialized

DolphinScheduleEngine dolphinScheduleEngine = new DolphinScheduleEngine(INLONG_DS_TEST_ADDRESS, INLONG_DS_TEST_PORT, INLONG_DS_TEST_USERNAME, INLONG_DS_TEST_PASSWORD, DS_URL, DS_TOKEN);

then use .start() or .stop() to manage DS engine

Verifying this change

How is the token generated?
DS url
for example：http://{ip}:{port}/dolphinscheduler
testRegisterScheduleInfo

the callback request was successfully sent to inlong, and the related tasks are processed in flink

log info:

2024-11-05 21:43:03.104 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.104 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.111 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296993905792
2024-11-05 21:43:03.154 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296993945728
2024-11-05 21:43:03.167 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.183 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=0, scheduleUnit=S, scheduleInterval=2, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=null, version=null)
2024-11-05 21:43:03.199 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true
2024-11-05 21:43:03.203 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.204 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.209 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:144 - Process definition exists, process definition id: 124296993945728, deleting...
2024-11-05 21:43:03.237 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296994033792
2024-11-05 21:43:03.262 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296994056320
2024-11-05 21:43:03.275 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.290 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=1, scheduleUnit=null, scheduleInterval=null, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=*/1 * * * * ?, version=null)
2024-11-05 21:43:03.306 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true

testUnregisterScheduleInfo

log info:

2024-11-05 21:43:03.314 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.314 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.319 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:144 - Process definition exists, process definition id: 124296994056320, deleting...
2024-11-05 21:43:03.356 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296994155648
2024-11-05 21:43:03.368 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296994165888
2024-11-05 21:43:03.382 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.397 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=0, scheduleUnit=S, scheduleInterval=2, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=null, version=null)
2024-11-05 21:43:03.412 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true
2024-11-05 21:43:03.413 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:184 - Dolphin Scheduler handle Unregister begin for test-group
2024-11-05 21:43:03.413 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:185 - Checking process definition id uniqueness...
2024-11-05 21:43:03.417 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:190 - Deleting process definition, process definition id: 124296994165888
2024-11-05 21:43:03.445 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:195 - Process definition deleted
2024-11-05 21:43:03.445 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:198 - Un-registered dolphin schedule info for test-group
2024-11-05 21:43:03.445 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.445 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.451 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296994252928
2024-11-05 21:43:03.460 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296994260096
2024-11-05 21:43:03.475 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.491 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=1, scheduleUnit=null, scheduleInterval=null, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=*/1 * * * * ?, version=null)
2024-11-05 21:43:03.508 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true
2024-11-05 21:43:03.508 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:184 - Dolphin Scheduler handle Unregister begin for test-group
2024-11-05 21:43:03.508 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:185 - Checking process definition id uniqueness...
2024-11-05 21:43:03.512 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:190 - Deleting process definition, process definition id: 124296994260096
2024-11-05 21:43:03.540 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:195 - Process definition deleted
2024-11-05 21:43:03.540 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:198 - Un-registered dolphin schedule info for test-group

testUpdateScheduleInfo

2024-11-05 21:43:03.541 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.541 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.547 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296994352256
2024-11-05 21:43:03.554 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296994357376
2024-11-05 21:43:03.569 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.586 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=0, scheduleUnit=S, scheduleInterval=2, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=null, version=null)
2024-11-05 21:43:03.601 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true
2024-11-05 21:43:03.601 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:212 - Update dolphin schedule info for test-group
2024-11-05 21:43:03.601 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:184 - Dolphin Scheduler handle Unregister begin for test-group
2024-11-05 21:43:03.601 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:185 - Checking process definition id uniqueness...
2024-11-05 21:43:03.606 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:190 - Deleting process definition, process definition id: 124296994357376
2024-11-05 21:43:03.635 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:195 - Process definition deleted
2024-11-05 21:43:03.635 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:198 - Un-registered dolphin schedule info for test-group
2024-11-05 21:43:03.635 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:136 - Dolphin Scheduler handle register begin for test-group
2024-11-05 21:43:03.636 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:137 - Checking process definition id uniqueness...
2024-11-05 21:43:03.641 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:153 - Generate task code for process definition success, task code: 124296994448512
2024-11-05 21:43:03.648 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:158 - Create process definition success, process definition code: 124296994453632
2024-11-05 21:43:03.663 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:161 - Release process definition success, release status: ONLINE
2024-11-05 21:43:03.678 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:164 - Create schedule for process definition success, schedule info: ScheduleInfo(id=null, inlongGroupId=test-group, scheduleType=1, scheduleUnit=null, scheduleInterval=null, startTime=2024-11-05 21:43:03.0, endTime=2024-11-05 21:43:13.0, delayTime=null, selfDepend=null, taskParallelism=null, crontabExpression=*/1 * * * * ?, version=null)
2024-11-05 21:43:03.695 - INFO [ main] .i.m.s.d.DolphinScheduleEngine:167 - Online schedule for process definition, status: true

(Please pick either of the following options)

This change is a trivial rework/code cleanup without any test coverage.
This change is already covered by existing tests, such as:
(please describe tests)
This change added tests and can be verified as follows:

(example:)
- Added integration tests for end-to-end deployment with large payloads (10MB)
- Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

aloyszhang · 2024-11-08T10:19:02Z

Nice work!
Overall LGTM, just one thing, does dolphinscheduler only support crontab schedule type?

emptyOVO · 2024-11-08T10:38:00Z

Nice work! Overall LGTM, just one thing, does dolphinscheduler only support crontab schedule type?

Yes, natively, only Crontab expressions are accepted for task scheduling. And i also check DS related implements, their underlying implementation were similar to inlong's built-in engine, uses quartz for scheduling，the difference is that DS only use TriggerBuilder in quartz for parsing crontab expressions and create trigger, unlike that used in inlong which can create trigger based on scheduling units and scheduling intervals

...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java

...ava/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleContainerTestEnv.java

inlong-manager/manager-web/src/main/resources/application-dev.properties

inlong-manager/manager-web/src/main/resources/application-prod.properties

inlong-manager/manager-web/src/main/resources/application-test.properties

...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java

.../src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleUtils.java

...ule/src/main/java/org/apache/inlong/manager/schedule/exception/DolphinScheduleException.java

inlong-manager/manager-schedule/pom.xml

.../src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleUtils.java

github-actions bot added the component/manager label Nov 7, 2024

emptyOVO force-pushed the INLONG-11401 branch 6 times, most recently from bd7fb5d to 6bf84a2 Compare November 8, 2024 08:10

[INLONG-11401][Manager] Support Dolphinscheduler schedule engine

b0b99c4

emptyOVO force-pushed the INLONG-11401 branch from 55c7d24 to b0b99c4 Compare November 8, 2024 08:49

aloyszhang assigned emptyOVO Nov 8, 2024

fix: change boundary type to TIME

47d38a3

aloyszhang previously approved these changes Nov 10, 2024

View reviewed changes

dockerzhang previously approved these changes Nov 12, 2024

View reviewed changes

fuweng11 reviewed Nov 12, 2024

View reviewed changes

Merge branch 'master' into INLONG-11401

893f755

emptyOVO dismissed stale reviews from dockerzhang and aloyszhang via be574d1 November 13, 2024 08:48

fix: exception handle, default configure value

44da2e2

emptyOVO force-pushed the INLONG-11401 branch from be574d1 to 44da2e2 Compare November 13, 2024 08:54

fix: use jackson instead of gson

a9e606a

fuweng11 reviewed Nov 14, 2024

View reviewed changes

.../src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleUtils.java Outdated Show resolved Hide resolved

fuweng11 reviewed Nov 14, 2024

View reviewed changes

...ule/src/main/java/org/apache/inlong/manager/schedule/exception/DolphinScheduleException.java Show resolved Hide resolved

emptyOVO mentioned this pull request Nov 17, 2024

[Feature][Manager] Support schedule engine configuration #11504

Open

2 tasks

feat: improve exception handling, test with dependency injection

dafbafe

emptyOVO force-pushed the INLONG-11401 branch from 5aec945 to dafbafe Compare November 17, 2024 14:57

vernedeng reviewed Nov 18, 2024

View reviewed changes

inlong-manager/manager-schedule/pom.xml Outdated Show resolved Hide resolved

fix: use properties

5fe79a1

fuweng11 reviewed Nov 18, 2024

View reviewed changes

fix: exception format

b10b560

fuweng11 previously approved these changes Nov 18, 2024

View reviewed changes

dockerzhang previously approved these changes Nov 18, 2024

View reviewed changes

fix: configure settings

1f0c99e

emptyOVO dismissed stale reviews from dockerzhang and fuweng11 via 1f0c99e November 18, 2024 09:41

fuweng11 previously approved these changes Nov 19, 2024

View reviewed changes

aloyszhang previously approved these changes Nov 19, 2024

View reviewed changes

dockerzhang previously approved these changes Nov 20, 2024

View reviewed changes

Merge branch 'master' into INLONG-11401

e5ded2e

dockerzhang dismissed stale reviews from aloyszhang, fuweng11, and themself via e5ded2e November 20, 2024 01:47

dockerzhang approved these changes Nov 20, 2024

View reviewed changes

aloyszhang approved these changes Nov 20, 2024

View reviewed changes

dockerzhang merged commit 042e568 into apache:master Nov 20, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468

[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468

emptyOVO commented Nov 7, 2024 •

edited

Loading

aloyszhang commented Nov 8, 2024

emptyOVO commented Nov 8, 2024 •

edited

Loading

[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468

[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468

Conversation

emptyOVO commented Nov 7, 2024 • edited Loading

Motivation

Modifications

Verifying this change

Documentation

aloyszhang commented Nov 8, 2024

emptyOVO commented Nov 8, 2024 • edited Loading

emptyOVO commented Nov 7, 2024 •

edited

Loading

emptyOVO commented Nov 8, 2024 •

edited

Loading