The following report summarizes my work done during Google Summer of Code 2022 with the Apache Software Foundation.
- Name: Xin Huang (中文: 黄鑫)
- Github Username: isHuangXin
- Mentor: Raigor Jiang
- Organization: The Apache Software Foundation
- Project Title: Develop an external tool to convert YAML configuration into DistSQL scripts
The ecosystem to transform any database into a distributed database system.
Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.
Since version 5.0.0, shardingSphere provides its own management language: DistSQL, which greatly facilitates users to manage distributed databases. There are now many users who want to convert from legacy YAML configuration to DistSQL, and we want to design and implement a command line tool that allows the user to enter a path to a YAML configuration file and output a DistSQL script.
More details:https://shardingsphere.apache.org/document/current/en/reference/distsql/syntax/rdl/resource-definition/add-resource/
Develop an external tool to convert YAML configuration into DistSQL scripts. I have built a prototype for this proposal: DistSQL RDL, e.g.ADD and CREATE.It has the function of reading YAML files, parsing the content, and then using DistSQL RDL commands to output DistSQL scripts.
A pototype of converting Yaml file to DistSQL script.
All the DistSQL convert functions and test cases I need to finish.
| ADD RESOURCE DATASOURCE | TEST CASE |
| CREATE SHARDING TABLE RULE | TEST CASE |
| CREATE SHARDING BINDING TABLE RULE | TEST CASE |
| CREATE SHARDING BROADCAST TABLE RULE | TEST CASE |
| CREATE SHARDING ALGORITHM | TEST CASE |
| CREATE SHARDING STRATEGY | TEST CASE |
| CREATE SHARDING KEY GENERATOR | TEST CASE |
| CREATE READWRITE_SPLITTING RULE | TEST CASE |
| CREATE DB_DISCOVERY RULE | TEST CASE |
| CREATE ENCRYPT RULE | TEST CASE |
| CREATE SHADOW RULE | TEST CASE |
- Reading the yaml file, parsing its content and outputing the DistSQL RDL script, e.g. ADD, Create.
- Wrap DistSQL RDL syntax into a command-line tool (ConvertYamlConfigurationHandler). This allows converting yaml to DistSQL without starting Proxy
- Add new corresponding Yaml files in Convert folder and expected converted DistSQL in Expected floder.
- Run ConvertYamlConfigurationHandlerTest to make sure there are no exceptions.
DistSQL Convert Process
Includes Community Issues and PRs.
- ⊙ #16915: Issues: Apply for this project in Discussions
Includes PRs and Issues related to GSoC project
- ⊙ #17939: [ GSoC 2022 ] Develop an external tool to convert YAML configuration into DistSQL scripts
- ✔ #19280: Support
ADD RESOURCE DATASOURCEDistSQL Convert - ✔ #19509: Support
CREATE SHARDING TABLE RULEDistSQL Convert - ✔ #19509: Support
CREATE SHARDING BINDING TABLE RULEDistSQL Convert - ✔ #20843: Support
CREATE SHARDING BROADCAST TABLE RULEDistSQL Convert - ✔ #19509: Support
CREATE SHARDING ALGORITHMDistSQL Convert - ✔ #19509: Support
CREATE SHARDING STRATEGYDistSQL Convert - ✔ #19509: Support
CREATE SHARDING KEY GENERATORDistSQL Convert - ✔ #20648: Support
CREATE READWRITE_SPLITTING RULEDistSQL Convert - ✔ #20707: Support
CREATE DB_DISCOVERY RULEDistSQL Convert - ✔ #20748: Support
CREATE ENCRYPT RULEDistSQL Convert - ✔ #20843: Support
CREATE SHADOW RULEDistSQL Convert
Intended Goals were met during the standard coding period, DistSQL Convert for the Yaml files were achieved.
- Example: the yaml file needs to be converted
Yaml File
- Example: the converted DistSQL script
Converted DistSQL
Now that the project of Developing an external tool to convert YAML configuration into DistSQL scripts is completed. Next, I will focus on refactoring the ConvertYamlConfiguration.java to achieve the mixed configuration of DistSQL.
The past 3 months have challenged me to learn and grow. I had a fruitful summer coding and interacting with the Apache ShardingSphere community. This experience made me interested in open source projects, and I will continue contributing solutions and code to the shardingsphere via participating in WeOpen-Star 2022 (by Tencent).
And finally, I also would like to show my gratitude to my mentor Raigor Jiang for guiding me throughout this journey and Google for organizing such a beneficial program.