Plan

[Samyak2]

Summer of Code Weekly Plans

Week 1 Plan

• New name #2
• Select an SQL parser #3
• Setup Rust project #4
• Design data structures for the data store according to the design
• Start implementing the most basic CREATE DATABASE instruction

[tyt2y3]

Can you first write the data structures for the data store?
i.e. Database -> Schema -> Table ?

[Samyak2]

Good idea at this stage. I have updated the plan.

[Samyak2]

Week 1 update

Progress made this week

• Signed copyright doc
• Opened issues for week 1 plans (Plan #5)
• Selected a parser for SQL - sqlparser-rs (Select an SQL parser #3)
• Setup a Rust project with a temporary directory structure and sqlparser as dependency. The files had empty structs and enums (feat: setup Rust project and directory structure #6)
• Designed data structures for data storage (wip: data structures #8)
  • Designed the hierarchy of Database -> Schema -> Table
  • Columns are represented by a name and the sqlparser::ast::DataType of it -> avoids re-writing all possible types again.
  • A row is vec of ColumnKind which is an enum of Types and the data stored inside it.
    • This means that a few sqlparser::ast::DataTypes will be mapped to exactly one variant of ColumnKind and the storage for them will be the same.
    • The storage could be optimized by using highly specific variants for each type, but that would increase complexity for performance. Since performance is a non-goal, this is a good trade-off.
  • The VirtualMachine keeps a HashMap of registers.
    • A register can contain a Table or a Filter applied on a Table.
    • The VM executes an IntermediateCode which is just a vec of Instructions.
    • The design of the instructions is yet to be done.
  • Tests are placeholders for now as there's no functionality yet.
• Since there wasn't any actual functionality implemented yet, I would consider this to be a part of the thinking or design phase - although at a code level.

Learnings:

• I need to provide better deadlines and scope the tasks better - one of the main tasks for this week was unbounded and did not have a concrete goal. Chris gave a better idea - designing the data structures - that I went ahead with.
• Prefer using to_owned() instead of to_string() when converting from a &str to a String -> Thank you Sanford for this.
• Explored some of sqlparser

Next steps:

• Add doc strings for all the data structures (structs and enums) and methods.
• Design the Instruction set.

[tyt2y3]

Week 2 Plan

• Get feat: setup Rust project and directory structure #6 and wip: data structures #8 reviewed and merged
• Design the instruction set #9
• Add doc strings for the data structures implemented in wip: data structures #8

[Samyak2]

Week 2 Update

Progress made this week

• Got feat: setup Rust project and directory structure #6 and wip: data structures #8 into main and squashed commits to make history nicer.
• Designed the instruction set
  • Took into consideration CREATE, SELECT, INSERT, UPDATE, ALTER and JOINs.
  • GROUP BY and aggregates were missed out.
• Improved data structures based on feedback, added some very basic tests and added doc strings (Data structures improvements #11)
• Started working on implementing the instruction set (instruction set #12)
  • Implemented data structures for a Register

Learnings

• Explored quite a bit of MySQL to see what kind of queries it supports.
  • Learnt about all the different types of JOINs possible and how MySQL supports them.
  • Learnt to read MySQL's grammar - it used a variant of EBNF
• Explored a few other Databases written in Rust to see how they implement nullable Value among other things.
• Learnt how a new type instead of a built-in type helps when it has a specific semantic meaning.

[Samyak2]

Week 3 Plan

• Complete implementing the instruction set with all data structures (Design the instruction set #9 and instruction set #12)
• Think of how GROUP BY and aggregates can be supported in the instruction set
• sqlparser integration and code generation #13

[tyt2y3]

Yeah actually the MySQL documentation is awesome. I remember reading it a lot when I first wrote SeaQuery. And reading it a second time when writing SeaSchema

[Samyak2]

Yeah actually the MySQL documentation is awesome. I remember reading it a lot when I first wrote SeaQuery. And reading it a second time when writing SeaSchema

The documentation is pretty good. Although, in this particular case of JOINs, I prefer the PostgreSQL version. It explains clearly what types of JOINs are possible and the difference between them. In contrast, the MySQL version reads more like a list of quirks with JOINs in MySQL. To be clear, it also provides the same information as the PostgreSQL version, but it's much more scattered and harder to read.

[Samyak2]

Week 4 Plan

• continue to get feedback on and improve instruction set #12
• sqlparser integration and code generation #13
• Setup CI #7

[Samyak2]

Week 5 Plan

• Code generation from AST (part of sqlparser integration and code generation #13)
• Setup CI #7