Split implementation into modules

[hishamhm]

This is a long-awaited development; as we get more people contributing, it's about time we split the code into multiple modules.

I must admit I do find editing a single huge file comfortable, but I obviously understand that splitting the code in modules has other advantages, including making it easier for people to grasp the overall structure more easily, and also to exercise multi-module project structure in the compiler itself.

I still want to be able to have a single-file redistributable version of the compiler, so I'm splitting it into a teal.* namespace, and I plan to make tl.tl become an auto-generated single-file version.

Steps involved:

• Begin splitting some parts into modules, require sub-modules into main tl.tl
• Adapt Makefile for multi-file bootstrap development workflow, so we can start running using multiple modules, build them incrementally and bootstrap-test them (it's ugly but it works)
• Finish splitting all parts into modules, require sub-modules into main teal/init.lua
• Adapt tl to use split modules instead
• Adapt extras/binary.sh generator for multi-file structure
• Adapt extras/release.sh for multi-file structure
• Make a simple generator for the single-file amalgamation
• Write some developer documentation on project structure and dev tools

Closes #549.

[github-actions]

Teal Playground URL: https://1003--teal-playground-preview.netlify.app

[mbartlett21]

For a generator for a single file, you should be able to do something like the below:

Details

local module_to_files = {
   ['teal.attributes']              = 'teal/attributes.lua',
   ['teal.checker.checker']         = 'teal/checker/checker.lua',
   ['teal.checker.file_checker']    = 'teal/checker/file_checker.lua',
   ['teal.checker.require_file']    = 'teal/checker/require_file.lua',
   ['teal.checker.string_checker']  = 'teal/checker/string_checker.lua',
   ['teal.checker.type_checker']    = 'teal/checker/type_checker.lua',
   ['teal.debug']                   = 'teal/debug.lua',
   ['teal.environment']             = 'teal/environment.lua',
   ['teal.errors']                  = 'teal/errors.lua',
   ['teal.facts']                   = 'teal/facts.lua',
   ['teal.gen.lua_generator']       = 'teal/gen/lua_generator.lua',
   ['teal.lexer']                   = 'teal/lexer.lua',
   ['teal.parser']                  = 'teal/parser.lua',
   ['teal.precompiled.default_env'] = 'teal/precompiled/default_env.lua',
   ['teal.traversal']               = 'teal/traversal.lua',
   ['teal.types']                   = 'teal/types.lua',
   ['teal.type_errors']             = 'teal/type_errors.lua',
   ['teal.type_reporter']           = 'teal/type_reporter.lua',
   ['teal.util']                    = 'teal/util.lua',
   ['teal.variables']               = 'teal/variables.lua',
}

local keys = {}
for k in pairs(module_to_files) do table.insert(keys, k) end
table.sort(keys)

-- populate the preload field so that we leave worrying about dependency order
-- to Lua
local content = {}
for _, k in ipairs(keys) do
   local ke = ('%q'):format(k)
   local fn = module_to_files[k]
   local f = assert(io.open(fn, 'rb'))
   local modcontent = assert(f:read'a')
   assert(f:close())

   table.insert(content,
string.format([[
-- module %s from %s
package.preload[%q] = function(...)
%s
end

]], k, fn, k, modcontent)
   )
end

do
   local f = assert(io.open('tl.lua', 'rb'))
   local modcontent = assert(f:read'a')
   assert(f:close())
   table.insert(content, modcontent)
end

local f = assert(io.open('tl_combined.lua', 'wb'))
assert(f:write(table.concat(content)))
assert(f:close())

[hishamhm]

@mbartlett21 I pulled in your generator into extras/! I'll change it to produce a tl.lua instead of tl_combined.tl once I no longer have one in the top-level (I just need to move out the high-level API functions now, the split is almost ready).

[hishamhm]

And here is my shot at a new API for Teal!

Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> teal = require("teal")
> compiler = teal.compiler()
> input = compiler:input([[record R end; function R.func(s:string) print(s:upper()) end; return R]])
> module, errs = input:check()
> module
nil
> require("tabular")(errs)
┌───────────────┬───────────────────────────────────────────────────────────────────────────────────┐
│syntax_errors :│┌──────────┬──────────────────────────────────────────────────────────────────┬─┬─┐│
│               ││filename  │msg                                                               │x│y││
│               ││────────  │───                                                               │─│─││
│               ││<input>.tl│record needs to be declared with 'local record' or 'global record'│1│1││
│               │└──────────┴──────────────────────────────────────────────────────────────────┴─┴─┘│
│type_errors ..:│┌┐                                                                                 │
│               │└┘                                                                                 │
│warnings .....:│┌┐                                                                                 │
│               │└┘                                                                                 │
└───────────────┴───────────────────────────────────────────────────────────────────────────────────┘
> input = compiler:input([[local record R end; function R.func(s:string) print(s:upper()) end; return R]])
> module = assert(input:check())
> module:gen()
local _tl_compat; if (tonumber((_VERSION or ''):match('[%d.]*$')) or 0) < 5.3 then local p, m = pcall(require, 'compat53.module'); if p then _tl_compat = m end end; local string = _tl_compat and _tl_compat.string or string; local R = {}; function R.func(s) print(s:upper()) end; return R

Bonus stage!: I ported tl (the CLI driver) to Teal! \o/

This was very useful in test-driving the API. @euclidianAce I also kept an eye on how Cyan uses the "v2" API, but I didn't go as far as attempt to port it.

[hishamhm]

Rebased on top of the latest changes. @mbartlett21 the test updates in #1027 were a lifesaver for the rebase process!

[hishamhm]

Merged manually!