Missing pieces to improve Teal's DX

[mtdowling]

Teal has a ton of potential to become the TypeScript of Lua, but there are a few developer experience gaps that I think are currently holding it back. I'll raise a few here, but feel free to raise other ideas too. Please let me know if I'm just not aware of a solution for things I'm listing too.

1. It's unclear the best way to package and distribute Teal

I've raised this previously, and I think it's still a bit of a rough edge. We definitely do want our .tl files in rocks so that developers using our libraries get autocomplete and type checking.

I've published a few Teal/Lua packages to LuaRocks now, but it's tedious to have to maintain rockspecs where you need to write out each .lua file and then each .tl file manually. Sure, it's no big deal when you have a single file package, but it balloons for multi-module packages.

build = {
  type = "builtin",
  modules = {
    ["a"] = "src/a.lua",
    ["b"] = "src/b.lua",
    ["c"] = "src/c.lua",
    ["d"] = "src/d.lua",
  },
  install = {
    lua = {
      ["a"] = "src/a.tl",
      ["b"] = "src/b.tl",
      ["c"] = "src/c.tl",
      ["d"] = "src/d.tl",
    },
  },
}

Ideally there could be a plugin to LuaRocks to manage all this automatically and also include Teal files in the rock for those that use Teal and want autocomplete. luarocks-build-cyan exists, but doesn't do include Teal, and I'm not sure if it's usable at the moment. Maybe this is the solution, and the project just needs some help finishing it up? If so, any notes on the vision and what it should do would be great.

2. I don't think the norm should be to commit generated .lua files to repos

The standard practice at the moment for Teal repositories seems to be to commit both the .tl files and .lua files into VCS. However, I think we should find a way to not have to do this because:

1. It makes it harder to reason about diffs because you have to look at .lua files and .tl files. A newcomer to the repo may not know that they can just skip reviewing the .lua files.
2. It becomes a bit of a quirky requirement for Teal packages where you need to commit the Teal code you actually edited, and then also commit the Lua code you generated. (and then you probably also have to remind people to do this in PRs).
3. It makes navigating a repo harder since you double up on files. This isn't a huge deal with single file packages, but gets annoying when you have a big multi-module package with lots of files.

3. Providing a pure Lua version of a Teal library

So, if we don't commit .lua files into Teal repos, then we need to figure out how we can easily provide a pure .lua solution for developers that don't use Teal. With LuaRocks, this doesn't really matter since the rock could contain the .tl and .lua files. However, many Lua developers, particularly those that work with game engines like Love2D, just want to copy and paste Lua code into their projects.

For my own projects, I was thinking I could maybe make a GitHub release that has a few zip files, one of which is the Lua version of the library. Installation would be as simple as downloading and unzipping. This would be automated on every release. I'm sure some kind of open source tool could be made to share the code too.

Alternatively, maybe there's a way to have LuaRocks somehow manage this and put the generated files in a specific directory, and it would just a matter of documentating how to get the Lua code.

I'm just totally making this up and inventing the idea of "variants" and specifying a custom local dir, but:

luarocks install emitter.tl --variant lua --local-dir path/to/lib/emitter

4. We need a way to integrate teal-types with build tools

We need a solution for how libraries and applications can rely on types from teal-types and similar repos (see this related issue).

LuaCATS has a similar types repo, and you can pull in specific parts of it using an addon manager of their VS code plugin. The addon manager seems to modify VS Code's settings.json to tell their LSP about library types.

For Teal, I wonder if we could:

1. Add something to Cyan to define dependencies on specific entries of teal-types.
2. Cyan checks out the repo, grabs the relevant packages, and puts them in your module path.
3. The Teal LSP hopefully gets this for free because Cyan managed it for the LSP.
4. I'm not sure if there's a LuaRocks story here, though there might need to be... For example, I'm trying to build a sort of game engine on top of Love2D and it requires some of the LuaJIT and Love2D definitions. Do I embed them in my library, or state something unfortunate in my project's README like "you need to copy and paste these specific files into your project" or "you need to paste these lines into your tlconfig.lua.".

5. Testing Teal libraries needs some work

Writing tests for a library that uses Teal still has some papercuts. Some libraries are simply just writing tests in Lua. But if we're writing Teal libraries, the ideal experience is to write tests in Teal.

busted-tl works pretty well, but it hasn't shipped a stable release yet. This means you can't install it using normal luarocks commands and need to install it as a one-off using the CLI.

test_dependencies = {
  "busted",
  "busted-tl", -- < doesn't work
}

To get it to work, you have to manually run this in a Makefile or something similar.

luarocks install --local --lua-version=5.1 --server=https://luarocks.org/dev busted-tl

(let me know if there's some trick I can use in my rockspec to make this work).

Given how ubiquitous busted it for testing, maybe if Cyan were able to run tests for a package, it automatically wires up busted-tl?

6. We need a documentation generator

Lua has ldoc. Ldoc is great for Lua code, but teaching it how to generate docs for Teal might be stretching it too far. I opened an issue to see if anyone else has looked at Teal support.

It looks like Luau has a few dedicated documentation generators.

LuaCATs has documentation generation built into their LSP plugin.

Just like how TypeScript has dedicated documentation generators separate from JavaScript, we need something for Teal too.

7. We need to beef up Cyan

Cyan can handle compiling a Teal project, but I still find myself needing to use Makefiles to manage tasks that Cyan doesn't handle yet. Things like installing LuaRocks dependencies, running tests, making a new rock, etc. I've opened a few issues on the repo for these already, just mentioning them for completeness.

8. Other stuff

I'm sure there's a bunch of other stuff to talk about (e.g., we need a code formatter, would be nice to have the ability to add custom linters to the compiler, etc), but I'll stop here :)

[catwell]

Hello @mtdowling, it is great that you created this thread although if we want to discuss each thing in detail it could become a bit unwieldy.

Responding to your points one by one:

1. It's unclear the best way to package and distribute Teal

I agree with you 100%. I discussed this with @hishamhm at last FOSDEM. Given that Teal and LuaRocks have the same author I think there should just be first class support for Teal packages in LuaRocks.

2. I don't think the norm should be to commit generated .lua files to repos
3. Providing a pure Lua version of a Teal library

It probably depends on the project a bit. In general I think the .src.rock should contain both sets of source files (luarocks should not build tl -> lua at install time) so it could potentially be used as the reference for people who want to copy Lua files.

4. We need a way to integrate teal-types with build tools

Once LuaRocks has proper Teal support I think this should be dealt with at LuaRocks-level, i.e. if a user installs a Lua package in a Teal project and that package does not provide its own .d.tl files, it should have the option to fallback to teal-types.

5. Testing Teal libraries needs some work

Same here, I think luarocks test should just support Teal.

I don't use Busted for my projects, I have my own test helper that I have converted to Teal, but I agree Busted should be the default.

6. We need a documentation generator

Not real opinion, I used LDoc a lot with Lua and even contributed to it a bit (I am still a maintainer technically). With typed languages I rely on docstrings a lot less (same in Python by the way) so I do not use a documentation generator anymore.

7. We need to beef up Cyan

You may have noticed from my responses earlier but my feeling is most of those features should become part of LuaRocks rather than Cyan.

I think it is important to have good tooling (package manager, linter, etc) that is "centralized", similar to what Astral is doing in the Python world. I love Astral tools as a user but something I do not like that the tools are written in Rust. I think most of the performance gains could be kept with a Python codebase (and maybe a few modules in a lower-level language). The DX is super important though, especially that it is distributed as a single binary.

In Lua, I would like the centerpiece of that tooling to be LuaRocks. If it does not gain features like this people will keep launching other projects etc, you may have seen this recently which is in Rust and very reminiscent of Astral. I would prefer we do not go this route and provide a good DX for Lua and especially Teal in LuaRocks while we can.

[mtdowling]

Teal support in LuaRocks: Yeah, Teal support in LuaRocks would definitely solve a lot of the problems I listed. Would love Hisham's take on this.

Testing: I'll check out your library! wrt to Busted, I think the DX is improved a lot if busted-tl cuts a normal release on LuaRocks. I think that would solve 5 well enough.

Build tooling: It seems like the most interesting conversation we should all have is about the vision for build tooling for Teal. Given that luarocks already has a test command, and it has the unique ability to install test dependencies before running tests, it does make a lot of sense to move that there. It also would make sense to somehow leverage it to install teal-types dependencies.

[hishamhm]

Given that luarocks already has a test command, and it has the unique ability to install test dependencies before running tests

And it is also extensible in the same way as build.type, so a new test.type could be created with supercharged powers too.

[Aire-One]

Hello @mtdowling

busted-tl works pretty well, but it hasn't shipped a stable release yet.

I'm happy to read it's working for you!

Busted-tl was some kind of experimentation to write tests with teal directly, as opposed to running busted with the Lua generated files.

To allow a better experience, we need to solve issues #532. As long as we can't redefine the standard library with Teal, luassert type definitions are useless. It means only the basic assert() function is usable for tests and the developer experience is degraded.

Given how ubiquitous busted it for testing, maybe if Cyan were able to run tests for a package, it automatically wires up busted-tl?

While it could work, I don't think tests/busted should be managed by Cyan. It seems out of scope to me.

The way forward should be to submit the loader I wrote to busted directly. But I was seeking more tests and feedback before going to upstream.

[mtdowling]

Yes, it works great, and thank you for writing it!

I think your point about seeing if it will be accepted upstream makes sense too. Honestly, if there was a release of busted-tl and I could install it with a rockspec, then I think the DX of using Busted and Teal is good enough. We could maybe make a guide or something in the Teal wiki.

wrt to Busted typing and #532, I've just been using luassert directly instead of their overloaded assert.X stuff. IMO, I think this is good enough for Teal to have a good enough testing DX (though I could see how this hurt users migrating their Lua code to Teal).

[hishamhm]

@Aire-One as for #532: do you think having a flag such as --std mycustomstd.d.tl (and equivalent in the tl module API) would be sufficient?

[Aire-One]

@hishamhm yes, that would work. My implementation in busted-tl currently uses tl.load(content, filename) internally to provide the runner to busted. So, I guess both implementations (as a flag and API) should be required:

• The CLI flag would be used by the end-user to lint with the tl CLI.
• The API would be used by busted-tl to load the test file.

I guess cyan and other tools would also benefit from the API.

[hishamhm]

A while ago I ported the compiler's built-in standard library definition from a data structure to a source string that is "tl.loaded", also with the intent of making it easier to implement its replacement (and also potentially --no-std). This shouldn't be hard to implement now. It's not at the top of my proverbial to-do list, so I'll gladly take a PR if someone beats me to it!

[hishamhm]

Hi @mtdowling ! All of these points are super valid! I think I'll reply to each point on a separate post to make threading easier (also for others to jump in)

1. It's unclear the best way to package and distribute Teal

luarocks-build-cyan exists, but doesn't do include Teal, and I'm not sure if it's usable at the moment.

Oh, it doesn't? I think I thought that by pulling in cyan it would pull in tl. In any case, this can be fixed. luarocks-build-cyan was something I put together quickly as a PoC/starting point but it did work when I tried it (perhaps because I already had the dependencies in?). In any case, it should be fixable. Any love sent its way is welcome!

But I think what everyone wants is first-class Teal support in LuaRocks (as per @catwell's comment). I refrained myself from adding it at first precisely because I'm the maintainer of both (and there are other languages out there which compile to .lua) but I think at this point no one would complain.

The LuaRocks repo is very much "in flux" right now so I'm not sure if we'll get a major release adding features very soon, so I guess getting build.type = "cyan" working the best it can would be a better short-term bet (and it would be instantly backwards compatible with existing LuaRocks releases in package distros, since non-bundled build.type entries automatically pull the equivalent luarocks-build-* build dependency. We could then pull luarocks-build-cyan into LuaRocks 4 if it helps streamlining things, but perhaps that might not even be necessary in the end.

[hishamhm]

luarocks-build-cyan exists, but doesn't do include Teal, and I'm not sure if it's usable at the moment.

luarocks-build-cyan does depend on cyan which does depend on tl -- so not sure what problems you experienced?

[mtdowling]

I should have been more clear. From what I understand, luarocks-build-cyan doesn't automatically include .tl files in the rock, like the lua section here:

build = {
  type = "builtin",
  modules = {
    ["a"] = "src/a.lua",
  },
  install = {
    lua = {
      ["a"] = "src/a.tl",
    },
  },
}

And yes, Teal support in LuaRocks would be amazing!

[hishamhm]

2. I don't think the norm should be to commit generated .lua files to repos

This was something I did early on as a "bootstrap" measure since nobody had Teal-aware environments in their Lua projects in the beginning. Ultimately that's something that's up to each project to decide so we can move way from that if we feel it's time. I guess point 2 depends on point 1, as in, how easy it is to get runnable .lua modules out of your .tl files.

For the Teal compiler itself, to avoid chicken-and-egg situations, I think I still want to ship a tl.lua file. But then I want also to eventually split the compiler into multiple files, so tl.lua might end up becoming an auto-generated amalgamation of multiple modules for ease of deployment in odd environments (people have been running Teal in all sorts of places, including within scripting environments of games where they don't have a CLI!).

[hishamhm]

4. We need a way to integrate teal-types with build tools

Yes. The boundary gets a bit confusing between Cyan and LuaRocks there, and I remember discussing this with @euclidianAce in some issue in the past.

Even if LuaRocks handles the dependency tracking / file management side of this, Cyan will need to be aware of it. My personal idealized view of this is that I'd like to see tl as rustc and cyan as cargo, but that implies Cyan becoming both the build manager (which it is) and package manager (which it isn't). If LuaRocks functioned well as a library (which it currently doesn't), then the package management bits in Cyan could be only a frontend to LuaRocks without actually implementing the dep tracking / file mgt parts. I think that would be the ideal DX.

As a first step, we could make luarocks-build-cyan to be teal-types aware. That could work... 🤔

Another technical problem related to teal-types is that the repo itself does not account for versioning. (Mea culpa: I set it up this way, for simplicity, and based on the design of other similar repos such as the ones for Python and TypeScript.) I think this could be fixed if it becomes a practical problem (nobody ever complained about versioning mismatches between libraries in the LR tree and teal-types definitions), but I thought I'd raise it here as a known thing.

[hishamhm]

3. Providing a pure Lua version of a Teal library

Alternatively, maybe there's a way to have LuaRocks somehow manage this and put the generated files in a specific directory, and it would just a matter of documentating how to get the Lua code.

LuaRocks almost-sorta-kinda can do that if you make a very custom configuration file that flattens the output deployment tree as much as possible (see the --tree flag to control output and the deploy_lua_dir key in luarocks config), but it would probably still output both the .tl and .lua files.

[hishamhm]

6. We need a documentation generator

There is currently a Google Summer of Code project open for that. I'm not sure if we'll have a student working on it this summer, but if we do we might end up getting this tool!

[hishamhm]

@mtdowling btw, a quick heads up: I emailed you earlier this week (pinging here in case my email fell in the spam folder)

[jeremytenjo]

A Teal formatter is essential.

Please create it :)

[alerque]

Adding Teal support to StyLua would be the best way to do this. Feel free to send a PR ;-) I think you'd need to start with adding support to full-moon. There is an existing feature request upstream so future discussion can probably continue there.

[jeremytenjo]

I would love to help if I had experience writing Rust. I've decided to use roblox-ts instead.