New Func backend: First iteration

[byakuren-hijiri]

Closes #526
Closes #619 (FunC grammar & parser by @novusnota)

Main roadmap

1. Write AST/formatter, and utilities to generate and emit Func code.
2. Rewrite the existing codegen using the API created on the step 1.
   It should be rewritten entirely at this step, since everything is interconnected in the old backend, and there are no any unit tests checking any specific functionality of the generated code.
3. Provide differential testing between two backends achieving good code coverage for the new backend.
   The previous backend is basically untested. It requires to write a number of Tact contracts and setup code coverage to ensure that all the implemented features of the new backend work in the same way as in the old one.
4. Refactor the new backend and remove the old backend.

Sub-roadmap of @novusnota

The goal is to create a comprehensive FunC grammar and parsed using the Ohm parsing toolkit. Should cover FunC version 0.4.4:

• #pragma ...
  • Grammar (grammar.ohm)
  • Semantic actions (semantics.addOperation)
  • Unit tests (positive and negative)
• #include ...
  • Grammar
  • Semantic actions
  • Unit tests
• global ...
  • Grammar
  • Semantic actions
  • Unit tests
• const ...
  • Grammar
  • Semantic actions
  • Unit tests
• Identifiers & Types (unexpectedly tough!)
  • Grammar
  • Semantic actions
  • Unit tests
• Comments and literals
  • Grammar
  • Semantic actions
  • Unit tests
• Functions (declarations & definitions, including asm ones):
  • Grammar
  • Semantic actions
  • Unit tests
• Statements
  • Grammar
  • Semantic actions
  • Unit tests
• Expressions (just like in C++'s parser, with all its precedences matched!)
  • Grammar
  • Semantic actions
  • Unit tests
• Done?
• (maybe) Add tests with .fc files from ton-blockchain/ton monorepo

Current progress

• FunC syntax tree (AST + comments)
• FunC syntax iterators
• FunC formatter (we can skip unit tests for now, as it will be tested as a part of the Tact compiler pipeline)
  • Support JSON dumps for debugging purposes
  • Support indentation
  • Support parens
• New FunC generators (based on src/generators/writers/write*.ts; similar code structure if possible)
  • Expression
  • Statement
  • Function
  • Literal
  • Contract
  • Interfaces
  • Primitives
  • Constant
• Rewrite utilities from the current codegen (WriteContext)
  • Type utilities (src/generators/resolveFunc.*.ts)
  • ABI helpers
  • Routers
  • Receivers
  • Getters
  • Serializers
  • Accessors
  • Stdlib
  • Init serializers
  • Storage functions
  • Static functions
  • Extenstions
  • "used" identifiers
• Write the dependency management used when emitting code from WriteContext. The existing structure based on .context/.used calls is too imperative and makes the source code unreadable. Thus, a better design should be developed.
• Write a brief description of the structure of the generated Func code and the high-level overview of the architecture of the codegen (in e.g. src/codegen/README.md)
• Merge with main incorporating upstream changes to the new backend:
  • fix(codegen): allocator not accounting for extra ref in a cell resulting in cell overflows in some cases #615
• Provide differential testing of two backends
  • Write new unit tests if needed Write unit tests for the codegen, since we don't have any
  • Provide functions for differential testing of two backends
  • Configure the fuzzer to support both backends
• Refactor the new backend
  • Change names and code structure inherited from the original backend
  • Extracts methods where possible; as for now it is unreadable
• Remove the previous backend

Notes

• Running the new backend: yarn build && NEW_CODEGEN=1 PRINT_FUNC=1 ./bin/tact --config <project.config.json>. It will print the generated source in stdout. Unset NEW_CODEGEN to run the same with the old backend.
• Running the differential test: yarn test src/codegen/codegen.spec.ts
• All the elements in the AST have the FuncAst prefix. When creating a separate packge, we could remove the Func part; and add it only to our project. This could be done creating a separate module that includes index.ts with public exports like: export { AstComment as FuncAstComment } from '@func-package-name';

---

• I have updated CHANGELOG.md
• I have documented my contribution in Tact Docs: https://github.com/tact-lang/tact-docs/pull/PR-NUMBER
• I have added tests to demonstrate the contribution is correctly implemented: this usually includes both positive and negative tests, showing the happy path(s) and featuring intentionally broken cases
• I have run all the tests locally and no test failure was reported
• I have run the linter, formatter and spellchecker
• I did not do unrelated and/or undiscussed refactorings

[jubnzv]

A summary of our recent conversation with @anton-trunov:

• This PR should be merged iteratively. The main blocker is that the old backend is not sufficiently covered by tests. Thus, it is not realistic to provide good differential testing to ensure everything works as expected in the new backend.
• We should use functions from syntaxConstructors to map Tact expressions/statements to FunC (writeFunction.ts and writeExpression.ts). Other FunC functions implicitly introduced by the backend should be parsed as a standard library using the FunC parser. Using AST generator functions there makes the code less readable and maintainable.
• The new backend should be merged and maintained at the same time as the old backend. However, we need to enable it in tact.config.json, not with an environment variable as we do now.
• We could use diff in differential tests on the artifacts generated by the compiled code, not on the FunC itself.

Some goals we would like to achieve in the new backend later:

• Some code generation functions might be optimized to reduce code size and gas consumption.
• Refactoring to make it more maintainable.

Thus, I suggest limiting the scope of this backend to the following:

• FunC parser
• Test suite for differential testing
• New backend, which is essentially a copy of the old one, implementing the following features:
  • syntaxConstructors.ts
  • An option in the configuration file to choose the backend
  • Generators of implicit code ported to the new parser:
    • writeStdlib.ts
    • writeRouter.ts
  • Generators that use syntaxConstructors.ts to emit source code
    • Expressions
    • Statements
    • Functions
  • Unit tests for diff testing that cover the implemented logic

After implementing this, I suggest to merge this PR and add more functionality to it iteratively, covering it with tests.

[novusnota]

However, we need to enable it in tact.config.json, not with an environment variable as we do now.

How about adding the new experimental option for this. Like, newFuncBackend:

{
  "projects": [
    {
      "name": "some_prefix",
      "path": "./contract.tact",
      "output": "./contract_output",
      "options": {
        "experimental": {
          "newFuncBackend": true
        }
      }
    }
  ]
}

P.S.: Whoops, a miss-click — didn't mean to close the PR, sorry!