Move all stuff to monorepo?

[dafanasiev]

Is your feature request related to a problem? Please describe.
Feature request

Describe the solution you'd like
haraka/Haraka depends on many other haraka/* packages (plugins, core...). Its not easy to maintain all of them (and packages deps).
I thinks that monorepo can help with that. Using this solution plugins will be easy to debug and develop.

Describe alternatives you've considered

• npm workspaces
• yarn workspaces
• lerna

Additional context
For example: I try to make PoC (I use master for all haraka/* packages):

$> npm --version
9.6.4
$> node --version
v20.0.0
$> npm i

added 86 packages, and audited 925 packages in 6s

74 packages are looking for funding
  run `npm fund` for details

13 vulnerabilities (6 moderate, 5 high, 2 critical)

$> npm outdated
Package                 Current  Wanted  Latest  Location                                                    Depended by
@elastic/elasticsearch   7.17.0  7.17.0   8.7.0  node_modules/@elastic/elasticsearch                         haraka-plugin-elasticsearch@1.0.8
ldapjs                    1.0.2   1.0.2   3.0.2  node_modules/ldapjs                                         haraka-plugin-auth-ldap@1.0.3
ldapjs                    1.0.2   1.0.2   3.0.2  node_modules/ldapjs                                         haraka-plugin-rcpt-ldap@1.0.1
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-net-utils/node_modules/mocha                haraka-net-utils@1.5.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-aliases/node_modules/mocha           haraka-plugin-aliases@1.0.1
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-karma/node_modules/mocha             haraka-plugin-karma@2.1.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-limit/node_modules/mocha             haraka-plugin-limit@1.1.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-recipient-routes/node_modules/mocha  haraka-plugin-recipient-routes@1.2.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-redis/node_modules/mocha             haraka-plugin-redis@2.0.5
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-spf/node_modules/mocha               haraka-plugin-spf@1.2.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-watch/node_modules/mocha             haraka-plugin-watch@2.0.2
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-plugin-uribl/node_modules/mocha             haraka-plugin-uribl@1.0.7
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-tld/node_modules/mocha                      haraka-tld@1.1.0
mocha                     9.2.2   9.2.2  10.2.0  packages/haraka-utils/node_modules/mocha                    haraka-utils@1.0.3
nodeunit-x               0.15.0  0.15.0  0.16.0  node_modules/nodeunit-x                                     haraka@npm:Haraka@3.0.1
nopt                      7.0.0   7.0.0   7.1.0  node_modules/nopt                                           haraka@npm:Haraka@3.0.1
nopt                      7.0.0   7.1.0   7.1.0  node_modules/nopt                                           haraka-plugin-spf@1.2.0
redis                     4.5.1   4.5.1   4.6.5  node_modules/redis                                          haraka@npm:Haraka@3.0.1
redis                     4.1.1   4.1.1   4.6.5  packages/haraka-plugin-karma/node_modules/redis             haraka-plugin-karma@2.1.0
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          haraka-plugin-limit@1.1.0
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          haraka-plugin-redis@2.0.5
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          haraka-plugin-recipient-routes@1.2.0
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          haraka-plugin-watch@2.0.2
redis                     4.5.1   4.6.5   4.6.5  node_modules/redis                                          haraka-results@2.2.2
semver                    7.3.8   7.3.8   7.5.0  node_modules/semver                                         haraka@npm:Haraka@3.0.1

[msimerson]

Short answer: been there, done that, no thanks.

Haraka used to be a monorepo. Although some things were better in a monorepo, many things were not. Things that I find worse in a monorepo:

• CI build & test times. All dependencies must be installed for every tiny change. No, I really don't want to have to install ldapjs when all I did was a a doc fix in this far-from-ldap module.
• Same with email-message. It's now the one haraka lib that loads iconv, a slow dependency to build, so every single change in the Haraka monorepo caused minutes of extra CI build times. Days of developer time were wasted fiddling with CI caches (with also have edge cases) to work around that. The solution was simple: repackage it in its own dependency, with its own tests. That made building the main Haraka project much faster.
• with discrete packages, we can iterate in them and fire off releases much faster. Releases in Haraka were much harder before and the pressure to make sure everything was perfect (which it never is) was higher. It took weeks to make changes in the monorepo, and for some things, years. For example, we're still using the long since deprecated nodeunit in the main Haraka repo. Most of the repackaged modules are using a modern and supported test runner. It's quick, easy, and not very risky to try that in a small library. It's so much work to do that in a large monorepo that years later it's still not done.
• some types of changes (lint, refactoring, es6, etc.) were really really unwieldy in the monorepo. Way too many changes all at once, takes a long time to review, and there's always an exception to every general rule. We've got enough code that when you apply a "simple" change 300 times, it inevitably breaks several somethings. That imposes a heavy burden on code reviewers to try and catch those needles in the haystack.
• Revisions. If a change in a lower level Haraka module breaks a build or causes a problem, switch versions. No need to wait for releases or monkey patches.