language agnostic templates

[hohwille]

Note: This is a visionary feature request that will cause a lot of effort and refactoring. However, I think that it would cause a huge benefit.

It would be awesome if the templates could be compatible with the language they produce. The template-engine could have a grammar configuration for each language that defines how the language syntax is used to express templating logic.
E.g. for Java this would mean:

• Within the code you could have macros like for loops, etc. in comments (see comments below)
• variables in package and classnames are expressed via some silly syntax like x_varname_x instead of ${varname} what would also be allowed in package segments, type names, methods, fields, etc.

UPDATE: initially proposed $_varname_$ or __varname__ but due to strange problems with tools (eclipse, git) we changed to prevent problems.

The new variable syntax with underscores could also express the case-transformation (assuming variable names are then treated case-insensitive):

• x_varname_x for lowercase
• X_VarName_X for PascalCase
• x_varName_x for camlCase
• X_VARNAME_X for UPPERCASE
• X_VAR_NAME_X for CAP_CASE
• x_var-name_x for train-case

(Please note that I am intentionally mixing concepts of velocity and freemarker, because I think both have pros and cons but we need to create something new and better).

• The most tricky part is IMHO to allow writing control statements that are not comments and can be compiled as Java but allow dynamic usage of the templating-engine:
• If you have method cascades such as currently ${var}.getFoo().getBar() and the type of ${var} is also dynamic it can get really tricky. However I assume that getFoo() has to assume at least a common supertype to work on and then you could also do

Supertype /* <#replace-left varType> */ $_var_$ = ...;
$_var_$.getFoo().getBar();

If the type is easy to extend you could also create a subclass X_VarType_X in the CobiGen templates that extends from Supertype and mark it as not being a template so the class itself will be ignored on generation.
It would be nice to make some experiments and prototyping in this area to get more insights.
The same concept would easily work with JS, .NET, PHP, python, ruby, scala, etc.
Using this strange v_var_v approach might look ugly but it would be a very simple way to solve big problems.
Assume you can refactor your CobiGen Templates with Java in a typesafe way...

What I like very much about velocity and miss a lot in freemarker is direct access to java calls:

• foo.bar.some for foo.getBar().getSome()
• foo.myMethod() for foo.myMethod()
  This way we could simply wrap the input object (pojo) into a JavaContainer and provide rich navigations and operations on that and allow simple and native invocations on Class such as getSimpleName() or getName() such obvious things are so very tricky in freemarker.

[hohwille]

Please note that CobiGen could ship a minimal library for java intentionally containing a class V with public static constants such as pojo, etc.
Further it could ship a class Macro with public static methods such out(Object o) and so forth.
This would mean that inside a method block you could write something like:

int sum = 0;
for (Property property : V.pojo.properties) {
  if(property.getter != null) {
    Macro.out("sum +=" + property.getter.name + "();");
  }
}
return sum;

Then the CobiGen Java Plugin that parses the template could understand this as template macro and interpret it while the template can be processed by any java compiler and gets full IDE support with completion etc.
So each statement-block operating on CobiGen specifc types would automatically get interpreted by the template-engine and would be executed and replaced with its output. Maybe there could also be a way to mark code inside as output as currently "sum" is a string (not typesafe) and mixing variables from non-marco code with macro code needs some clear rules and syntax.

[hohwille]

This is so damn cool. Give me some time and a room with food and drinks and let me implement this. I am dying for all this :)
Death to freemarker!

[hohwille]

I did design the API. Have a look at this cool example template to get what I am dreaming of:
https://github.com/hohwille/tools-cobigen/blob/dev_core/cobigen/cobigen-api/src/test/java/__rootpackage__/__component__/common/api/__detail__/to/__EntityName__Eto.java

This is fully compliant Java Code that compiles and can be refactored.
It is IMHO quite expressive and readable. I would assume that you avoid 80% of the mistakes when writing templates for Java code and boost your productivity accordingly.

[hohwille]

A new challenge when implementing this in CobiGen will be that the templates defined as Java have to be unloaded and reloaded in its own classloader realm as they can frequently change.
However, I assume that this already works today when CobiGen executes custom Java-Code from freemarker.

[hohwille]

One thing to clarify for __VarName__ syntax is how to intuitively express to presume the original case? I guess that __varnamE__ or something like this is rather sick. Using single underscores will not work as single underscores occur often in regular Java Code e.g. in CONSTANT_NAME_VARIABLES and we do not want to accidently get _NAME_ resolved here.
Using double vs. tripple underscores is a pain to read and distinguish.
__-varname-__ is also strange. For occurrences outside of Java member names it will also be the regular case to keep the original case. Therefore we need to find an intuitive short and simple syntax for that.

Currently I am out of ideas and maybe better go to bed...

[hohwille]

Just discovered that the following chars can be used in java identifiers:
$ _ ¢ £ ¤ ¥ ؋ ৲ ৳ ৻ ૱ ௹ ฿ ៛ ‿ ⁀ ⁔ ₠ ₡ ₢ ₣ ₤ ₥ ₦ ₧ ₨ ₩ ₪ ₫ € ₭ ₮ ₯ ₰ ₱ ₲ ₳ ₴ ₵ ₶ ₷ ₸ ₹ ꠸ ﷼ ︳ ︴ ﹍ ﹎ ﹏ ﹩ ＄ ＿ ￠ ￡ ￥ ￦

[hohwille]

＄varname＄ - but how do you explain all the users how to enter such character via keyboard...

[hohwille]

We discussed about the design of this issue and came to the following conclusion:

• We should try to avoid too much mixture of template that contains "code" to put into the generated file (besides some variable substituions) and "code" that is actually the implementation of a generator that is executed by CobiGen in order to generate output.
• To enforce this separation and also reuse we came to the idea to allow creating arbitrary classes prefixed by CobiGenMacro that defines static methods. If a template contains a CobiGenMacro*.someStaticMethod() then CobiGen would actually replace this with the result of the reflective invocation converted to String or with nothing if the return type is void or the result is null. Initially we would not allow arguments to macro methods for simplicity. As a nice to have feature if the macro produces empty result/output and it was prefixed by "," (potentially followed by whitespace(s)) then this comma is eliminated. If a CobiGenMacro is inside a static initializer block that would be empty after resolution the entire block should also be eliminated. This would allow a fully blown Generator as CobiGenMacro.
• For the very long run we could actually provide CobiGenMacros by CobiGen API with parameters using lambdas that could allow "for each" and "if" statements that contain the repeated or conditional code to produce inside a lambda. However, this is still in question and could impact readability of templates and raise other complex issues.
• We agreed on the $_Variable_$ syntax and will support this in path names as a first quick win improvement to CobiGen.
• As a next improvement we will implement an own templating engine that will also support this $_Variable_$ syntax within the templates (but no freemarker stuff) so it will be extremely limited so far.
• Then we would improve this templating engine to support CobiGenMacro and eliminate com.capgemini.cobigen.* imports.
• As a big step we would establish, improve/re-design and implement the new CobiGen API that can then be used within CobiGenMacros. This task can also be started in parallel to all the other things.
• We had a long discussion whether it makes sense to have CobiGen.out().java() and allow other things than java here, because the API is actually written in Java so why should language agnostic templates be written in Java produce anything other than Java. Finally we came to the conclusion that it makes sense as CobiGenMacro execution can also be supported if embedded inside XML, Properties or even TypeScript files as kind of comments. Further, a template written in Java could also be a Generator (e.g. also by extending a super-class from CobiGen API) that produces XML or Properties files by generating java.util.Properties or a DOM XML Document.
• For the modular design of CobiGen we should however think about replacing CobiGen.in/out().java() with JavaPlugin.in/out() as otherwise the CobiGen API provided by cobigen-core would then be coupled with the actual plugins such as the JavaPlugin. Instead a language plugin should provide the language specific input and output API. Then the Cobigen_Templates project has to depend on the language specific plugins (or more precisely on their API).

[hohwille]

As my initial proposal code was unfortunately lost, I did a little coding from scratch with hopefully perfect design now:
m-m-m/code@f352826
Based on this we will be able to do source- and byte-code reflection as well as modification to the code including merging and writing the result to a defined location. The same API can also be implemented for typescript. I am also considering to drop qdox and simply generate a parser from the language grammar. Am I going too big here? Or is this cool? What do you think?

[hohwille]

Here is an update to the latest status.

I created a github project as sandbox for the vision how templates could look like with this new feature:
https://github.com/oasp-forge/cobigen-language-agnostic-templates
https://github.com/oasp-forge/cobigen-language-agnostic-templates/blob/master/cobigen-templates/src/main/java/x_rootpackage_x/x_component_x/common/api/x_detail_x/to/X_EntityName_XEto.java
https://github.com/oasp-forge/cobigen-language-agnostic-templates/blob/master/cobigen-templates/src/main/java/x_rootpackage_x/x_component_x/logic/impl/X_Component_XImpl.java
https://github.com/oasp-forge/cobigen-language-agnostic-templates/blob/master/cobigen-templates/src/main/java/x_rootpackage_x/x_component_x/logic/impl/x_detail_x/usecase/UcFindX_EntityName_XImpl.java
https://github.com/oasp-forge/cobigen-language-agnostic-templates/blob/master/cobigen-templates/src/main/java/x_rootpackage_x/x_component_x/dataaccess/impl/x_detail_x/dao/X_EntityName_XDaoImpl.java
You are invited to have a look, give feedback ("cool, when can I get this" or "OMG, I will stay with freemarker"), fork and improve, etc. This is the most important issue to get this design good and therefore I would love to get the best from the community.
BTW: I had to switch from $_EntityName_$ to X_EntityName_X instead as otherwise several tools and languages are causing problems. It is technically working with $ as well so the code compiles but completion and refactoring then does not work properly (edge cases that are not tested by Eclipse & Co.). We do not want to fight against tools we want to utilize them. Hence it is IMHO the best solution to be pragmatic and use an even more "strange" syntax but this way we are 100% compliant as AsciiLetters and Underscores are pretty much accepted everywhere as identifiers. Still the symmetry of X_ and _X gives the minimum readability required to see the variable with a sharp eye and also is exotic enough to not collide with regular code (if someone has a regular class in his customer project that would match this `X_[a-Z]+_X" pattern please let me know, but I double we need to support cases where such a conflict can be realistic.

For the AST lib (mmm-code) I made great progress and is already working quite well (see linked JUnits and Types as demo):

• Parsing bytecode+sourcecode from maven project: https://github.com/m-m-m/code/blob/master/java/impl/src/test/java/net/sf/mmm/code/impl/java/JavaExtendedContextWithMavenAndSourceCodeTest.java
• Generating code and getting output: https://github.com/m-m-m/code/blob/master/base/src/test/java/net/sf/mmm/code/base/type/BaseTypeTest.java
• Fundamentals: https://github.com/m-m-m/code/blob/master/java/impl/src/test/java/net/sf/mmm/code/impl/java/JavaRootContextTest.java
• Merge already integrated: https://github.com/m-m-m/code/blob/master/api/src/main/java/net/sf/mmm/code/api/merge/CodeAdvancedMergeableItem.java (implemented by all major elements)
• Copy and resolving is currently in progress: https://github.com/m-m-m/code/blob/master/api/src/main/java/net/sf/mmm/code/api/copy/CodeNodeItemCopyable.java (I am working on a copy variant taking a CodeCopyMapper call-back interface as input that allows to transform code on copy and to copy entire AST sub-trees with their cross-references pointing to the resolved copies).
• Explore yourself to see more

Review feedback is most welcome. Crucial aspects therefore are:

• Source-Code-Parser: https://github.com/m-m-m/code/tree/master/java/impl/src/main/java/net/sf/mmm/code/impl/java/parser - I spent a lot of energy into this, but anyhow I think it is crap and needs to be dumped and replaced with an (ANTLR?) parser generater + some custom transformer code. However, there are still so many complex cases.
• CodeLanguage: https://github.com/m-m-m/code/blob/master/api/src/main/java/net/sf/mmm/code/api/language/CodeLanguage.java - How can we avoid ending up with flaws like in HibernateDialect?
• How shall we deal with in-deepth-AST stuff (content of bodies, field declaration with assignments, etc.)? This is adding huge complexity. Currently I handle Statements as Strings so a Block/Body is more or less a list of strings. But then resolving variables will remain on search+replace level, merging of bodies will not be possible, etc. I think this is all fine and anything else is nice-to-have for 2020+.
• Lazy initialization and byte-code + source-code loading and analysis plus merge is solved but still complex. Traversing packages can lead to performance issues.

[hohwille]

In our latest workshop we came to the conclusion to define an interface for marcros that will be implemented. In templates you just provide the class references to the template annotation. CobiGen would provide ready to use macros for generation of getters, setters, equals, hashCode, etc. but users could write their own marcos for advanced use-cases. Templates themselves will remain rather simple and therefore easy to read.

[hohwille]

After some more hacking I came to the conclusion that we should create an abstract class CobiGenMarco that macros need to implement. We should design this class stateful such that it is instanciated per use and then thrown away. This would give us the best design for simplicity and also future enhancements without breaking compatibility. So all parameters would simply be set via setters of the abstract parent class CobiGenMacro. The macro developer only needs to implement a single method

public abstract void generate();

The "parameters" are set before by CobiGen via the setters of the parent class and are thereby stored in protected members and hence easily accessible (alternatively via getters to even avoid collision with user defined members).
With this design we can also avoid mistakes of accidentally stateful macros as we have them stateful and therefore simple by desgin.

[hohwille]

To take this even further think of that we would also provide UML models or potentially even OpenAPI contracts via our AST. Generating from an input type like a data model class from Java code, C# code, TypeScript code, UML, or OpenAPI could work using the same templates. Nice...

[hohwille]

During my experiments I came back to the point why it is better to introduce a variable type for dynamic type expressions instead of using a concrete type with an annotation containing the expression:
Even if you put the actual expression in a constant you need to repeat the combination of annotation and type many times.

@CobiGenTemplate(...)
public class UcFind$_EntityName_$Impl implements UcFind$_EntityName_$ {
  @CobiGenDynamicType(condition="$_dataaccesstype_$ == 'dao'", type = $_EntityName_$Dao.class) $_EntityName_$Repository $_entityName_$Dao;

  @Inject
  public void set$_EntityName_$Dao(@CobiGenDynamicType(condition="$_dataaccesstype_$ == 'dao'", type = $_EntityName_$Dao.class) $_EntityName_$Repository $_entityName_$Dao) {
    this.$_entityName_$Dao = $_entityName_$Dao;
  }
  ...
}

If you do not always use the exact same combination it will not work as expected. With my suggestion the expression and type hierarchy is bundled in a single point of information.

@CobiGenTemplate(...)
public class UcFind$_EntityName_$Impl implements UcFind$_EntityName_$ {
  $$EntityNameDaoOrRepository $_entityName_$Dao;

  @Inject
  public void set$_EntityName_$Dao($$EntityNameDaoOrRepository $_entityName_$Dao) {
    this.$_entityName_$Dao = $_entityName_$Dao;
  }
  ...
}

and:

@CobiGenTypeExpression(...)
public interface $$EntityNameDaoOrRepository extends @CobiGenCondition("$_dataaccesstype_$ == 'dao'") $_EntityName_$Dao.class, @CobiGenCondition("$_dataaccesstype_$ != 'dao'") DefaultRepository<$_EntityName_$Entity> {
}

or maybe we can even combine both to allow the planned reuse but also having what you suggested:

@CobiGenTypeExpression(...)
public interface $$EntityNameDaoOrRepository extends @CobiGenDynamicType(condition="$_dataaccesstype_$ == 'dao'", type = $_EntityName_$Dao.class) DefaultRepository<$_EntityName_$Entity> {
}

Please also note that @CobiGenDynamicType could also have a parameter method suffix that could be set to "<$EntityName$Entity>" but if omitted it would default to the generic declaration of the substituted type.

I suggested to use a syntax like $$EntityNameDaoOrRepository instead of $_EntityName_$DaoOrRepository to better distinguish templates and type expressions and also simplify the elimination of imports. However, with mmm-code we can even go for a clean approach and remove all imports and then re-generate them after the template is fully resolved. The feature is already implemented: m-m-m/code#10

[hohwille]

Still I am not finally convinced regarding our variable syntax. IMHO it is readable but however, IDE support is not working fine this way. Eclipse often messes up imports or fails with auto-completion. Maybe we should come back to original suggestions like __EntityName__ or X_EntityName_X as originally suggested. Maybe that would be even more agnostic as I did not test $ in all occurences of XML, JSON, C#, Kotlin, etc. We can change now or never...
WDYT?

[hohwille]

I still like __EntityName__ syntax and tested it on my mac with git having no issues. Can someone test this again on windows as we once had issues with files and folders of such names in git on windows (long time ago).