:toc: macro toc::[] = Introduction to CobiGen external plug-ins Since September of 2019, a major change on CobiGen has taken place. CobiGen is written in Java code and previously, it was very hard for developers to create new plug-ins in other languages. Creating a new plug-in means: * Being able to parse a file in that language. * Create a human readable model that can be used to generate templates (by retrieving properties from the model). * Enable merging files, so that user's code does not get removed. For the Java plug-in it was relatively easy. As you are inside the Java world, you can use multiple utilities or libraries in order to get the link:https://en.wikipedia.org/wiki/Abstract_syntax_tree[AST] or to merge Java code. With this new feature, we wanted that behaviour to be possible for any programming language. == General intuition Below you will find a very high level description of how CobiGen worked in previous versions: image::images/howtos/todo-plugin/oldCobiGen.png[Old CobiGen,width="450"link="images/howtos/todo-plugin/oldCobiGen.png"] Basically, when a new input file was sent to CobiGen, it called the input reader to create a model of it (see link:https://github.com/devonfw/cobigen/wiki/cobigen-tsplugin#object-model[here] an example of a model). That model was sent to the template engine. Afterwards, the template engine generated a new file which had to be merged with the original one. All this code was implemented in Java. On the new version, we have implemented a handler (`ExternalProcessHandler`) which connects through TCP/IP connection to a server (normally on localhost:5000). This server can be implemented in any language (.NET, NodeJS, Python...) it just needs to implement a REST API defined link:https://github.com/devonfw/cobigen-template-plugin/blob/master/APIContract.yml[here]. The most important services are the input reading and merging: image::images/howtos/todo-plugin/newCobiGen.png[New CobiGen,width="450"link="images/howtos/todo-plugin/newCobiGen.png"] CobiGen acts as a client that sends requests to the server in order to read the input file and create a model. The model is returned to the template engine so that it generates a new file. Finally, it is sent back to get merged with the original file. == How to create new external plug-in The creation of a new plug-in consists mainly in three steps: * Creation of the server (external process). * Creation of a CobiGen plug-in. * Creation of templates. === Server (external process) The server can be programmed in any language that is able to implement REST services endpoints. The API that needs to implement is defined with link:https://github.com/devonfw/cobigen-template-plugin/blob/master/APIContract.yml[this contract]. You can paste the content to https://editor.swagger.io/ for a better look. We have already created a NestJS server that implements the API defined above. You can find the code link:https://github.com/devonfw/cobigen-devon4node-server/blob/master/cobigen-nest-server/src/processmanagement/processmanagement.controller.ts[here] which you can use as an example. As you can see, the endpoints have the following naming convention: `processmanagement/todoplugin/nameOfService` where you will have to change `todo` to your plug-in name (e.g. `rustplugin`, `pyplugin`, `goplugin`...) When implementing service `getInputModel` which returns a model from the input file there are only two restrictions: * A `path` key must be added. Its value can be the full path of the input file or just the file name. It is needed because in CobiGen there is a link:https://github.com/devonfw/cobigen/wiki/eclipse-plugin_development#batch-mode[batch mode], in which you can have multiple input objects inside the same input file. You do not need to worry about batch mode for now. * On the root of your model, for each found key that is an object (defined with brackets `[{}]`), CobiGen will try to use it as an input object. For example, this could be a valid model: + ```JSON { "path": "example/path/employee.entity.ts" "classes": [ { "identifier": "Employee", "modifiers": [ "export" ], "decorators": [ { "identifier": { "name": "Entity", "module": "typeorm" }, "isCallExpression": true } ], "properties": [ { "identifier": "id", ... ... ... }] "interfaces": [{ ... }] } ``` For this model, CobiGen would use as input objects all the `classes` and `interfaces` defined. On the templates we would be able to do `model.classes[0].identifier` to get the class name. These input objects depend on the language, therefore you can use any key. In order to test the server, you will have to deploy it on your local machine (localhost), default port is 5000. If that port is already in use, you can deploy it on higher port values (5001, 5002...). Nevertheless, we explain <> the testing process as you need to complete the next step before. IMPORTANT: Your server must accept one argument when running it. The argument will be the port number (as an integer). This will be used for CobiGen in order to handle blocked ports when deploying your server. Check this link:https://github.com/devonfw/cobigen-devon4node-server/blob/master/cobigen-nest-server/src/main.ts#L47[code] to see how we implemented that argument on our NestJS server. === CobiGen plug-in You will have to create a new CobiGen plug-in that connects to the server. But *do not worry*, you will not have to implement anything new. We have a CobiGen plug-in template available, the only changes needed are renaming files and setting some properties on the pom.xml. Please follow these steps: * Get the CobiGen plug-in template from link:https://github.com/devonfw/cobigen-template-plugin[here]. It is a template repository (new GitHub feature), so you can click on "Use this template" as shown below: + image::images/howtos/todo-plugin/usePluginTemplate.png[Plugin CobiGen template,width="550"link="images/howtos/todo-plugin/usePluginTemplate.png"] * Name your repo as `cobigen-name-plugin` where `name` can be python, rust, go... In our case we will create a `nest` plug-in. It will create a repo with only one commit which contains all the needed files. * Clone your just created repo and import folder `cobigen-todoplugin` as a Maven project on any Java IDE, though we recommend you devonfw ;) + image::images/howtos/todo-plugin/importPluginEclipse.png[Import plugin,width="450"link="images/howtos/todo-plugin/importPluginEclipse.png"] * Rename all the `todoplugin` folders, files and class names to `nameplugin`. In our case `nestplugin`. In Eclipse you can easily rename by right clicking and then refactor -> rename: image::images/howtos/todo-plugin/renamePlugin.png[Rename plugin,width="450"link="images/howtos/todo-plugin/renamePlugin.png"] NOTE: We recommend you to select all the checkboxes image::images/howtos/todo-plugin/renameCheckbox.png[Rename checkbox,width="450"link="images/howtos/todo-plugin/renameCheckbox.png"] * Remember to change in `src/main/java` and `src/test/java` all the package, files and class names to use your plug-in name. The final result would be: + image::images/howtos/todo-plugin/packageStructure.png[Package structure,width="300"link="images/howtos/todo-plugin/packageStructure.png"] * Now we just need to change some strings, this is needed for CobiGen to register all the different plugins (they need unique names). In class `TodoPluginActivator` (in our case `NestPluginActivator`), change all the `todo` to your plug-in name. See below the 3 strings that need to be changed: + image::images/howtos/todo-plugin/pluginActivator.png[Plugin activator,width="450"link="images/howtos/todo-plugin/pluginActivator.png"] * Finally, we will change some properties from the `pom.xml` of the project. These properties define the server (external process) that is going to be used: .. Inside `pom.xml`, press `Ctrl + F` to perform a find and replace operation. Replace all `todo` with your plugin name: + image::images/howtos/todo-plugin/setPomProperties.png[Pom properties,width="550"link="images/howtos/todo-plugin/setPomProperties.png"] .. We are going to explain the server properties: ... `artifactId`: This is the name of your plug-in, that will be used for a future release on Maven Central. ... `plugin.name`: does not need to be changed as it uses the property from the `artifactId`. When connecting to the server, it will send a request to `localhost:5000/{plugin.name}plugin/isConnectionReady`, that is why it is important to use an unique name for the plug-in. ... `server.name`: This defines how the server executable (_.exe_) file will be named. This _.exe_ file contains all the needed resources for deploying the server. You can use any name you want. ... `server.version`: You will specify here the server version that needs to be used. The _.exe_ file will be named as `{server.name}-{server.version}.exe`. ... `server.url`: This will define from where to download the server. We *really* recommend you using NPM which is a package manager we know it works well. We explain <> how to release the server on NPM. This will download the _.exe_ file for Windows. ... `server.url.linux`: Same as before, but this should download the _.exe_ file for Linux systems. If you do not want to implement a Linux version of the plug-in, just use the same URL from Windows or MacOS. ... `server.url.macos`: Same as before, but this should download the _.exe_ file for MacOS systems. If you do not want to implement a MacOS version of the plug-in, just use the same URL from Linux or Windows. [[testing_phase]] == Testing phase Now that you have finished with the implementation of the server and the creation of a new CobiGen plug-in, we are going to explain how you can test that everything works fine: . Deploy the server on port 5000. . Run `mvn clean test` on the CobiGen-plugin or run the JUnit tests directly on Eclipse. .. If the server and the plug-in are working properly, some tests will pass and other will fail (we need to tweak them). .. If every test fails, something is wrong in your code. . In order to fix the failing tests, go to `src/test/java`. The failing tests make use of sample input files that we added in sake of example: + image::images/howtos/todo-plugin/failingTest.png[Pom properties,width="550"link="images/howtos/todo-plugin/failingTest.png"] Replace those files (on `src/test/resources/testadata/unittest/files/...`) with the correct input files for your server. == Releasing Now that you have already tested that everything works fine, we are going to explain how to release the server and the plug-in. [[release_server]] === Release the server We are going to use link:https://www.npmjs.com/[NPM] to store the executable of our server. Even though NPM is a package manager for JavaScript, it can be used for our purpose. * Get the CobiGen server template from link:https://github.com/devonfw/cobigen-template-server[here]. It is a template repository (new GitHub feature), so you can click on "Use this template" as shown below: + image::images/howtos/todo-plugin/useServerTemplate.png[Server CobiGen template,width="550"link="images/howtos/todo-plugin/useServerTemplate.png"] * Name your repo as `cobigen-name-server` where `name` can be python, rust, go... In our case we will create a `nest` plug-in. It will create a repo with only one commit which contains all the needed files. * Clone your just created repo and go to folder `cobigen-todo-server`. It will just contain two files: `ExternalProcessContract.yml` is the OpenAPI definition which you can modify with your own server definition (this step is optional), and `package.json` is a file needed for NPM in order to define where to publish this package: + ```JSON { "name": "@devonfw/cobigen-todo-server", "version": "1.0.0", "description": "Todo server to implement the input reader and merger for CobiGen", "author": "CobiGen Team", "license": "Apache" } ``` Those are the default properties. This would push a new package `cobigen-todo-server` on the `devonfw` organization, with version 1.0.0. We have no restrictions here, you can use any organization, though we always recommend devonfw. NOTE: Remember to change all the `todo` to your server name. * Add your executable file into the `cobigen-todo-server` folder, just like below. As we said previously, this _.exe_ is the server ready to be deployed. + ---- cobigen-template-server/ |- cobigen-todo-server/ |- ExternalProcessContract.yml |- package.json |- todoserver-1.0.0.exe ---- * Finally, we have to publish to NPM. If you have never done it, you can follow this link:https://www.freecodecamp.org/news/how-to-make-a-beautiful-tiny-npm-package-and-publish-it-2881d4307f78/[tutorial]. Basically you need to login into NPM and run: + [source, cmd] cd cobigen-todo-server/ npm publish --access=public NOTE: To release Linux and MacOS versions of your plug-in, just add the suffix into the package name (e.g. `@devonfw/cobigen-todo-server-linux`) That's it! You have published the first version of your server. Now you just need to modify the properties defined on the pom of your CobiGen plug-in. Please see next section for more information. === Releasing CobiGen plug-in * Change the pom.xml to define all the properties. You can see below a final example for `nest`: + ```XML ... com.devonfw.cobigen nestplugin CobiGen - Nest Plug-in 1.0.0 jar CobiGen - nest Plug-in ${project.artifactId} nestserver 1.0.0 https\://registry.npmjs.org/@devonfw/cobigen-nest-server/-/cobigen-nest-server-${server.version}.tgz https\://registry.npmjs.org/@devonfw/cobigen-nest-server-linux/-/cobigen-nest-server-linux-${server.version}.tgz https\://registry.npmjs.org/@devonfw/cobigen-nest-server-macos/-/cobigen-nest-server-macos-${server.version}.tgz ... ``` * Deploy to Maven Central. == Templates creation After following above steps, we now have a CobiGen plug-in that connects to a server (external process) which reads your input files, returns a model and is able to merge files. However, we need a key component for our plug-in to be useful. We need to define templates: * Fork our CobiGen main repository, from link:https://github.com/devonfw/cobigen.git[here] and clone it into your PC. Stay in the `master` branch and import into your IDE `cobigen-templates\templates-devon4j`. Set the Java version of the project to 1.8 if needed. * Create a new folder on `src/main/templates`, this will contain all your templates. You can use any name, but please use underscores as separators. In our case, we created a folder `crud_typescript_angular_client_app` to generate an Angular client from a TypeORM entity (NodeJS entity). + image::images/howtos/todo-plugin/templatesProject.png[Templates project,width="450"link="images/howtos/todo-plugin/templatesProject.png"] * Inside your folder, create a `templates` folder. As you can see below, the folder structure of the generated files starts here (the sources). Also we need a configuration file `templates.xml` that should be on the same level as `templates/` folder. For now, copy and paste a `templates.xml` file from any of the templates folder. + image::images/howtos/todo-plugin/templatesInside.png[Templates project,width="450"link="images/howtos/todo-plugin/templatesInside.png"] * Start creating your own templates. Our default templates language is FreeMarker, but you can also use Velocity. Add the extension to the file (`.ftl`) and start developing templates! You can find useful documentation link:https://github.com/devonfw/cobigen/wiki/cobigen-templates_helpful-links[here]. * After creating all the templates, you need to modify `context.xml` which is located on the root of `src/main/templates`. There you need to define a trigger, which is used for CobiGen to know when to trigger a plug-in. I recommend you to copy and paste the following trigger: + ```XML ``` * Change `templateFolder` to your templates folder name. `id` you can use any, but it is recommendable to use the same as the template folder name. `type` is the `TRIGGER_TYPE` we defined above on the `NestPluginActivator` class. On `matcher` just change the `value`: `([^\.]+).entity.ts` means that we will only accept input files that contain `anyString.entity.ts`. This improves usability, so that users only generate using the correct input files. You will find more info about `variableAssignment` link:https://github.com/devonfw/cobigen/wiki/cobigen-core_configuration#variableassignment-node[here]. * Finally, is time to configure `templates.xml`. It is needed for organizing templates into increments, please take a look into this link:https://github.com/devonfw/cobigen/wiki/cobigen-core_configuration#templates-configuration[documentation]. === Testing templates * When you have finished your templates you will like to test them. On the templates-devon4j `pom.xml` remove the SNAPSHOT from the version (in our case the version will be 3.1.8). Run `mvn clean install -DskipTests` on the project. We skip tests because you need special permissions to download artifacts from our Nexus. Remember the version that has just been installed: + image::images/howtos/todo-plugin/templatesSnapshot.png[Templates snapshot version,width="550"link="images/howtos/todo-plugin/templatesSnapshot.png"] NOTE: We always recommend using the devonfw console, which already contains a working Maven version. * Now we have your last version of the templates ready to be used. We need to use that latest version in CobiGen. We will use the CobiGen CLI that you will find in your cloned repo, at `cobigen-cli/cli`. Import the project into your IDE. * Inside the project, go to `src/main/resources/pom.xml`. This pom.xml is used on runtime in order to install all the CobiGen plug-ins and templates. Add there your latest templates version and the previously created plug-in: + image::images/howtos/todo-plugin/cliPom.png[CLI pom,width="450"link="images/howtos/todo-plugin/cliPom.png"] * Afterwards, run `mvn clean install -DskipTests` and CobiGen will get your plug-ins. Now you have three options to test templates: 1. Using Eclipse run as: + a. Inside Eclipse, you can run the CobiGen-CLI as a Java application. Right click class `CobiGenCLI.java` -> run as -> run configurations... and create a new Java application as shown below: + image::images/howtos/todo-plugin/runConfigurations.png[Create configuration,width="450"link="images/howtos/todo-plugin/runConfigurations.png"] b. That will create a `CobiGenCLI` configuration where we can set arguments to the CLI. Let's first begin with showing the CLI version, which should print a list of all plug-ins, including ours. + image::images/howtos/todo-plugin/runAsArgs.png[Run version,width="450"link="images/howtos/todo-plugin/runAsArgs.png"] + ```Text ... name:= propertyplugin version = 2.0.0 name:= jsonplugin version = 2.0.0 name:= templates-devon4j version = 3.1.8 name:= nestplugin version = 1.0.0 ... ``` c. If that worked, now you can send any arguments to the CLI in order to generate with your templates. Please follow link:https://github.com/devonfw/cobigen/wiki/howto_Cobigen-CLI-generation[this guide] that explains all the CLI commands. 2. Modify the already present JUnit tests on the CLI project: They test the generation of templates from multiple plug-ins, you can add your own tests and input files. 3. Use the CLI jar to execute commands: + a. The `mvn clean install -DskipTests` command will have created a `cli.jar` inside your target folder (`cobigen-cli/cli/target`). Open the jar with any unzipper and extract to the current location `class-loader-agent.jar`, `cobigen.bat` and `cg.bat`: + image::images/howtos/todo-plugin/extractFilesCLI.png[Extract files,width="450"link="images/howtos/todo-plugin/extractFilesCLI.png"] + b. Now you can run any CobiGen CLI commands using a console. link:https://github.com/devonfw/cobigen/wiki/howto_Cobigen-CLI-generation[This guide] explains all the CLI commands. + image::images/howtos/todo-plugin/runningCLI.png[Run CLI,width="450"link="images/howtos/todo-plugin/runningCLI.png]