Templates extensions

[pawamoy]

So I've wrote this PoC project copier-templates-extensions that allows a template writer to provide Jinja2 extensions directly within a template, as Python files, and reference them in copier.yml configuration, in the (soon merged ❤️?) _jinja_extensions option, as relative paths instead of Python dotted-paths.

In short, using this special extension, you won't have to actually wrap your extensions as packages so they can be installed by users and used by Copier while rendering. They can live directly within your template code!

But! For this to work, my special extension needs to know where the local clone of the template being rendered is. Copier must provide it some way or another. I've thought of two possibilities:

1. set an environment variable. I've tried it and it works. A simple os.environ["COPIER_TEMPLATE_PATH"] = str(self.template.local_abspath) at the beginning of Worker.run_copy did the trick
2. subclass the Jinja environment, and pass Copier's data (config, template, anything needed) when instantiating it in Worker.jinja_env. This way my special extension would just access it from the environment argument it receives.

---

Of course, before discussing the technical possibilities, we can discuss the feature itself. Maybe you'd like to support something like that directly in Copier? It required patching the import_string function in the jinja.environment module though.

Well, let me know what you think 😄!

[yajo]

It's amazing how far you arrived with that PoC. Very useful indeed. 🙌🏼

Although I didn't dig too much through the code, my guess is that a Jinja template that uses the filesystem loader should be able to rely on the loader itself to find the path where it's being executed.

Maybe it wouldn't work for dir/file names, which are just strings and not files.

However, something that worries me is the security. One of my goals for Copier 6 is to be more explicit on its insecure parts. For example, right now migrations and tasks can actually install a virus without the user knowing what happened. I plan to ask independently by task/migration if the user wants to execute it, or to add some --UNSAFE flag to allow those things.

One template could potentially live without the tasks/migrations stuff and still be useful, but if we load extensions, those would be completely required. Just take a look at https://github.com/metwork-framework/jinja2_shell_extension to see how dangerous an extension already is. Bundling that code in the template would make Copier more powerful, but always undeniably dangerous.

Apart from all that, the workflow I try to encourage is to use macros more than extensions, as macros are safe, well documented, predictable and reliable. They are a core feature for Jinja, that can be bundled with your template already.

If I were to package some extension that allows for free python usage inside Jinja, I'd just add some simple eval filter. Imagine:

{% import "_macros" as macros %}{{ macros.some_python_code()|eval }}

The macro produces the code, and eval executes it. Seems quite simple, and not strictly tied to Copier.

Another problem of supporting this is that we don't have strong requirements in a template. It would be hard to check if your code would conflict with some library version. OTOH, if you install an extension from pypi into Copier's venv, you are covered by pip's automatic anti-conflict system.

Finally, I don't know if you've been able to play with https://github.com/semirook/jinja-atoms, which seems to do what you want here. Also, https://pypi.org/project/jinja2-custom-filters-extension/ is small and includes a slug filter (if you don't want to import the whole cookiecutter and that's what you really want).

Yikes, this was long! Well, summarizing:

• I don't think Copier is the place to implement this. Instead, a Jinja extension should be able to do it, and that extension should probably work fine without Copier too.
• Security is important. Copier needs a way to distinguish what stuff is potentially insecure, to warn users.
• If you can use a macro, use a macro.
• Remember that we support tasks, so maybe a task would be enough for you? You wouldn't need this too in that case.

I didn't answer yes or no haha. Just sharing some thoughts. WDYT?

[pawamoy]

my guess is that a Jinja template that uses the filesystem loader should be able to rely on the loader itself to find the path where it's being executed.

HA! Indeed, I'll try that and report back, thanks!

---

I get your point about security, and I'm totally OK with maintaining such a feature in an external extension 🙂

About macros: I'll stay away from eval 😅 also, I don't think I could import something using eval?
About jinja-atoms: I've read the README two times now, and I'm still not sure I understand what it does, or how to use it 😅
But I think you can load snippets/macros from a module, and then use it with the special atom tag? In that case, it's the same problem: I'd need to know where to load these atoms from (the temporary template clone).

Thanks for your answer, I'll investigate that loader thing! Provided it's a file system loader, it's true that the extension could work without Copier too!

[pawamoy]

In the end I was able to retrieve the path to the locally cloned template from the Jinja environment's file system loader, so it's perfect 🙂

I was even able to implement a "context hook" that allows a template writer to update the context before rendering a template 😛

If you have any suggestions about how to improve security (between Copier and the extension I've written), please don't hesitate to share them!

[yajo]

That seems awesome!

Regarding security, I'll try to improve that in copier itself, but obviously extensions are absolutely insecure and there's nothing I can do apart from warning its usage, so it seems good as is. 😁

I'll accept a PR advertising that extension along with the others if you want. 👌

[pawamoy]

Nice, I'll send one a bit later then 😄