Is there a supported way to edit .copier-answers.yml

[ppg]

This question spun off of #313 but was getting off topic there; moving to an issue more directly associated with the topic at hand.

Believe me, I'm the maintainer, I know how this thing works.

I understand what you're saying (and that #317 is documenting) but I think I'm still confused on how that is/would happen, especially since I haven't seen what is being described. When copier reads .copier-answers.yml and it has those answers + templates, how does it tell the difference between an answer in .copier-answers.yml and one provided via the command line? In the cases I've seen it simply renders the templates with the answers and writes out the files, thus resulting in the behavior I desire of being able to change .copier-answers.yml and having it update the files.

I think the broader question is how can/could this be supported? Using --data doesn't seem very user friendly, as it requires a user to know all the things they could set, and type it correctly. In our case, the .copier-answers.yml holds those options, as well as documentation on what they do and what it could/should be set to. So from a usage perspective, that's an ideal entry point to find question_1 and see the answer, see what it does, what the options are, if it's non-conforming and change right there. It also is potentially a nicer way to bootstrap a project, where I copy a file of an existing project as a starting point rather than have to answer a bunch of questions.

One item that I just thought of that I noted on a different issue that might be responsible for some of what I'm seeing is that I am specifically removing _commit from the .copier-answers.yml specifically because I'm trying to avoid the update_diff code path. Perhaps that's the code path that is doing the detection you're talking about, and where updating .copier-answers.yml directly isn't supported and won't affect the generated files? In that case, is updating .copier-answers.yml directly supported with the copy command? If so maybe this all boils down to me wanting that recopy command you mention on that issue, since update is enforcing some workflow that I don't personally want for updates?

If this is the case, an unfortunate side effect of forcing down the copy_local code path (which I want for both initial layout and later updates) is that shutil.rmtree which blows away my working directory often if anything isn't right. It'd be nice per #302 if I could tell copier (and if it was the default) to not delete my directory no matter what workflow it thinks it's using.

[yajo]

I understand what you're saying (and that #317 is documenting) but I think I'm still confused on how that is/would happen, especially since I haven't seen what is being described.

If you watch the recording in #317 (comment), you can see one of the scenarios that would trigger a problematic case.

When copier reads .copier-answers.yml and it has those answers + templates, how does it tell the difference between an answer in .copier-answers.yml and one provided via the command line?

You can see it really does. It stores answers separately to be able to distinguish them:

copier/copier/config/objects.py

Lines 134 to 144 in 62d38a8

	@property
	def data(self) -> t_ChainMap[str, Any]:
	"""The data object comes from different sources, sorted by priority."""
	return ChainMap(
	self._data_mutable,
	self.data_from_asking_user,
	self.data_from_init,
	self.data_from_answers_file,
	self.data_from_template_defaults,
	DEFAULT_DATA,
	)

the .copier-answers.yml holds those options, as well as documentation on what they do

This seems like a wrong usage of that file too. The common place to store docs about questions is copier.yaml instead.

Also, if your template evolved and modified, added or removed some questions, how is your user supposed to know that by just looking at his last answers?

I think the broader question is how can/could this be supported?

I think it's more important to know why. You say you don't like to use --data, but why not just run over the questionary again?

If all you want is to forbid writes to specific files, possibly a pre-commit check is a better tool for that.

You seem to be trying to forbid your template users to smart update their projects, but I don't think you should. Usually you either don't support updates at all (in which case your template is bootstrap-only) or support them and let your users to evolve their subprojects freely.

If you opt for supporting only bootstrapping, #235 would help to avoid repeating stuff. If you opt to support updates, then the recopy command explained in #302 (comment) would let your users choose to update or recopy. I think it's a decision that belongs to the user.

OTOH another way to improve usage experience would be to support passing in a data file instead of only individual data flags.

[ppg]

This seems like a wrong usage of that file too. The common place to store docs about questions is copier.yaml instead.

Also, if your template evolved and modified, added or removed some questions, how is your user supposed to know that by just looking at his last answers?

The issue with that is that copier.yaml is in a different repo, so as a user looking to change my repo I'd be hoping that knobs I would turn would be easily accessible in my repo where I'm working. We apply the scaffolding nightly, so the last answers should generally have all the things a person working in a repo would care to change. Also, nothing stops them from going to look at copier.yaml if they need to.

I think my general idea here is I'm trying to optimize for our common case, which is there's a non-conforming answer in the .copier-answers.yml someone is looking to fix (i.e. change go:lint:changes_only to false and fix lint issues) or to alter some major, well-established switch (change from travisci to circleci). Having to go research what to put into --data (and type it correctly) or re-running the questionnaire seems to cater to how copier wants to work, but not how one of our developers looking to make changes would want to work.

If you opt for supporting only bootstrapping, #235 would help to avoid repeating stuff. If you opt to support updates, then the recopy command explained in #302 (comment) would let your users choose to update or recopy. I think it's a decision that belongs to the user.

Yes, I think recopy is what I'm ultimately looking for and misunderstood what update was meant to be and yes I think that choice belongs to the user. It's somewhat semantic, but how I look at that is that in both cases I want to 'update' an existing repository; in the cases you're describing the user would like to use a smart update feature that tries to share control of a file between the automated tooling and the manual user input; in my cases I'd rather have a file be owned by scaffolding or not, and I can expose entry points where it's applicable.

You seem to be trying to forbid your template users to smart update their projects, but I don't think you should. Usually you either don't support updates at all (in which case your template is bootstrap-only) or support them and let your users to evolve their subprojects freely.

I think this is where our workflows/philosophies diverge. I am specifically trying to stop the smart update process you describe as I don't want to worry about and have to think through end-users editing a file that scaffolding is generating, as the behavior is unknown then. I want projects to have consistency and thus want to keep certain things/parts fixed, and allow certain parts to change. To me that says things like I control bin/test and I expose knobs they can turn; if absolutely necessary I can expose a hook/entry point for them to add custom stuff, but I'd rather wait until that configuration is necessary and stick with a well-established convention in the file I control until that point.

I think my feelings come from the fact that I see two main things I need copier to do:

• Boostrap - runs once at the start of a project (infrequent)
• Update - runs in an ongoing fashion (very frequent)

Details on how I update in an ongoing fashion seem to vary based on the user; you see a desire for a 'smart update' that wants to try to share ownership of files; I would rather have a 'recopy' that wants to own files fully and let the owned files define entry points if desired; preferably I'd be able to control on a per-file basis so I could mark some files to write once, some to rewrite ongoing, and some to write once and merge ongoing. I have use cases that would utilize each of those, but if forced to pick one I want recopy as I'm worried that uses will change shared files in ways that prevent me from automatically reapplying scaffolding. It also seems easier (to me) to rationalize about a file being owned or not: I can easily know what the resultant file will be and I can easily tell which projects have parts customized and which ones don't.

[yajo]

to alter some major, well-established switch (change from travisci to circleci).

This is an interesting point. The only way for copier to autodelete the travis CI file would be to use update. Maybe your expectation is that the user deletes the file on his own, or maybe you workaround the problem with a task, but that's not the UX copier wants to support.

Having to go research what to put into --data (and type it correctly) or re-running the questionnaire seems to cater to how copier wants to work, but not how one of our developers looking to make changes would want to work.

This is never going to be supported for updates, because of the technical limitations that I explained already. However you don't seem to want updates, so it shouldn't affect you. What you want is recopy, but until that is implemented, sadly, you're on your own.

OTOH I'm open to suggestions on how to improve the answering workflow (as long as it does not mean editing the answers file in place 😁). BTW it got much better in the last alpha, with live validation, multiline input, placeholders and conditional questions.

in my cases I'd rather have a file be owned by scaffolding or not

This won't be something you'll be able to define in the template, even with recopy. It would introduce too many edge cases.

What I would recommend in this case is to split into 2 templates: one that doesn't support updates and one that does. Or to update and then --exclude some_files recopy in 2 steps.

OTOH these days there are better tools to handle code ownership. Both Gitlab and Github support that. You also have pre-commit. Maybe you'll get a better experience leveraging one of those tools, that exist specifically for what you're trying to do here, and provide more features around that.

I don't want to worry about and have to think through end-users editing a file that scaffolding is generating, as the behavior is unknown then.

I understand this, but it's also true that it would be unexpected that something I fixed yesterday in my project's CI is broken again today. Putting both into balance, the latter problem would produce more frustration. A bot that unmakes human changes every night can easily lead into problems. It would be much better in general to forbid edits of those files in the 1st place, which also points to using code ownership tools.

[yajo]

an unfortunate side effect of forcing down the copy_local code path (which I want for both initial layout and later updates) is that shutil.rmtree which blows away my working directory often if anything isn't right

BTW, have you tried passing --no-cleanup?

[ppg]

an unfortunate side effect of forcing down the copy_local code path (which I want for both initial layout and later updates) is that shutil.rmtree which blows away my working directory often if anything isn't right

BTW, have you tried passing --no-cleanup?

I'm on the latest stable release, v5.1.0 which I don't think has this option? Also, from perusing the master code, this isn't supported on update right? Meaning someone looking to update a repo would need to re-specify src-path each time, which doesn't seem user friendly.

[yajo]

Ah yes sorry, I forgot it landed on v6.0.0a0. However it'll disappear soon probably to be automatic.