Move from_email_address options to new SiteEmailAddresses entity

[colemanw]

Overview

Moves the 'from_email_address' option group into its own table: Introducing the new SiteEmailAddress entity. 🎉

Migration and Backward Compatibility Notes

New-to-Old column map

SiteEmailAddress	OptionValue
id	value
display_name	label (in quotes)
email	label (in brackets)
description	description
is_active	is_active
is_default	is_default
domain_id	domain_id

Backward Compatibility

This adds an apiWrapper for backward-compatibility, to allow SiteEmailAddress records to be accessed as if it were an option group. When accessing it via the OptionValue api, the new email and display_name columns are concatenated to produce familiar value of "Name" <[email protected]>, and id is returned as value. Here's what is and is not supported:

Function	Works	Notes
Api3 get	✅	$params must contain 'option_group_id' => 'from_email_address'
Api3 create	✅	$params must contain 'option_group_id' => 'from_email_address'
Api3 update (aka. create + an id)	⚠️	works if passed id of an existing siteEmailAddress and a correctly formatted email header as value
Api3 delete	🔴	delete action only gets passed an id which is not enough info for the adapter
Api3 chaining	✅	get chained with create/update/delete will all work
Api4 get	✅	WHERE must contain 'option_group_id:name', '=', 'from_email_address'
Api4 update	⚠️	Update by id won't work. WHERE must contain 'option_group_id:name', '=', 'from_email_address'
Api4 create	✅	$values must contain 'option_group_id:name' => 'from_email_address'
Api4 delete	⚠️	Delete by id won't work. WHERE must contain 'option_group_id:name', '=', 'from_email_address'
Api4 chaining	🔴	Couldn't find any uses in universe so didn't add support for this
CRM_Core_BAO_Domain::getNameAndEmail CRM_Core_BAO_Domain::getFromEmail	✅	No signature change
CRM_Core_BAO_Email::getFromEmail() CRM_Core_BAO_Email::domainEmails()	✅	No signature change
CRM_Core_OptionGroup::values	✅	Supported params: $labelColumnName, $keyColumnName, $onlyActive, $condition
CRM_Core_OptionGroup::getDefaultValue	✅	Returns id of default email address for current domain
Other core functions	🔴	CRM_Core_OptionGroup::valuesByID, CRM_Core_Pseudoconstant::get, etc.
SQL Queries, etc.	🔴	The options no longer exist in the civicrm_option_value table so can't be queried.

Before

The list of from email addresses was not well-placed in the OptionValue table because:

1. It was the only domain-specific option, overcomplicating that table with domain handling just for this one option list (aside, there was actually one other: Grant Type, but probably wasn't actually being used for multidomain).
2. Had to mush the display name and email address together, like "My Name" <[email protected]> which wasn't fun for users to type and involved a lot of splitting/unsplitting strings.
3. Nonstandard label in the option_value table contained < and > characters, making general rules about html escaping difficult.

After

New table has just the fields needed for storing site email addresses. UI switched to SearchKit/FormBuilder:

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

Quick links for reviewers...

• ➡️ PR Demo site (can be rebuilt with the command test this please).
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[totten]

Personal feeling: yes, please, for the love of god, yes.

---

Big consideration is going to be technical interop for extensions and sysadmin scripts. (In my copy of universe, there are ~90 references to from_email_address in ~45 extensions. That's a mix of things, and some may not be OG-references. But a lot are) Maybe there's some clever mechanism to help smooth it over?

(Like... hook_optionValues? Or, anticipating a mix of reads+writes and BAO/APIv3/APIv4, it may merit a bidirectional sync -- eg an extension legacyfromaddress which uses hook_post to sync the tables. I don't really know.)

---

In the future, we may get some better options to address permissions and routing if it's a proper entity.

---

Small aside #bikeshed: in ezcMailAddress and Civi\WorkflowMessage\Traits\AddressingTrait, they refer to these properties as name,email not display_name,email. I think name is probably more common in email-land, so I would gently vote for that. Not a huge deal.

[highfalutin]

I don't deal with from_email_address much, but the proposed schema looks good to me. What would be the best practice in Civi for maintaining a single is_default = 1 per domain?

[colemanw]

@highfalutin What would be the best practice in Civi for maintaining a single is_default = 1 per domain?

I added a unique index to the table for that, and the business logic would include an implementation of hook_civicrm_pre to set all others in the domain to false whenever setting one to true.

UPDATE: You can't do that sort of unique index in mySql, unfortunately, but I did add the business logic to enforce it during every write operation that goes through the Api/BAO.

[colemanw]

@totten I think name is probably more common in email-land, so I would gently vote for that. Not a huge deal.

Well in Civi-land, name usually means "machine name" so I avoided it for that reason, but since there's precedent with other email entities, sure let's go with name.

[totten]

Well in Civi-land, name usually means "machine name" so I avoided it for that reason, but since there's precedent with other email entities, sure let's go with name.

Ah, good point. Terms are painted into a corner. Dealers choice.

[totten]

Hmm, sensible functionality, but this highlights some confusion in naming:

• Before: one phrase, "from email address"
• After: three phrases, varying by context:
  • Sometimes "domain email address" (eg APIv4 and SQL)
  • Sometimes "site email address" (eg new routes)
  • Sometimes "from email address" (eg existing routes)

Each name has its own semantic niggles, so I can share some ambivalence. And TBH, I hardly noticed that the initial draft spoke to from_email_address and DomainEmailAddress. (The option-group namespace differs from the APIv4 namespace, so sure... there's gonna be something different there.)

But three different names? That's gonna be confusing later, right?

(If there is some ambiguity for the name of this $thing, then I guess... the protocol is to talk that through and pick one?)

[ufundo]

+1 for switching to Site Email Address everywhere

[colemanw]

Agree with @ufundo. I consider "Domain" to be techie speak for "Website" or just "Site" so this PR as-is moves us in the direction of:

• Entity/Api name: DomainEmailAddress
• User-facing name: Site Email Address

All that's left is to cleanup the old references to "from_email_address"

[totten]

On the auto-build demo site, I tried registering a new address. The admin GUI seemed to work properly. 🎉

But then I went to "Mailings => New Mailing", and the page crashes:

[totten]

(@eileenmcnaughton) I feel like I might have a use case for adding name as in machine name to the table

Yeah, fair enough. We can do display_name here. But just looking toward the name contingency... the existing OptionGroup/OptionValue already has machine-name. And with the way it works, it tends to become misleading gibberish -- i.e. your typical/default record winds up with name="FIXME" <[email protected]> (which is uneditable and which contradicts the real data in the record).

There may exist some good use-case for machine name on an email address. I don't know, but assuming it exists... I'd probably set the default/computed name to something like md5(random_bytes(16)) (which may also be gibberish, but at least it's truthful gibberish 🙃). In that arrangement, someone using dev-tools could still assign a mnemonic machine-name for records they care about.

[totten]

OK, so I did some light/local r-run with an upgrade scenario. The idea is:

• Install 5.81
  • Add several addresses and exercise the various options (default, disabled, description, etc)
  • Click around some UI's
  • Call some of those common APIs. Dump output to a file.
• Upgrade to PR
  • Click around some UI's
  • Call some of those common APIs. Dump output to a file. Compare.

The main thing that stuck to me is the change re: ordering/weights. This isn't mentioned in the "Description", but (in usage) it was apparent on both the UI level and API level.

• UI: Before

  

• UI: After

  

• To me, the old UI feels overwrought -- I would feel silly fiddling with the weights on a record-by-record basis. Alphabetical ordering would be much better balance between (a) providing predictable behavior and (b) not-wasting-time-with-clickery-clackery.

  So to my mind: woot! let's remove weight! it's an anti-feature!

• OTOH, order/weight has been there forever; it's a common idiom in Civi-land; the API-mapping fails if the API-call involves weight; and there's usually some kind of history for why a feature like that exists. My main lament is that we have no good way to identify/learn-from the user who actually likes the weight functionality. But I guess if we can get a couple dev's nodding in agreement that weight is a terrible idea, then that'll be enough...

• Spot-checking how the ordering actually behaves in a few UIs:

  • "View Contact > Contribution > Send Receipt > Email": Works beautifully. Alphabetical. Default respected. Disable/enable flag respected.
  • "New Email (activity)": The items are alphabetical. However, it doesn't respect the is_default flag.
    • This may not be obvious if your configuration gives precedence to personal emails (allow_mail_from_logged_in_contact=1). However, the suitability of that flag is very subjective (depending on both marketing preferences and mail-server config). In the safe case of allow_mail_from_logged_in_contact=0, the default email addr is particularly important.
  • "New Mailing": Crashes, so can't say.
  • "Administer > Communications > From Email Addresses": Doesn't sort by default

[ufundo]

+1 for Site Email Address over Domain Email Address or From Email Address (commented in a thread above but just collating here)

+1 for display_name

Machine name field

I'm with @eileenmcnaughton that there could be all kinds of uses for machine named records.

@totten it tends to become misleading gibberish -- i.e. your typical/default record winds up with name="FIXME" [email protected]

I don't dislike that auto-generated machine names correspond to the initial display_name and email, and may deviate from it if the display name is updated. That's a standard behaviour in lots of places where we have name - label. Yes if you create a record with label = Red, name = red it's unfortunate that if you change the label to Blue your name is still red. But it's often useful evidence trail if you're coming in to try and diagnose what an overambitious admin has done to their config. And it's a human-level CRM problem for when it's appropriate to edit an existing record versus delete it and add a new one. The name behaviour is useful to keep name red if you change the label from Red to Rouge. In your "misleading gibberish" screenshot, I think its a feature that you can seee the record now known as Foo Foo is the originally (and persistently) named as Foo. There's a bit of subtlety here - but that's not inappropriate for a below-the-hood feature.

I do think it's unfortunate that you get FIXME in the starting one. Could we hard code a value for that one "initial"?

I also think it would be good (and consistent) to munge what goes into the name field (ie at least replace special chars with _, and maybe use some conventional separator for the display_name and the email?)

Having said all this - it looks like adding a name field could be a separate follow up PR? @colemanw was that you thinking?

[colemanw]

@ufundo it looks like adding a name field could be a separate follow up PR?

Definitely! I'm not about to make this PR any bigger than it already is.

@totten So to my mind: woot! let's remove weight! it's an anti-feature!

Exactly. Also, just to make the obvious point, weight was never intentionally added to this option group, it was inherited from the OptionValue table, just as the overwrought UI in your screenshot was inherited from the generic "edit option values" screen. This new table is more intentional about only including columns that are actually relevant.

Re the is_default flag being respected. I'll look but that might be a preexisting bug. None of the legacy functions I adapted were doing any sorting by is_default.

[colemanw]

I've rebased/pushed this PR to address the following

1. CiviMail page crash: This was due to incomplete refactoring of the crmFromAddresses angular service. I've decided to revert my refactoring and instead put more work into the api compat layer to ensure smooth backward-compat with things like weight. Tests added. Mailing screen now works as-before, and switching it to use the new api can wait for another day (it's easier said than done because that core angular code is reused by some extensions in universe).
2. Entity description reworded to 'Sender addresses to use for outbound emails.' per @totten's feedback.
3. Administer > Communications > From Email Addresses: Default sort added to show default first, then sort alphabetically.

I also looked into @totten's testing of the is_default flag behavior on email-related screens. I agree it's not working as expected in this PR, but then I build a vanilla site from master and got the exact same behavior. So the not-workingness is a pre-existing condition and I'm happy to take a look at it but am wary of bogging down this PR even more so I opened a new issue for that bug.

[totten]

Re-tested. Added a couple more tweaks. Merge on pass.