Dedupe location names against script-root

[robertbastian]

Fixes #5901

I'm not introducing script fallback (because time zone names are not meant to fall back by script), and I'm not introducing und-Xxxx locales, as those would increase data size (due to the other fields in the struct). Instead, I store a dedupe_locale in each data struct, which contains the locale that was deduplicated against. So for sr, this would be ru, because sr -> und-Cyrl -> ru, and for ru it'd be ru as well, and ru is complete. This way, we also directly know which dedupe locale to load; if we didn't store it we'd have to load a fallbacker and step through it until we hit und (and then use the previous value).

Depends on #5967

[dpulls]

🎉 All dependencies have been resolved !

[dpulls]

🎉 All dependencies have been resolved !

[sffc]

Fixes #5901

Do we not do similar deduplication in the other time zone keys as well?

[sffc]

So the problem with dedupe_locale is that the fallback engine has no knowledge that it needs to keep both payloads around. If you pass LocaleFamily("sr") into datagen, datagen is free to remove ru, but if we redefine the key to script fallback, it will know that it needs to retain und-Cyrl. (well, unless no-ancestor mode is used; but I would not block landing that since no-ancestor is less common and a power-user tool.)

[sffc]

Hmm. An approach that would work with our framework would be to have two keys:

1. location_base
2. location_override

Every locale has an entry in each key, but location_base has one unique payload per script, and location_override contains smaller payloads, which are empty if the data exactly equals the base.

[sffc]

Requesting changes because this code won't work with data slicing.

[robertbastian]

I do not want to change this to script fallback, because it should definitely do region fallback.

The two-key approach is pretty complex, the base keys would have to contain the intersection of all data structs that are deduped against the locale, which is hard to generate in our current framework (it's also not clear if grouping by script leads to the most deduping).

This currently has a data size improvement of <10%, I'm not sure it's worth pursuing at the moment.

[sffc]

I do not want to change this to script fallback, because it should definitely do region fallback.

Hm? Right now it is language fallback. Script fallback is identical except that it adds und-Script in the fallback chain.

The two-key approach is pretty complex, the base keys would have to contain the intersection of all data structs that are deduped against the locale, which is hard to generate in our current framework (it's also not clear if grouping by script leads to the most deduping).

Hm? I don't understand this, either. Generating the base key is trivial. It is equal to the language data for the likely language of the locale's script. We could do something fancier but that's not what I suggested above

This currently has a data size improvement of <10%, I'm not sure it's worth pursuing at the moment.

1'407'337 B to 1'297'285 B, or 110 kB.

It's about the raw data size improvement, yes, but also about language equity and not privileging English.

[sffc]

@robertbastian Can you clarify your comment about why you don't want to use script fallback?

[robertbastian]

Hm? Right now it is language fallback. Script fallback is identical except that it adds und-Script in the fallback chain.

I guess that's right.

Generating the base key is trivial. It is equal to the language data for the likely language of the locale's script.

It is not. en contains US-specific display names (like "CT") that en-GB should not fall back to. Nevermind, that's non-location. Maybe it is trivial but this is not a priority for me right now.