[FEATURE REQUEST] Mod content

[Terrorforge]

It would be great if the Guide was able to include content from the mainline mods, notably Aftershock and Magiclysm.

Ideally, it would be possible to select which mods you want to have "active", and mod content should definitely be clearly marked as such.

[nornagon]

Yep, I'd like to have this but there are some significant complications:

1. We need some UI for selecting what mods are or aren't enabled. There's not really any "config" place in the UI right now other than the version dropdown at the bottom of the page. Possibly mod config could go there?
2. Currently, the URL completely describes the page you see (version + type + id). If we support mod content, in order to preserve linkability, we'd need to include the list of enabled mods in the URL, which could get quite long.
3. Currently, the tests render every single page to check for errors before deploying. This becomes practically impossible with mods, because mods interact with each other combinatorially, so the number of possible combinations of enabled mods is so large that it would make it impossible to test every combination.
4. Mod content isn't as well-normalized as mainline content, so there are probably more data errors to deal with.

This is all on top of the base work of parsing & interpreting the mod content itself (+ any mod-specific types, such as spells in magiclysm).

Ultimately, I think this is largely solvable, but it is not a small amount of work, and it's not a priority for me right now.

[Fris0uman]

Quick note that spells are not specific to Magiclysm, there's a whole bunch of spells used by monsters in vanilla.

[nornagon]

Yeah, I'm aware, the guide already reads some spell data, e.g.

cdda-guide/src/types/Monster.svelte

Line 420 in db54842

{#if item.death_function.effect?.id && data.byId("SPELL", item.death_function.effect.id)}

but the features that Magiclysm uses aren't all used in core, so we don't support all of them.

[LyleSY]

+1 for DinoMod. I manually create this content for myself every now and then to make sure things are somewhat balanced, but having this as a resource would speed that process up a ton.

[nornagon]

I had some thoughts about some of the blockers for this.

1. For selecting what mods are enabled, I think the bottom bar where version + tileset are currently selected is the right place for some sort of mod selection UI to go.
2. For storing the mods in the URL, there are somewhere around ~30 mods in mainline. Leaving aside the problem of supporting non-mainline mods, we could represent each mod by a single bit in an N-bit integer, then encode that as base64, so the URLs would looks something like https://nornagon.github.io/cdda-guide/?v=0.G&mods=YWJjZAE#/item/cotton_patchwork, which I think is manageable.
3. For tests, I think it would be a reasonable trade-off to test every page with each single mod enabled. If there are failures with certain combinations of mods, I think that's obscure enough that it's okay to let those errors through.
4. We should just deal with any data errors that we encounter, either by being permissive in HHG or by fixing the data upstream :)

---

In order to represent the set of selected mods as an N-bit integer, we need to assign each mod a small integer. There are (at least) two ways to go about this:

1. Assign each mod a stable integer ID, e.g. DinoMod is 1, CrazyCataclysm is 2, and so on. Commit a hard-coded list of mod name => integer, and any mod that isn't in the list can't be selected in the HHG UI. The list would be manually maintained.
2. Each mod's ID is its position in the alphabetically-sorted list of mods in the current version. e.g. for the current version, Aftershock would be 1, BlazeIndustries would be 2, and so on.

Each of these approaches has trade-offs.

(1) is a higher maintenance cost, as the list of mods would be manually edited. Mods wouldn't be immediately available to select in the UI until someone added them to the list and gave them an ID. The size of the mod-set representation would grow without bound as mods are added, because the IDs of deleted mods wouldn't be able to be reused. If for example, over the next 4 years, 50 mods are added and 30 deleted, we would have an 80-bit-long string, which would look like YWJjYmFvZXVhYQ== in the URL. That's still pretty short, so that's maybe not a big problem.

However, with (1), since each mod has a stable ID, it would be possible to switch versions and have the mod-set stay the same. e.g. the URL /cdda-guide/?mods=c would always mean DinoMod, no matter what version was selected.

(2) would mean that when switching versions, the mod set would have to be recalculated, using data from both the old version and the new version, which would be a bit complicated to code. It would also mean that a URL with mods enabled would always have to have a version encoded, too, i.e. the URL /cdda-guide/?mods=c would be meaningless—it would have to be /cdda-guide/?v=0.F&mods=c in order to specify particular mods. As such, you wouldn't be able to bookmark a URL for "current experimental with this set of mods enabled"; you'd have to reselect the mod set every time.

However, mods would be automatically available in HHG as soon as they were available in experimental, and the size of the mod-set ID would be linear in the number of mods available in mainline.

---

Regardless of which numbering scheme is chosen, the mod data needs to be added to https://github.com/nornagon/cdda-data. Currently, cdda-data exposes the full game JSON as a single JSON file. Mainline content is ~3MB of gzipped JSON, mod content is ~1MB, so it'd be about a 25% increase if they were packed together.

Options here are:

1. Pack everything together, accept the 1MB of extra gzipped-JSON + extra parsing time for another 9MB of JSON. This would slow down boot time, but simplify code for enabling mods, as it wouldn't have to re-fetch anything from the network. It would also mean that if you had already downloaded mainline data and then went offline, enabling mods would work without having to download anything new.
2. Split out the mod data from the mainline data, so there would be two bundles: mainline and all-the-mods. Download the mod bundle only if any mod is selected.
3. Split out each mod into its own file, download only the mods that are selected.

I'm thinking probably (1) is the right way to go for now. It will slow down the initial load, but that's already pretty slow, and we can possibly do some other things to ease the burden. For example, I've been thinking about how we might be able to move the database into a shared worker, so that opening new tabs won't require reloading the full database in each separate window.

[waveyl]

Maybe just display all the mod content? Mod content is not large compared to the vanilla. And for the overrided, switch version in its page to check out the differences with vanilla

[nornagon]

@waveyl it’s unfortunately not so simple, as some mods remove content (eg. Innawoods). So for example the locations you might find something in will be different depending on what mods are enabled.

[RDru]

Make separate entries for each variation. Put mods name in some kind of brackets. Don't over complicate this. I believe most site users are smart enough to figure it out how to browse it.

[Procyonae]

Rather than hardcoding the mod list couldn't each mod be assigned the next number available on the first version it appears in so that there's no maintenance needed (at least until the hypothetical time that the total number of mods to ever have existed exceeds your integer limit) while still preserving the numbers between versions?

[Procyonae]

If you can point me at stuff that would need changing to accommodate this I'd like to at least attempt bits of it, with this being the sole always up to date resource for the game solving this would be pretty big for the game both from playing and modding perspectives.

[nornagon]

Rather than hardcoding the mod list couldn't each mod be assigned the next number available on the first version it appears in so that there's no maintenance needed (at least until the hypothetical time that the total number of mods to ever have existed exceeds your integer limit) while still preserving the numbers between versions?

That's a pretty good idea. It would need a bit of cleverness in cdda-data.

The first thing to note is that cdda-data doesn't keep every historical version, because it would run out of disk space trying to git clone. (This is something I since figured out how to work around with this technique but haven't applied it to cdda-data yet.) So re-building the full mod-number-index on every update in a stateless way won't work, because we don't have the full mod history. (And also it could be kinda slow to read every historical version to figure out which mods got added in which order.)

So we need to keep some state around. That is, every time we update cdda-data, we'd need to look over every version older than $X, check each one in chronological order to see if any new mods were added, and if so, assign them a number and save them in a state file in the repo along with the new value of $X. The state file would look something like:

{
  "latestVersion": "cdda-experimental-2024-05-10-0048",
  "mods": [
    "aftershock",
    "backrooms",
    // ...
  ]
}

where the index in the mods list gives the bit of the integer to use.

---

All this said, perhaps I overindexed on "represent a lot of mods in a small URL". What if we just put the list of mod IDs in the URL? e.g.

https://cdda-guide.nornagon.net/?v=0.G&mods=magiclysm,megafauna

This is human-readable, human-editable and will be stable over time (since mod IDs are stable). It's longer in the URL, but do we really need to optimize for the case of someone enabling 20 different mods? I think the 95% use case is one or two mods enabled.

[nornagon]

Regardless of which numbering scheme is chosen, the mod data needs to be added to https://github.com/nornagon/cdda-data. Currently, cdda-data exposes the full game JSON as a single JSON file. Mainline content is ~3MB of gzipped JSON, mod content is ~1MB, so it'd be about a 25% increase if they were packed together.

Options here are:

1. Pack everything together, accept the 1MB of extra gzipped-JSON + extra parsing time for another 9MB of JSON. This would slow down boot time, but simplify code for enabling mods, as it wouldn't have to re-fetch anything from the network. It would also mean that if you had already downloaded mainline data and then went offline, enabling mods would work without having to download anything new.
2. Split out the mod data from the mainline data, so there would be two bundles: mainline and all-the-mods. Download the mod bundle only if any mod is selected.
3. Split out each mod into its own file, download only the mods that are selected.

I'm thinking probably (1) is the right way to go for now. It will slow down the initial load, but that's already pretty slow, and we can possibly do some other things to ease the burden. For example, I've been thinking about how we might be able to move the database into a shared worker, so that opening new tabs won't require reloading the full database in each separate window.

Since writing this, the compressed size of all mod data has nearly tripled to 2.7 MB, and the amount of JSON to be parsed up to 18 MB. That's about a 50% premium on the base data. This is steering me away from (1), accepting a 50% worse boot time for everyone regardless of mods enabled seems bad, load time is crappy already. So that leads us to (2), split the bundles into main and mods, and only load the mods bundle if a mod is enabled. We could further hack this later on to only parse the JSON for mods that are loaded (even though we'll still download the whole bundle). It means you won't be able to enable mods while offline if you didn't enable them while online, but I think that's a fairly small price to pay for preserving speedier boot times.

The next step here would be to update cdda-data to collect and store mod data in addition to mainline data. The existing logic for collecting mainline data starts here if you want to take a stab at a PR.

[LyleSY]

Would it be helpful to only include mods that are not obsolete? That should reduce the bloat quite a bit

[nornagon]

@LyleSY no, that only reduces the gzipped size to about 2.4 MB which isn't really enough of a difference to change the course of action. So we might as well keep them.