Bump hometown to Mastodon 4.2.9

[nachtjasmin]

Update 2023-12-29: It is ready for review!
Update 2023-12-30: I just deployed it on queer.group, so yes, this PR is already running in production.

This PR is a replacement for #1325, since I started from scratch. The last tag, v4.0.10+hometown-1.1.1 is used as the root for this PR.

For anyone who wants to support me, feel free: https://ko-fi.com/queergroup

---

Current TODO list:

• Add a database migration for the renaming of exclusive lists (hometown used is_exclusive, upstream uses just exclusive)
• Check whether the default styles need adaptions (they seem to work fine, haven't checked the pastel themes tho)
• Re-implement the Permalink component which was added in b8802af and was only removed by me (f825010) for easier conflict resolution
• Find out why the tests are failing

And then it's time for the integration tests.

[ClearlyClaire]

Chiming here to notify you that we updated the stable-4.2 branch with a few more backported fixes. Those should probably be merged before the release of the security fixes, but they seem to apply cleanly on your hometown-4.2-merge branch, and I don't think they should affect Hometown features.

Same thing with the security patches, I haven't checked whether they interfere with Hometown features, but I don't think they should, and they also apply cleanly, so merging them should not be an issue, and you should be on track for the release!

[nachtjasmin]

@ClearlyClaire Thank you for letting us know! I've merged the stable-4.2 branch into this again, went smooth as butter. 👍

[nachtjasmin]

I just realised that I accidentally leaked stuff from my personal fork into this branch. I don't know when it happened. Anyway, I removed said commits, sorry for any confusion caused by this.

[dariusk]

Oops! Can you give an example personal commit for me to look out for just to make extra sure it's not in the final release or my test builds?

[nachtjasmin]

af8b8ee would be one such example, I think it's in your branch as well. 🙈

[dariusk]

Okay I did a force update on https://github.com/hometown-fork/hometown/tree/hometown-4.2-merge -- going to attempt to fix the pastel themes since those are the main thing broken.

[dariusk]

@nachtjasmin Interesting -- it seems like no user themes (pastel or not) are saving state for me. Like I go to change my user set theme and no matter what I change it to it always reverts to Mastodon (Dark). But if I change the site theme that does change and save it for everyone. Can you reproduce this? Or is this just a problem for me...

[dariusk]

Hmm looks like a rollback is happening here when it tries to save the setting...

09:20:36 web.1     |   Setting Load (0.3ms)  SELECT "settings".* FROM "settings" WHERE (thing_type is NULL and thing_id is NULL) AND "settings"."var" = $1 ORDER BY "settings"."id" ASC LIMIT $2  [["var", "theme"], ["LIMIT", 1]]
09:20:36 web.1     |   ↳ app/models/setting.rb:28:in `block in []'
09:20:36 web.1     | Cache write: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     | Cache read: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     | Cache fetch_hit: rails_settings_cached/8d13a7825611181d5a70a3c9add28ef1/theme ({:compress=>true, :compress_threshold=>1024})
09:20:36 web.1     |   UserRole Load (0.1ms)  SELECT "user_roles".* FROM "user_roles" WHERE "user_roles"."id" = $1 LIMIT $2  [["id", 3], ["LIMIT", 1]]
09:20:36 web.1     |   ↳ app/models/user.rb:159:in `role'
09:20:36 web.1     |   TRANSACTION (0.1ms)  ROLLBACK
09:20:36 web.1     |   ↳ app/controllers/settings/preferences/base_controller.rb:7:in `update'
09:20:36 web.1     |   Rendering layout layouts/admin.html.haml

Am I reading this right... it thinks it hit the cache so it doesn't need to change anything? no, cache is fine, this is what it looks like on vanilla 4.2.9:

10:10:41 web.1     |   Setting Load (0.2ms)  SELECT "settings".* FROM "settings" WHERE (thing_type is NULL and thing_id is NULL) AND "settings"."var" = $1 ORDER BY "settings"."id" ASC LIMIT $2  [["var", "theme"], ["LIMIT", 1]]
10:10:41 web.1     |   ↳ app/models/setting.rb:28:in `block in []'
10:10:41 web.1     | Cache write: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     | Cache read: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     | Cache fetch_hit: rails_settings_cached/e23eceea1dbbecf09d779eb1d9007c03/theme ({:compress=>true, :compress_threshold=>1024})
10:10:41 web.1     |   UserRole Load (0.1ms)  SELECT "user_roles".* FROM "user_roles" WHERE "user_roles"."id" = $1 LIMIT $2  [["id", 3], ["LIMIT", 1]]
10:10:41 web.1     |   ↳ app/models/user.rb:159:in `role'
10:10:41 web.1     |   User Update (0.2ms)  UPDATE "users" SET "updated_at" = $1, "settings" = $2 WHERE "users"."id" = $3  [["updated_at", "2024-07-03 17:10:41.849632"], ["settings", "{\"theme\":\"default\",\"web.advanced_layout\":false,\"web.trends\":true,\"web.use_blurhash\":true,\"web.use_pending_items\":false,\"web.use_system_font\":false,\"web.disable_swiping\":false,\"web.delete_modal\":true,\"web.reblog_modal\":false,\"web.unfollow_modal\":true,\"web.reduce_motion\":false,\"web.expand_content_warnings\":false,\"web.display_media\":\"default\",\"web.auto_play\":false}"], ["id", 1]]
10:10:41 web.1     |   ↳ app/controllers/settings/preferences/base_controller.rb:7:in `update'
10:10:41 web.1     |   TRANSACTION (0.5ms)  COMMIT

See how it goes straight from the UserRole Load into User Update without a rollback...

[dariusk]

Oh I think I am starting to see... Mastodon introduced UserRole since v4.0. There is probably a missing migration or something where users lack an appropriate UserRole so the validation doesn't happen and triggers a rollback?

[dariusk]

So I have confirmed that this error only occurs when a server has been migrated from Hometown running v4.0.15 to the new v4.2.9 merge branch. If you install it clean it's fine, but something about the migration is an issue. After migration there is indeed a user_roles table, and it has the correct data, and the account being queried in the UserRole check even has the correct role_id so I am not sure what the problem is here...

[dariusk]

Lol okay this is a problem with email addresses on the default admin account!! In 4.0.15 it was admin@localhost:3000 and now it's like "that is not a valid email address". Fixing this now...

[dariusk]

Okay I changed my admin email to admin@localhost and now everything is saving just fine, siiiiigh. @ClearlyClaire, this branch now includes stuff from stable-4.2 so we are ready to apply the patch tomorrow. Since I haven't actually done a 4.2 release yet, I am just going to advise everyone to update to 4.2 tomorrow and do the release all at once.

[pronoiac]

It's been 16 months since Mastodon 4.1.0 dropped, and over 2k commits. Releasing today, maybe with a note about tomorrow's expected security release, could help surface any issues sooner. It could give the hosted services some room to work and do some indexing, too.

(Apologies if that's just my anxiety.)

[dariusk]

@pronoiac Hmm I see your point but I do not have physical availability to monitor a release until tomorrow morning anyway. I think it's more important that I'm around to help put out fires than to do a prerelease and disappear until a second actual release tomorrow morning. But if anyone wants to try out the prerelease, they can run this branch:

https://github.com/hometown-fork/hometown/tree/hometown-4.2-merge

It's currently running on Friend Camp just fine. Here is my draft installation guide for anyone who wants to try it:

Hometown v4.2.9+1.1.1 release (not a real release, I'm waiting for the security patch tomorrow)

This is a security release that keeps us up to date with Mastodon v4.2.9. Thank you once again to [@nachtjasmin](https://github.com/nachtjasmin) for the help with this merge.

Changelog

Please [see the Mastodon 4.2.0 release notes for details](https://github.com/mastodon/mastodon/releases/tag/v4.2.0). You can check the release notes up to 4.2.9 as well, though they are mostly bugfixes minus a big moderation change that I highlight below.

The really big changes for users are:

• there is now full-text search for public posts from all over the place where users have opted in to have their posts full-text-searchable. If you go to Preferences -> Public profile -> Privacy and reach you can OPT-IN your posts via the "Include public posts in search results" option.
• I can no longer maintain the custom web mobile UI with the retractable sidebar. It diverged too much from Mastodon and was really problematic to maintain. I'm going to see if I can do it "smarter" for a future release because tbh I really miss it but for now the mobile web interface is going to be more cluttered than #Hometown users are used to. Sorry about that

Upgrade steps

These are instructions for upgrading from Hometown v4.0.15+1.1.1.

As always, make sure you have backups of the database before performing any upgrades. A postgres backup command would look something like this:

pg_dump -Fc -U postgres mastodon_production > name_of_the_backup.dump

Dependencies

External dependencies have changed since v4.1.7, with the Ruby, PostgreSQL and Node.js minimum version being higher.

• Ruby: 3.0 to 3.2.3
• PostgreSQL: 10 or newer
• Elasticsearch (recommended, for full-text search): 7.x (OpenSearch should also work)
• LibreTranslate (optional, for translations): 1.3.3 or newer
• Redis: 4 or newer
• Node: 16 or newer
• ImageMagick: 6.9.7-7 or newer

Tip

If your uploaded images are broken after the upgrade, it means your installed ImageMagick version is older than the new minimum version (6.9.7-7), for example if you are running Ubuntu 18.04. If this happens, you can find more information and ways to fix it [on this page](mastodon#25776).

Moderation and registration changes

Important

This update changes registrations to be closed by default.

Running a social media platform where anyone can sign up without active moderation is dangerous.

We are changing the default, so that opening registrations is always a conscious choice. If you have never changed or saved the registrations mode yourself, this update will switch your server to not accepting new users. Simply change the setting again after the update if you wish to restore the old behavior.

Database replica configuration

The way Mastodon handles read replicas has changed, removing the makara gem and using native Rails support instead.

This changes how database replicas are configured. Instead of editing config/database.yml, you should use an unmodified one and use the REPLICA_DB_NAME, along with REPLICA_DB_USER, REPLICA_DB_PASS, REPLICA_DB_HOST and REPLICA_DB_PORT, if they differ from the primary database.

If you are using DATABASE_URL, you can configure your read replica in a similar way using REPLICA_DATABASE_URL.

If you aren't using database replicas or have no idea what this is about... don't sweat it, you're fine.

Streaming server changes

We have dropped built-in clustering support from the streaming server, which means, depending on the load you are facing, that you may need to run multiple instances of it and configure a load-balancer.

Unless you are using Docker, it is recommended that you update your mastodon-streaming unit scripts with the ones we provide:

1. sudo cp ~mastodon/live/dist/mastodon-streaming*.service /etc/systemd/system/
2. sudo systemctl daemon-reload
3. sudo systemctl restart mastodon-streaming

If you then need to run more than one mastodon-streaming server, you can:

1. Start a new instance with sudo systemctl start mastodon-streaming@port (e.g. mastodon-streaming@4001)
2. Edit your nginx configuration file to add the new server to the load-balancing (an example is provided in the comments in dist/nginx.conf)

Automatic update checking

Starting from this release, Mastodon will periodically check for updates by querying https://api.joinmastodon.org/update-check every 30 minutes in a background job.

That URL can be changed using the UPDATE_CHECK_URL environment variable, and the feature outright disabled by setting that variable to an empty string (UPDATE_CHECK_URL=).

Update steps

1. If you are using rbenv, [update the list of available versions](https://github.com/rbenv/ruby-build/wiki#updating-ruby-build) and install Ruby 3.2.3 by doing RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install in the Mastodon install directory (e.g. /home/mastodon/live)
2. Ensure that you are using Node.js v16 or newer -- I (Darius) actually encountered errors on 16 and needed to upgrade to v18 to make everything work.
3. Install dependencies: bundle install and yarn install --frozen-lockfile
4. Precompile the assets: RAILS_ENV=production bundle exec rails assets:precompile
5. Run the pre-deployment database migrations by specifying the SKIP_POST_DEPLOYMENT_MIGRATIONS=true environment variable: SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate
6. Restart all Mastodon processes
7. Run the post-deployment database migrations: RAILS_ENV=production bundle exec rails db:migrate
8. If you use Elasticsearch, rebuild the search indexes with RAILS_ENV=production bin/tootctl search deploy --reset-chewy

[graue]

Starting from this release, Mastodon will periodically check for updates by querying https://api.joinmastodon.org/update-check

Does that URL also know about Hometown releases? or should this be changed to account for running a fork?

[pronoiac]

What that api returns right now, prettified:

{
  "updatesAvailable": [
    {
      "version": "4.2.9",
      "type": "patch",
      "releaseNotes": "https://github.com/mastodon/mastodon/releases/tag/v4.2.9",
      "urgent": false
    }
  ]
}

[dariusk]

It does not - I can probably provide a new URL for Hometown releases but it wasn't priority for this security push.

…

On Wed, Jul 3, 2024, 6:57 PM James Young ***@***.***> wrote: What that api returns right now, prettified: { "updatesAvailable": [ { "version": "4.2.9", "type": "patch", "releaseNotes": "https://github.com/mastodon/mastodon/releases/tag/v4.2.9", "urgent": false } ] } — Reply to this email directly, view it on GitHub <#1337 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACBBVQKY5BFOTAME7EWOQLZKSTZTAVCNFSM6AAAAAA7O5HSMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBXHA3DENRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[reesericci]

Is there a clean path to migrate from this to v4.2.10? Just attempted it this morning and ran into a whole host of issues myself

[pronoiac]

@reesericci I suggest filing a new issue, and including details of the issues there

[reesericci]

I more just ended up hurting myself with git & bundler than anything specific (updated the dependencies too far, etc)

[nachtjasmin]

clean path to migrate from this

The recent Hometown release already includes the patch, so you should be fine following the instructions there.

This PR is also part of said release.

[reesericci]

That worked for me!