Platform Request: Hetzner #1324

aral · 2022-10-19T08:49:29Z

bgilbert · 2022-10-19T16:34:06Z

Thanks for filing this. Note that we'll need these answers from the OS perspective, not the user perspective. E.g., how does the OS fetch userdata? How does it learn its own hostname?

bgilbert · 2022-10-19T16:38:23Z

Some previous discussion in:

Apparently the platform is sometimes called hcloud.

jlebon · 2022-10-19T18:19:51Z

We discussed this in today's community meeting:

13:10:54 < jlebon> #agreed we would like to add support for Hetzner. we are looking
                   for volunteers to pick it up and push it forward.

aral · 2022-10-19T20:03:52Z

Happy to hear it. Would you like me to ask around to see if I can find some contacts there or do you already have folks you can talk to?

bgilbert · 2022-10-20T03:12:54Z

Yes, that would be helpful, thanks!

aral · 2022-10-21T13:49:32Z

Quick update: I’ve been in touch with an engineer at Hetzner:

“I passed it to the responsible people but they are on vacation for the next two weeks … I will force it to be answered then :)”

der-On · 2022-10-25T18:02:06Z

At least for reseting passwords a "QEMU Guest Agent" is running on the OS.

bgilbert · 2022-10-30T03:46:27Z

@der-On That might be the standard QEMU one, discussed in #74. We generally avoid shipping third-party agents (and reimplement pieces of them when necessary to avoid it), and don't currently ship the QEMU one.

asciiprod · 2022-11-07T15:16:57Z

So, how could we help to get CoreOS running on Hetzner Cloud?

bgilbert · 2022-11-08T04:59:05Z

@asciiprod Thanks for joining in! We could use some help answering the questions at the top of this issue. We have some answers already in the old Container Linux PRs, but it'd be good to make sure our understanding is up to date.

asciiprod · 2022-11-08T06:15:00Z

Sure, I'll try to answer them as good as I can:

What is the official name of the platform? Is there a short name that's commonly used in client API implementations?
That's already a tricky one. It is the Cloud product of the company Hetzner Online GmbH. It is usually just called Hetzner. That is also the name used for the cloud-init datasource. Other implementations like terraform or ansible use hcloud. Since ignition is more like cloud-init, I guess it should be hetzner.
How can the OS retrieve instance userdata? What happens if no userdata is provided?
We provide a meta/userdata endpoint at http://169.254.169.254/hetzner/v1. Userdata is optional, metadata is not. So if the endpoint would be absent cloud-init would run to a certain degree using DMI information, but fail to retrieve essential data like SSH-keys.
Does the platform provide a way to configure SSH keys for the instance? How can the OS retrieve them? What happens if none are provided?
Yes, via metadata endpoint. If no SSH key is selected at instance creation, a password hash is provided. If neither is retrieved/configured, the instance has no fallback login password.
How can the OS retrieve network configuration? Is DHCP sufficient, or is there some other network-accessible metadata service?
IPv4 configuration is provided via DHCP, IPv6 currently via metadata service only.
In particular, how can the OS retrieve the system hostname?
via metadata service.
Is there a mechanism for the OS to report to the platform that it has successfully booted? Is the mechanism required?
No, there isn't one and so not required.
Does the platform have an agent that runs inside the instance? Is it required? What does it do? What language is it implemented in, and where is the source code repository?
There is the qemu guest agent, which is used to reset passwords. There is also a package called hc-utils (https://github.com/hetznercloud/hc-utils). This contains bash scripts/systemd services/udev rules to automatically mount additional blockstorage volumes or start a DHCP client for new or unconfigured network interfaces. These are purely for ease of use and not required.
How are VM images uploaded to the platform and published to other users? Is there an API? What disk image format is expected?
Currently there is no direct upload option. Standard images are provided and updated by Hetzner on a regular basis. For customers to deploy their own images, there are two options: ISO or via rescue system. The latter is a Debian-based live Linux and allows to write anything to the virtual disk. Using snapshots this image can be used by a customer to create new instances.
If CoreOS would work out of the box, we could add it to the list of standard images as we already have Fedora there.
Are there any other platform quirks we should know about?
Intel-based instances are currently using i440fx and AMD-based are Q35. Both legacy. UEFI possible, but not exposed via API (yet). No secure boot.

lucab · 2022-11-08T10:17:16Z

@asciiprod thanks for the detailed feedback! Some additional thoughts from my side:

it would be great to have an official Hetzner URL as the canonical documentation for the userdata/metadata endpoint and its API
for the default user setup, I'd personally would prefer to stay away from the "fallback to password hash". Ignition can still set up SSH keys or other credentials from its configuration, without having to transmit password hashes over HTTP.
as we are starting from scratch, it would be great to opt-in "UEFI only" for Fedora CoreOS images (even if without SecureBoot). That would spare many future troubles.
what disk format would you need for importing our images? Is that documented somewhere? Is your import flow adaptable to our release process (three parallel streams, each one releasing at least every two weeks)?
are there ARM aarch64 instances too? Do they work the same way?

asciiprod · 2022-11-08T11:01:37Z

The canonical documentation is available at:
https://docs.hetzner.cloud

If we want to include the image on the platform, it must support passing a password hash as instance creation does not force selecting an SSH key.

Using UEFI-only for given image is something that is currently not implemented. I'd have to check internally if we could do that.

The internal workflow for images does not import any external disk images. Hetzner Cloud images are generated by automated installations (e.g. kickstart/subiquity) from distribution ISOs using packer & ansible. This leads to a compressed (zstd) raw disk image, which is uploaded as an image snapshot and used to test and validate the new build on the platform. That is the point were it could be possible to import a pre-build external disk image. However that would have to be discussed internally, if it is acceptable to open this process up for 3rd party generated images.

From a release and support point of view, I think we could only support the stable version.

Currently no aarch64 Cloud instances, but as we offer Ampere dedicated servers, that's something I would keep on the list and I'd say they work the same way (probably UEFI-only)

lucab · 2022-11-08T11:11:56Z

Ah great, thanks. The page I was looking for is https://docs.hetzner.cloud/#server-metadata (though it doesn't currently cover the userdata part).

bgilbert · 2022-11-08T17:28:39Z

Thanks for the detailed info, this is very helpful!

If we want to include the image on the platform, it must support passing a password hash as instance creation does not force selecting an SSH key.

I don't think we should support this. Fedora CoreOS tries to encourage the use of best practices, and passwords aren't that. On other platforms, Fedora CoreOS instances are usually configured with an SSH key passed in the Ignition config.

From a release and support point of view, I think we could only support the stable version.

We always recommend that users run some testing and next instances alongside their stable instances to help us catch regressions before they're promoted to a stable release. Thus, those streams are an important part of any Fedora CoreOS deployment strategy. It's entirely reasonable for Hetzner not to provide customer support for those streams, but it's important that they be available alongside stable. If that isn't possible, I think we shouldn't pursue adding stable either, and either only document the custom deployment flow or not document Hetzner Cloud at all.

asciiprod · 2022-11-09T08:07:58Z

I totally agree that SSH keys should be used and we also strongly recommend it during instance creation. But we do offer a password fallback for the existing OS images. So if CoreOS does not support it, we would need to enforce it.

In any case the more CoreOS specific changes we would need to make, the more difficult it becomes to adopt it for Hetzner Cloud.

jlebon · 2022-11-16T17:35:08Z

I totally agree that SSH keys should be used and we also strongly recommend it during instance creation. But we do offer a password fallback for the existing OS images. So if CoreOS does not support it, we would need to enforce it.

In any case the more CoreOS specific changes we would need to make, the more difficult it becomes to adopt it for Hetzner Cloud.

IIUC from the docs, it seems like the password hash is injected into the user-data, which is assumed to be a cloud-init config. Is that correct? I derived this from the fact that there's no entry for it in the Server Metadata section. (Aside: it seems like that section is missing an entry for public-keys, no?)

If that's the case, that logic would have to learn to support Ignition configs too. Password authentication is disabled by default on FCOS, so it would have to inject a drop-in for it. Also, the default sshd config (at least on Fedora) prohibits password authentication for the root user so it would have to undo that too.

What happens if no SSH keys are provided and the user-data isn't a cloud-init config? Does the API return an error because it doesn't know how to inject a root password? That seems like acceptable behaviour for the time being and avoids adding anything FCOS-specific.

asciiprod · 2022-11-17T09:00:07Z

The metadata API provides either the user-selected SSH-key or a random generated password hash if no SSH-key is selected. So instance creation will always succeed.
I have to apologize for the incomplete docs. The metadata service has of course a field/path for the public-keys and network-config.
Please correct me if I am wrong. As far as I understand it, we are currently only missing an afterburn provider to make CoreOS work on our platform. If that is correct, having it would enable us and anyone else to start using/testing it.
And it would also allow to resolve the other questions (password support, UEFI-only, releases) separately and step by step.

lucab · 2022-11-17T10:04:49Z

Yes, if we want to start making incremental progresses on this then the next immediate things to sort out on FCOS side are:

pick up and document a platform identifier (see https://coreos.github.io/ignition/supported-platforms/)
make Afterburn and Ignition aware of this new platform (see providers: add support for hetzner cloud afterburn#125 and 🇩🇪 Add support for Hetzner cloud ignition#1262)

aral · 2022-12-09T11:46:34Z

Hey everyone (@asciiprod, @lucab), any progress on this?

It would be really amazing to be able to boot up a Fedora CoreOS instance on Hetzner in under a minute (that’s how fast the supported instances boot up; it’s a game-changer for Small Web use) :)

aral · 2023-02-12T11:22:59Z

Hey folks, any updates on this? Would still love to see it happen. Has communication between Fedora and Hetzner stalled? If so, how do we get it going again? :)

bgilbert · 2023-02-13T22:42:49Z

@aral I think this thread has all of the needed information now, or at least most of it. There are some old Afterburn and Ignition PRs that'll need a rebase and an update based on the information here. I don't think anyone is currently working on that, but feel free to run with it if you'd like!

travier · 2023-09-04T17:10:44Z

I've created a PR with the "simplified" steps to add a new platform: #1562

travier · 2023-09-04T17:15:15Z

Ignition PR: coreos/ignition#1707
Afterburn PR: coreos/afterburn#996

travier · 2023-10-04T10:26:26Z

Folks interested for initial support for this platform in Fedora CoreOS should open an issue with the emerging platform template and follow the steps there. Thanks!

aral · 2024-02-10T13:29:40Z

Any updates on this for 2024?

I can’t imagine how launching a CoreOS installation on Hetzner’s cloud in under a minute would be bad for either Fedora or Hetzner. (Not to mention that this would have the Small Web launch on CoreOS instead of Ubuntu as that’s really the only option I see at the moment otherwise for an affordable platform with instance creation measured in the seconds.)

Anyone know what’s blocking this and how we can try and route around it?

nachtjasmin · 2024-05-21T08:28:03Z

@aral There's a really good guide by @swick that explains how to install Fedora CoreOS on Hetzner servers. It's not as easy as the other operating systems provided by Hetzner, but it's a good enough workaround until they provide official support.

aral · 2024-05-21T08:55:55Z

@nachtjasmin Thanks, Jasmin, that is a good guide indeed. Sadly, for my needs (we will eventually have thousands of servers), that isn’t good enough so I’ve decided to go with AlmaLinux on Hetzner instead. It doesn’t automatically update like CoreOS, sadly, which would have been my first choice, but eight years of security updates should give us enough time to either implement a major version update system or transition to a transactional OS later.

aral · 2024-05-21T08:56:45Z

Since this doesn’t look like it’s going to be implemented and since I’m moving ahead with using a different OS, I’m closing this. Please feel free to reopen if anything changes.

thomasaull · 2024-05-21T08:58:35Z

@aral Why not just leave it open, since it’s not solved yet?

aral · 2024-05-21T09:02:26Z

@thomasaull I’ll leave that decision to the Fedora CoreOS folks. They can reopen it if they decide to work on it. It’s been open for over two years, there’s no reason to keep it open longer in my view.

thomasaull · 2024-05-21T09:04:43Z

@aral Got it. Just out of curiosity: What exactly is the issue with the snapshot approach? Boot duration too long?

aral · 2024-05-21T09:10:02Z

@thomasaull It’s too convoluted and specific to Hetzner. I don’t want to tie Domain so closely to one provider, even if Hetzner is the one we’re initially going to be supporting and to have a hacky workaround be the core way that servers are deployed for the Small Web.

Also, hopefully, we (Small Technology Foundation) won’t be the only ones running Domain instances – other organisations around the world will so it’s just not feasible to base such a system on a workaround.

(Boot duration isn’t the issue as Domain now uses prewarmed instances.)

In the future, once we have more resources, etc., we can maybe review the decision.

Hope that helps give some insight into my, admittedly rather unconventional, needs :)

thomasaull · 2024-05-21T09:11:29Z

@aral Thanks for the insights! I'll read up on Domain/Small Web

aral · 2024-05-21T09:13:09Z

@thomasaull If you’re going to, the end-to-end encrypted Kitten chat (https://ar.al/2023/02/20/end-to-end-encrypted-kitten-chat/) and Streaming HTML (https://ar.al/2024/03/08/streaming-html/) posts/videos should give you a good idea of where everything is. It’s a new stack, specifically for a peer-to-peer web (Small Web) 💕

Manawyrm · 2024-05-21T12:20:41Z

Since this doesn’t look like it’s going to be implemented

One of the biggest roadblocks currently is the UEFI requirement.
We would love to have UEFI by default for everyone as well, but it would break the existing customers to roll this out for the current products (as new VMs with existing OS images might not boot in UEFI mode, if the image was created on a BIOS machine).

We also don't really want to offer a server image that doesn't boot in legacy BIOS mode (which then wouldn't boot on the older machine types).

travier · 2024-07-09T09:44:59Z

With support in Afterburn and Ignition now in stable, it should be possible to convert a QEMU FCOS image using the script in coreos/fedora-coreos-docs#651 to an Hetzner one and use it to setup FCOS on Hetzner.

Testing welcomed! If successful, we should document that in the docs.

Initial documentation to setup FCOS on Hetzner. Inspired by: https://www.flatcar.org/docs/latest/installing/cloud/hetzner/ See: coreos/fedora-coreos-tracker#1324

travier · 2024-07-16T10:14:42Z

While we do not yet provide ready made images for Hetzner, I've written documentation on how to setup Fedora CoreOS on Hetzner with what we have available right now: coreos/fedora-coreos-docs#654

Testing and feedback welcomed!

dustymabe · 2024-07-17T13:22:58Z

While we do not yet provide ready made images for Hetzner

What's preventing this last piece ^^?

Looks like from the docs PR you are just changing the platform ID, is that it?

travier · 2024-07-17T16:25:32Z

Looks like from the docs PR you are just changing the platform ID, is that it?

Yes, that's the only bit missing. If there are no objections then we could start building those and that would definitely make it easier to provision an instance.

yasminvalim · 2024-07-24T17:33:57Z

We discussed this issue on FCOS community meeting today and agreed that we will start producing Hetzner images for Fedora CoreOS.

gaufde · 2024-09-09T02:45:31Z

We discussed this issue on FCOS community meeting today and agreed that we will start producing Hetzner images for Fedora CoreOS.

Does this mean that FCOS will become a 1-click install option on Hetzner Cloud?

jbtrystram · 2024-09-09T07:09:19Z

Does this mean that FCOS will become a 1-click install option on Hetzner Cloud?

No, we will produce disk images that you will have to upload to hetzner, that's the best we can do

gaufde · 2024-09-09T07:12:28Z

Does this mean that FCOS will become a 1-click install option on Hetzner Cloud?

No, we will produce disk images that you will have to upload to hetzner, that's the best we can do

No worries! That still sounds easier than the recovery mode work-arounds I keep seeing

aral added area/platforms kind/enhancement labels Oct 19, 2022

jlebon added the meeting topics for meetings label Oct 19, 2022

jlebon removed the meeting topics for meetings label Oct 19, 2022

jlebon added status/decided status/pending-action Needs action labels Oct 19, 2022

prestist added the jira for syncing to jira label Dec 12, 2022

mhutter mentioned this issue Sep 4, 2023

providers/hetzner: add support for Hetzner Cloud coreos/afterburn#996

Merged

travier mentioned this issue Sep 7, 2023

Add emerging platforms process/flow #1569

Closed

aral closed this as completed May 21, 2024

jlebon reopened this May 21, 2024

travier mentioned this issue Jul 9, 2024

Document ways to contribute to FCOS coreos/fedora-coreos-docs#650

Open

travier added a commit to travier/fedora-coreos-docs that referenced this issue Jul 16, 2024

Provisioning: Add Hetzner

ce16674

Initial documentation to setup FCOS on Hetzner. Inspired by: https://www.flatcar.org/docs/latest/installing/cloud/hetzner/ See: coreos/fedora-coreos-tracker#1324

travier mentioned this issue Jul 16, 2024

Provisioning: Add Hetzner coreos/fedora-coreos-docs#654

Open

travier added the meeting topics for meetings label Jul 17, 2024

yasminvalim removed the meeting topics for meetings label Jul 24, 2024

Platform Request: Hetzner #1324

Platform Request: Hetzner #1324

Comments

aral commented Oct 19, 2022 • edited by bgilbert Loading

bgilbert commented Oct 19, 2022

bgilbert commented Oct 19, 2022

jlebon commented Oct 19, 2022

aral commented Oct 19, 2022

bgilbert commented Oct 20, 2022

aral commented Oct 21, 2022

der-On commented Oct 25, 2022

bgilbert commented Oct 30, 2022

asciiprod commented Nov 7, 2022

bgilbert commented Nov 8, 2022

asciiprod commented Nov 8, 2022 • edited Loading

lucab commented Nov 8, 2022

asciiprod commented Nov 8, 2022

lucab commented Nov 8, 2022

bgilbert commented Nov 8, 2022

asciiprod commented Nov 9, 2022

jlebon commented Nov 16, 2022

asciiprod commented Nov 17, 2022

lucab commented Nov 17, 2022

aral commented Dec 9, 2022 • edited Loading

aral commented Feb 12, 2023

bgilbert commented Feb 13, 2023

travier commented Sep 4, 2023

travier commented Sep 4, 2023

travier commented Oct 4, 2023

aral commented Feb 10, 2024

nachtjasmin commented May 21, 2024

aral commented May 21, 2024 • edited Loading

aral commented May 21, 2024

thomasaull commented May 21, 2024

aral commented May 21, 2024

thomasaull commented May 21, 2024

aral commented May 21, 2024 • edited Loading

thomasaull commented May 21, 2024

aral commented May 21, 2024 • edited Loading

Manawyrm commented May 21, 2024

travier commented Jul 9, 2024 • edited Loading

travier commented Jul 16, 2024

dustymabe commented Jul 17, 2024

travier commented Jul 17, 2024

yasminvalim commented Jul 24, 2024

gaufde commented Sep 9, 2024

jbtrystram commented Sep 9, 2024

gaufde commented Sep 9, 2024

aral commented Oct 19, 2022 •

edited by bgilbert

Loading

asciiprod commented Nov 8, 2022 •

edited

Loading

aral commented Dec 9, 2022 •

edited

Loading

aral commented May 21, 2024 •

edited

Loading

aral commented May 21, 2024 •

edited

Loading

aral commented May 21, 2024 •

edited

Loading

travier commented Jul 9, 2024 •

edited

Loading