Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OEM: Provide Proxmox images #1783

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

OEM: Provide Proxmox images #1783

wants to merge 1 commit into from

Conversation

pothos
Copy link
Member

@pothos pothos commented Mar 22, 2024

This pulls in flatcar/bootengine#91
and flatcar/init#115 to run afterburn for
hostname, network, SSH key, and metadata attribute setup. The afterburn
support for the SSH key and hostname parses the user-data when it's
cloud-init. The coreos-cloudinit support is not there but can be added
in addition: We need to add a new provider that varies from the existing
config drive support because the file is called user-data and not
user_data, and it needs to look for a filesystem label cidata and not
config-2.

Related to: flatcar/Flatcar#1040

How to use

Follow-up: fix coreos-cloudinit as mentioned above, and better don't make it work through the mount unit trigger but explicitly

Testing done

Jenkins

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

TODO: bump init and bootengine once related PRs are merged

Copy link

github-actions bot commented Mar 22, 2024

Build action triggered: https://github.com/flatcar/scripts/actions/runs/11589233717

@arcln
Copy link

arcln commented Apr 24, 2024

Hello, there is an existing Igntion PR that should work at coreos/ignition#1790. Could you include the Ignition PR into this one so we can do further testing?

@pothos
Copy link
Member Author

pothos commented Apr 24, 2024

Hello, there is an existing Igntion PR that should work at coreos/ignition#1790. Could you include the Ignition PR into this one so we can do further testing?

This needs to be reworked first and we have to drop the afterburn integration in Flatcar as the afterburn PR relies on cloud-init userdata and because Flatcar will run coreos-cloudinit anyway if it doesn't see Ignition user-data we would have it processed twice. For the network config that can be quite confusing because the afterburn network unit would only support a subset.

@pothos pothos force-pushed the kai/proxmox-support branch 2 times, most recently from 64b0e74 to 7c833ba Compare May 6, 2024 13:19
@pothos
Copy link
Member Author

pothos commented May 6, 2024

Triggered a new build with the Ignition PR included

@arcln
Copy link

arcln commented May 27, 2024

@pothos hey, it looks like the build never started, can you take a look please ? (cc @plgueugnon)

@tormath1
Copy link
Contributor

@pothos hey, it looks like the build never started, can you take a look please ? (cc @plgueugnon)

@arcln hey, pothos is AFK for now. I'll take over this PR. A CI build has been triggered :)

@tormath1
Copy link
Contributor

@arcln
Copy link

arcln commented May 29, 2024

Thank you, I tried the image and it didn't work because of a problem in the ignition PR (see coreos/ignition#1790 (comment)). Can you please run a new build when the PR is fixed ? Thanks

@tormath1
Copy link
Contributor

Thank you, I tried the image and it didn't work because of a problem in the ignition PR (see coreos/ignition#1790 (comment)). Can you please run a new build when the PR is fixed ? Thanks

@arcln ah I see, I already got this issue once. I started a new build with an updated patch: https://github.com/flatcar/scripts/pull/1783/files#diff-c65e2140094c80c4b4ce5c7c8494b6e4e72b78ef0692d236ffee1fb3932e11c5R157

@arcln
Copy link

arcln commented May 30, 2024

great. it seems that the build needs approval once again @tormath1

@tormath1
Copy link
Contributor

@arcln it builds directly on our Jenkins (github actions only build for qemu targets) - I'll let you know once it's done.

@tormath1
Copy link
Contributor

@arcln image is available here: http://bincache.flatcar-linux.net/images/amd64/9999.0.102+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2

@fhemberger
Copy link

@tormath1 Any news on this PR? Afterburn and ignition PRs have been merged.

@abuisine
Copy link

abuisine commented Oct 7, 2024

@tormath1 Any news on this PR? Afterburn and ignition PRs have been merged.

We are waiting for afterburn (done) and ignition (yet to happen) releases.
I am a bit puzzled by the release delay on the ignition side. Does someone know where to notify them about our dependency ?

@tormath1
Copy link
Contributor

tormath1 commented Oct 21, 2024

@fhemberger @abuisine Afterburn with Proxmox support is already available in the last Flatcar Alpha release: https://www.flatcar.org/releases#release-4116.0.0. For Ignition, we are waiting for upstream release, I see they are working on promoting a new Ignition stable spec: coreos/ignition#1922 I assume a release will follow.

EDIT: As expected an Ignition release is incoming: coreos/ignition#1960

@fhemberger
Copy link

@tormath1 I don't see it mentioned in the release notes for 4116.0.0, also no proxmoxve image file under downloads.

@tormath1
Copy link
Contributor

@tormath1 I don't see it mentioned in the release notes for 4116.0.0, also no proxmoxve image file under downloads.

@fhemberger No PromoxVE images yet because it's the purpose of this PR. As said, we are waiting for Ignition release before going ahead with this PR. I mentioned the release 4116.0.0 for the Afterburn part.

@tormath1
Copy link
Contributor

tormath1 commented Oct 24, 2024

🔈 Proxmox users - Ignition and Afterburn are now released and ingested into Flatcar. I need your help to test this freshly built image: http://bincache.flatcar-linux.net/images/amd64/9999.9.100+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2 using this documentation: https://github.com/flatcar/flatcar-website/pull/337/files (or any other method)

@abuisine @arcln @fhemberger @mcbenjemaa

@fhemberger
Copy link

fhemberger commented Oct 25, 2024

@tormath1 Tried to test the image following the given documentation, fails to start:

[   57.120718] localhost systemd[1]: Finished ignition-mount.service - Ignition (mount).
[   62.633185] localhost coreos-metadata[704]: Error: failed to run
[   62.633185] localhost coreos-metadata[704]: Caused by:
[   62.633185] localhost coreos-metadata[704]:     0: fetching metadata from provider
[   62.633185] localhost coreos-metadata[704]:     1: maximum number of retries (3) reached
[   62.633185] localhost coreos-metadata[704]:     2: failed to mount (read-only) source '/dev/disk/by-label/cidata' to target '/tmp/afterburn-Kk3KoQ', with type 'iso9660'
[   62.633185] localhost coreos-metadata[704]:     3: ENOENT: No such file or directory

Here is the complete dump of rdsosreport.txt

Ignition config is stored via qm set $VM_ID --cicustom "user=local:snippets/user-data". (Took it from another Flatcar VM I tested before.)

EDIT: It was a documentation issue, one important line was missing which activates the cloud-init support in Proxmox VE.

@fhemberger
Copy link

@tormath1 Added qm set $VM_ID --ide2 local:cloudinit, restarted the VM and the provisioning works! 🎉

@tormath1
Copy link
Contributor

@tormath1 Added qm set $VM_ID --ide2 local:cloudinit, restarted the VM and the provisioning works! 🎉

Great news, we should add this in the documentation. Thank you so much for testing and investigating this. So Ignition provisioning is working and what about coreos-metadata.service ? Do you have some metadata in /run/metadata/flatcar ? And what about flatcar-digitalocean-network.service ? (the name is legacy, it has nothing to see with Digital Ocean).

@fhemberger
Copy link

@tormath1 I already amended the documentation PR, this is the process for me:
flatcar/flatcar-website#337 (review)


what about coreos-metadata.service? Do you have some metadata in /run/metadata/flatcar?

core@flatcar ~ $ systemctl status coreos-metadata.service
○ coreos-metadata.service - Flatcar Metadata Agent
     Loaded: loaded (/usr/lib/systemd/system/coreos-metadata.service; disabled; preset: disabled)
     Active: inactive (dead)
  Condition: start condition unmet at Fri 2024-10-25 10:39:50 UTC; 24min ago
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=ec2 was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=ec2 was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=azure was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=azure was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=digitalocean was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=digitalocean was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=gce was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=gce was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=openstack was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=openstack was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=packet was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=packet was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=scaleway was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=hetzner was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=kubevirt was not met
             └─ ConditionKernelCommandLine=|flatcar.oem.id=akamai was not met

The coreos-metadata.service is missing a line ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve, that's why /run/metadata/flatcar wasn't created.

I ran the script manually, after that the metadata was present:

core@flatcar ~ $ sudo /usr/bin/coreos-metadata --cmdline --attributes=/run/metadata/flatcar
Oct 25 11:11:21.706 WARN user-data does not have the expected header `#cloud-config`, ignoring this file

core@flatcar ~ $ cat /run/metadata/flatcar
AFTERBURN_PROXMOXVE_INSTANCE_ID=ae1808cd94103dab0a5f9dcbc2b93e0b0f8d9b51

And what about flatcar-digitalocean-network.service? (the name is legacy, it has nothing to see with Digital Ocean).

core@flatcar ~ $ systemctl status flatcar-digitalocean-network.service
Unit flatcar-digitalocean-network.service could not be found.

There's no such service (or similarly named). 🤷

@mcbenjemaa
Copy link

This image is failed to boot for me

@fhemberger
Copy link

@mcbenjemaa Did you follow the procedure here?
flatcar/flatcar-website#337 (review)

Could you please provide a bit more context?

@mcbenjemaa
Copy link

mcbenjemaa commented Oct 25, 2024

flatcar/flatcar-website#337 (review)

@mcbenjemaa Did you follow the procedure here? flatcar/flatcar-website#337 (review)

Could you please provide a bit more context?

I just converted the VM into a template and then tried to add the config drive into a new clone, which didn't work.

Screenshot 2024-10-25 at 14 49 33

@fhemberger
Copy link

@mcbenjemaa You still need to provide your ignition.json. Proxmox' own cloud-init parameters are not enough (yet).

Creating the VM in the GUI doesn't work at the moment, but should work via CLI (steps linked in my comment above).

@mcbenjemaa
Copy link

@mcbenjemaa You still need to provide your ignition.json. Proxmox' own cloud-init parameters are not enough (yet).

Creating the VM in the GUI doesn't work at the moment, but should work via CLI (steps linked in my comment above).

Can you try to mount /dev/disk/by-label/cidata into some tmp folder, and let me know what is inside there?

@tormath1
Copy link
Contributor

@fhemberger @mcbenjemaa many thanks for trying again Flatcar on Proxmox! I started a discussion to avoid spamming and distracting the PR review. You can continue the discussion here if you need: flatcar/Flatcar#1573

@fhemberger
Copy link

@pothos Could you please add the following change to your PR:

https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/afterburn/files/coreos-metadata.service

--- coreos-metadata.service             2024-10-25 16:14:39.499357398 +0200
+++ coreos-metadata.service-proxmoxve   2024-10-25 16:14:55.180153512 +0200
@@ -25,6 +25,8 @@

 ConditionKernelCommandLine=|flatcar.oem.id=akamai

+ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve
+
 Description=Flatcar Metadata Agent

 [Service]

Co-authored-by: Kai Lüke <[email protected]>
Signed-off-by: Mathieu Tortuyaux <[email protected]>
@donch
Copy link

donch commented Oct 28, 2024

Hi @tormath1 , i've tested your build on Proxmox using the cloud-init configuration and it works well (at least for network configuration and SSH keys). By the way, the hostname configuration process strips any FQDN: if you set 'test.vm-1', only 'test' will be retained as the hostname.

Will continue testing the ignition part.

@tormath1
Copy link
Contributor

@pothos Could you please add the following change to your PR:

https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/afterburn/files/coreos-metadata.service

--- coreos-metadata.service             2024-10-25 16:14:39.499357398 +0200
+++ coreos-metadata.service-proxmoxve   2024-10-25 16:14:55.180153512 +0200
@@ -25,6 +25,8 @@

 ConditionKernelCommandLine=|flatcar.oem.id=akamai

+ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve
+
 Description=Flatcar Metadata Agent

 [Service]

Now available here: http://bincache.flatcar-linux.net/images/amd64/9999.9.101+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2

@fhemberger
Copy link

@tormath1 Works fine for me, /run/metadata/flatcar is now created successfully. 👍

@donch
Copy link

donch commented Oct 29, 2024

I was able to sucessfully configure a proxmox VM using ignition including user creation, file creation, custom sysext deployment, systemd unit 🎉
By the way, the service [email protected] is failing when no ssh pub key are supplied but may not be related to this PR

@tormath1
Copy link
Contributor

Thanks everyone involved in the testing of the image, that's truly appreciated ❤️ There is one last thing to sort out: flatcar/Flatcar#1573 (reply in thread) before going ahead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: ⚒️ In Progress
Development

Successfully merging this pull request may close these issues.

7 participants