[Tracking] TPM 2.0/Secure Boot workstream #630

jepio · 2022-02-16T15:46:42Z

jepio · 2022-05-13T08:38:43Z

Mantle now publishes GCP images with UEFI features including vTPM enabled. With Azure CommunityGalleries we can also enable vTPM support on our images. AWS has GA'd their vTPM, but requires switching the AMI to UEFI boot mode (dropping compat with Xen HVM instances). We will need to consider publishing separate ami's for UEFI and BIOS boot.

Secure boot has not been tackled yet, but started thinking about signing procedures and key handling.

tormath1 · 2022-08-31T07:19:42Z

As a side note, Clevis (and friends) are now available on ::guru: https://github.com/gentoo/guru/tree/master/app-crypt/clevis. Including Clevis should then follow the same process as including a package from ::gentoo overlay (https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#add-or-update-a-package).

Butane clevis restriction should also be dropped once done: https://github.com/coreos/butane/blob/87a7a686aff511dd13a6b873baf9a7d005170737/config/flatcar/v1_0/translate.go#L37-L41

krishjainx · 2023-06-06T01:37:24Z

@tormath1 Trying to add Clevis takes me into dependency hell. I'm essentially doing https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#add-or-update-a-package but with gentoo/guru instead of gentoo/gentoo

tormath1 · 2023-06-06T07:35:36Z

@krishjainx yes, there are some dependencies like jose and luksmeta that needs to be pulled too. Back in the days @jepio started a PoC for Clevis: https://github.com/flatcar/coreos-overlay/commits/jepio/wip-clevis you can base your work on this (or rebase his branch)

krishjainx · 2023-06-14T10:13:46Z

The preliminary support for clevis can be found at flatcar/scripts#909. I will return to working on this shortly. I encountered some issues after working on it with an alpha tag and then transitioning it to another branch based on master. So, I will need to rebase and test it with the new SDK container locally. If someone is interested in taking over my work, please feel free to have a look. @tormath1

jepio · 2024-04-10T12:20:46Z

On EFI we are now missing tpm eventlog access from linux because that depends on the linuxefi boot protocol which is a downstream RH grub patch. I think we'll need to switch to using https://github.com/rhboot/grub2

jepio · 2024-07-29T12:01:35Z

Gael from matrix shared this link to grub patches for booting an UKI on BIOS systems, which could be interesting: https://github.com/osteffenrh/grub2-blscfg/commits/blscfg-unified-kernel-f38

chewi · 2024-08-28T15:24:26Z

I've looked into the nature of Red Hat's fork. It's based around Fedora releases with 40 still using 2.06 and 41 moving to 2.12. The patch set is very heavy with 368 commits currently applied on top of 2.12. For some reason, this doesn't quite align with the order or number of patches in their RPM package. There's a lot of noise here, as it includes commits that do things like rename grub to grub2. However, if we need the Linux EFI support, trying to cherry-pick individual commits doesn't seem like a good idea as it's a big change, and it's hard to tell exactly which commits are applicable.

chewi · 2024-09-04T16:03:15Z

I now realise why I confused about Fedora 41 during our recent meeting. The RPM package has patches against 2.12, but the branch in the rhboot/grub2 repo is still based on 2.06. 😵

We did talk about taking a tarball from GitHub, but I'd also like to use Gentoo's ebuilds if we can because I think little to no changes are needed. That would mean applying Red Hat's changes using a patch, but even as a single file, that patch would be nearly 2MB. 😅 That's not necessarily an issue, but…

jepio · 2024-09-04T16:51:31Z

Wouldn't a single patch be a nightmare to maintain and rebase? The other thing that is important: we need to stay in control of the sbat.csv as we are responsible for the security fixes to grub.

chewi · 2024-09-05T17:16:22Z

I did ask about the weird repo situation in Fedora's Matrix, but they say the repos are for different purposes. Things are still in motion though, so maybe this will change soon. Until then, a single patch is the best option. It's actually not that hard, I'll write up the steps.

I've got sbat.csv in hand.

I have now successfully built Red Hat's fork using Gentoo's ebuild. The Gentoo patches conflict, but none of them are applicable to Flatcar, so I've simply unset PATCHES. I've also rebased our own two patches, which took a fair amount of conflict wrangling. The GPT changes are currently failing to build. More soon.

chewi · 2024-09-09T14:24:54Z

qemu_update and arm64's qemu_uefi passed, but the rest didn't. I tried the image locally, and it does seem like the verity hash isn't getting passed through, at least on amd64. No idea why yet, that patch is quite straightforward.

chewi · 2024-09-09T17:01:18Z

Still having trouble with this, despite trying a few things. I can rebuild GRUB and write the new grubx64.efi to an existing image, so the turnaround time isn't too bad, but it's still tricky. What's strange is that BIOS is also failing in the same way even though both amd64 and arm64 EFI now use grub-core/loader/efi/linux.c instead. Seemingly it worked on arm64? The result is that systemd waits forever for the usr device, but I can see from using rd.break=pre-mount that the verity argument is missing.

jepio · 2024-09-10T08:21:43Z

Still having trouble with this, despite trying a few things. I can rebuild GRUB and write the new grubx64.efi to an existing image, so the turnaround time isn't too bad, but it's still tricky. What's strange is that BIOS is also failing in the same way even though both amd64 and arm64 EFI now use grub-core/loader/efi/linux.c instead. Seemingly it worked on arm64? The result is that systemd waits forever for the usr device, but I can see from using rd.break=pre-mount that the verity argument is missing

Can you share your latest code/branch? amd64 and arm64 use different offsets where the verity hash is stored in the vmlinuz file (64 vs 512).
How about we get it working with 2.06 + patches first (since that is closer to what we currently have), and switch to 2.12 in a second step.

jepio · 2024-09-10T08:23:21Z

Updated the tracker with currently open PRs that are ready to review & merge:

Ready for a early review:

Add the infrastructure for shim signing & aarch64 support scripts#2292

chewi · 2024-09-10T10:07:01Z

See flatcar/scripts#2301. The offsets are still respected. I might try 2.06, but I'll try adding some debugging to 2.12 first.

chewi · 2024-09-10T16:16:18Z

I learnt how to debug GRUB with gdb today. I've fixed this for BIOS, the code needed some adjustment and movement from grub-core/loader/i386/linux.c to grub-core/loader/i386/pc/linux.c. The EFI issue seems to be different.

jepio · 2024-09-10T17:04:42Z

Now that you mention it: the original verity hash commit did touch grub-core/loader/i386/efi/linux.c but we dropped this when we dropped the patch that introduced linuxefi. This is the first commit (without subsequent fixups): flatcar-hub/grub@03b547c. So this will need to be brought back.

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

chewi · 2024-09-10T17:19:28Z

Yeah, I was about to say I think this code now needs to go into grub-core/loader/i386/efi/linux.c, which I only noticed a short time ago. My first attempt at this failed, will have another go using your link tomorrow.

chewi · 2024-09-11T15:29:20Z

I think I've got it now. Time for another Jenkins run.

chewi · 2024-09-11T15:32:00Z

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

I think linux effectively calls linux16 when the EFI module cannot be loaded.

  cmd_linux =
    grub_register_command ("linux", grub_cmd_linux,
                           0, N_("Load Linux."));
  cmd_linux16 =
    grub_register_command ("linux16", grub_cmd_linux,
                           0, N_("Load Linux."));

chewi · 2024-09-11T16:01:29Z

I can't see any mention of the TPM event log in Red Hat's patches, only in the vanilla 2.12 sources. Are you sure the patches were needed? 😅

jepio · 2024-09-12T07:11:08Z

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

I think linux effectively calls linux16 when the EFI module cannot be loaded.

  cmd_linux =
    grub_register_command ("linux", grub_cmd_linux,
                           0, N_("Load Linux."));
  cmd_linux16 =
    grub_register_command ("linux16", grub_cmd_linux,
                           0, N_("Load Linux."));

I think I see what's going on. Mainline grub 2.12 has this in grub-core/Makefile.core.def:

module = {
  name = linux;
  x86 = loader/i386/linux.c;
  (...)
  x86_64_efi = loader/efi/linux.c;

and the fedora-41 branch of rhboot/grub has:

module = {
  name = linux;
  i386_pc = loader/i386/pc/linux.c;
  x86_64_efi = loader/i386/efi/linux.c;
  i386_xen_pvh = loader/i386/linux.c;

That would explain why the patch needs to be adapted for rhboot/grub: vanilla grub compiles in loader/i386/linux.c on both x86 BIOS and EFI, while rhboot grub completely separates them. It would also seem that on rhboot/grub the commands linuxefi and linux are the exact same, so we don't need to change grub.cfg to use a different command.

I can't see any mention of the TPM event log in Red Hat's patches, only in the vanilla 2.12 sources. Are you sure the patches were needed? 😅

It's not so much the patches, as it is the way grub enters the kernel (boot protocol), and this differs between the two branches. I haven't investigated this in detail, but there may be a way to get vanilla grub to enter the x86 kernel on EFI such that the eventlog is exposed. Arm64 efi is simpler and this works in vanilla grub. Compare the lines from arm64 1 and amd64 2:

arm64:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1]
[    0.000000] Linux version 6.6.50-flatcar ([email protected]) (aarch64-cros-linux-gnu-gcc (Gentoo Hardened 13.3.1_p20240614 p17) 13.3.1 20240614, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP PREEMPT Wed Sep 11 12:18:14 -00 2024
[    0.000000] KASLR enabled
[    0.000000] efi: EFI v2.7 by EDK II
[    0.000000] efi: SMBIOS 3.0=0xdced0000 TPMFinalLog=0xd9760000 MEMATTR=0xdba62198 ACPI 2.0=0xd9700018 TPMEventLog=0xd9673018 RNG=0xd970e698 MEMRESERVE=0xd9b43e18

amd64:

[    0.000000] Linux version 6.6.50-flatcar ([email protected]) (x86_64-cros-linux-gnu-gcc (Gentoo Hardened 13.3.1_p20240614 p17) 13.3.1 20240614, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed Sep 11 12:15:49 -00 2024
[    0.000000] Command line: BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200 flatcar.first_boot=detected verity.usrhash=45a627c8a86cad518157501cec1c7096df60ba0f4835732718f05affdeecf110
[    0.000000] BIOS-provided physical RAM map:
(...)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] efi: EFI v2.7 by EDK II
[    0.000000] efi: TPMFinalLog=0x9cbf7000 SMBIOS=0x9c9ab000 ACPI=0x9cb7e000 ACPI 2.0=0x9cb7e014 MEMATTR=0x9b678298

jepio · 2024-09-12T08:05:36Z

I think I see what's going on. Mainline grub 2.12 has this in grub-core/Makefile.core.def:

Ha, fun stuff: grub-2.06 doesn't have grub-core/loader/efi/linux.c, while vanilla 2.12 does. So it may be that the redhat patch is unnecessary (for what we want to achieve) with vanilla grub-2.12. Would you try it? If you add set debug="linux" to your OEM grub.cfg file then grub prints debug messages for the loader code paths.

chewi · 2024-09-12T08:56:58Z

Can I tell more directly by looking at the kernel output? Does efi: TPMEventLog= mean that it worked? If so, it's working on my laptop with no effort at all. I guess systemd-boot already supports this!

chewi · 2024-09-12T09:59:18Z

I've compared my own local amd64 QEMU kernel log between our last release and my recent changes. TPMEventLog appears in the latter but not the former. I'll try without the patches.

jepio · 2024-09-12T10:13:07Z

Can I tell more directly by looking at the kernel output? Does efi: TPMEventLog= mean that it worked? If so, it's working on my laptop with no effort at all. I guess systemd-boot already supports this!

Yes that's the line we're after. The full test case we need to add to kola is:
tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements successfully reads and parses the event log.
In phase 2 we'll need to make sure that the PCRs match what we expect and can precompute the ones that the distro is responsible for.

chewi · 2024-09-12T10:47:36Z

I can confirm that efi: TPMEventLog= appears with vanilla(ish) 2.12 and also that sudo tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements works! I guess we'll proceed on that track if you're sure we don't need the Red Hat patches for anything else.

jepio · 2024-09-12T12:10:31Z

I can't think of anything else we need right now. There may be differences in how grub measures things into PCRs, but we can address that later.

chewi · 2024-09-13T14:43:19Z

New Kola test is at flatcar/mantle#558.

jepio added kind/feature A feature request area/tpm Issues depending on TPM support labels Feb 16, 2022

jepio self-assigned this Feb 16, 2022

t-lo added the kind/roadmap A Roadmap item. label Feb 18, 2022

krishjainx mentioned this issue Jun 6, 2023

[RFE] Harden Flatcar Linux Defaults Further #1060

Open

9 tasks

t-lo assigned sayanchowdhury Sep 26, 2023

bignay2000 mentioned this issue Nov 28, 2023

[RFE] Azure Flatcar Gen2 support TrustedLaunch #1249

Open

sayanchowdhury mentioned this issue Jan 30, 2024

Initial implementation for Secure boot support flatcar/scripts#1589

Merged

2 tasks

chewi mentioned this issue Sep 12, 2024

sys-boot/grub: Move to portage-stable, bump to 2.12, add Red Hat patches flatcar/scripts#2301

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tracking] TPM 2.0/Secure Boot workstream #630

[Tracking] TPM 2.0/Secure Boot workstream #630

jepio commented Feb 16, 2022 •

edited

Loading

jepio commented May 13, 2022

tormath1 commented Aug 31, 2022

krishjainx commented Jun 6, 2023

tormath1 commented Jun 6, 2023

krishjainx commented Jun 14, 2023 •

edited

Loading

jepio commented Apr 10, 2024 •

edited

Loading

jepio commented Jul 29, 2024 •

edited

Loading

chewi commented Aug 28, 2024

chewi commented Sep 4, 2024

jepio commented Sep 4, 2024

chewi commented Sep 5, 2024 •

edited

Loading

chewi commented Sep 9, 2024

chewi commented Sep 9, 2024

jepio commented Sep 10, 2024

jepio commented Sep 10, 2024

chewi commented Sep 10, 2024

chewi commented Sep 10, 2024

jepio commented Sep 10, 2024

chewi commented Sep 10, 2024

chewi commented Sep 11, 2024

chewi commented Sep 11, 2024

chewi commented Sep 11, 2024

jepio commented Sep 12, 2024

jepio commented Sep 12, 2024

chewi commented Sep 12, 2024 •

edited

Loading

chewi commented Sep 12, 2024

jepio commented Sep 12, 2024 •

edited

Loading

chewi commented Sep 12, 2024

jepio commented Sep 12, 2024

chewi commented Sep 13, 2024

[Tracking] TPM 2.0/Secure Boot workstream #630

[Tracking] TPM 2.0/Secure Boot workstream #630

Comments

jepio commented Feb 16, 2022 • edited Loading

jepio commented May 13, 2022

tormath1 commented Aug 31, 2022

krishjainx commented Jun 6, 2023

tormath1 commented Jun 6, 2023

krishjainx commented Jun 14, 2023 • edited Loading

jepio commented Apr 10, 2024 • edited Loading

jepio commented Jul 29, 2024 • edited Loading

chewi commented Aug 28, 2024

chewi commented Sep 4, 2024

jepio commented Sep 4, 2024

chewi commented Sep 5, 2024 • edited Loading

chewi commented Sep 9, 2024

chewi commented Sep 9, 2024

jepio commented Sep 10, 2024

jepio commented Sep 10, 2024

chewi commented Sep 10, 2024

chewi commented Sep 10, 2024

jepio commented Sep 10, 2024

chewi commented Sep 10, 2024

chewi commented Sep 11, 2024

chewi commented Sep 11, 2024

chewi commented Sep 11, 2024

jepio commented Sep 12, 2024

jepio commented Sep 12, 2024

chewi commented Sep 12, 2024 • edited Loading

chewi commented Sep 12, 2024

jepio commented Sep 12, 2024 • edited Loading

chewi commented Sep 12, 2024

jepio commented Sep 12, 2024

chewi commented Sep 13, 2024

jepio commented Feb 16, 2022 •

edited

Loading

krishjainx commented Jun 14, 2023 •

edited

Loading

jepio commented Apr 10, 2024 •

edited

Loading

jepio commented Jul 29, 2024 •

edited

Loading

chewi commented Sep 5, 2024 •

edited

Loading

chewi commented Sep 12, 2024 •

edited

Loading

jepio commented Sep 12, 2024 •

edited

Loading