Bug report: Some SATA hard drives are not visible in AlmaLinux but others are visible

[jamesdev01]

AlmaLinux install disk and booted system cannot see 2 drives, but CentOS install disk and booted system can.

I installed the latest AlmaLinux 9 onto an SAS SSD and everything is working fine. However, it doesn't see 2 of the SATA drives, but it does see another SATA drive. All 3 SATA drives are connected to the same backplane, so why it can see one but not the other two is the question.

I also tried the same with AlmaLinux 8, just to check, but it has the same issue.

I tried all these and the two 12TB drives do not show up in the AlmaLinux system:

• df -h
• lsblk
• blkid
• fdisk -l
• parted -l
• lsscsi
• lshw -class disk

What does show up are:

• The SAS SSD (AlmaLinux 9)
• The previous system's SATA SSD (CentOS 7)
• 18TB SATA HDD

What does not show in the AlmaLinux system:

• 12TB SATA HDD
• 12TB SATA HDD

Now, in the same system, I rebooted back into the original SATA SSD system (which is running CentOS 7) just to make sure the drives are there. Yes, they show just fine there.

The 3 SATA drives (18TB, 12TB, 12TB) are all connected to the backplane (which is plugged into the SATA ports on the motherboard), although the SAS drive is connected directly to the motherboard via an SAS connector. AlmaLinux can see one of these three drives on this same backplane.

It should also be noted that these 2 drives also did not show up during installation as an install location. I noticed this at that time but didn't want to install a system onto the 12TB drives, so I ignored it.

When I booted into a CentOS 7 install disk on the same SAS controller, all 4 drives shows up including the two 12TB drives. But when I boot into AlmaLinux (also happens with AlmaLinux 8, I tried that too), AlmaLinux doesn't see the disks.

So the problem is not the drives, or the controller, or the backplane, or the SAS, because I narrowed it down since all drives show up in the CentOS install disk booted from SAS drive and the CentOS booted system, but not from the AlmaLinux install disk or the AlmaLinux booted system.

So, summary:

• AlmaLinux 8 and 9 install disks can't see the two drives
• The booted AlmaLinux system can't see the two drives
• CentOS 7 install disk can see all the drives fine
• CentOS 7 booted system can see all the drives fine

Almalinux works fine, but since it cannot see two of my disks, my server is missing 24TB of data...

Debugging: This is on a a Supermicro motherboard with a hardware RAID controller which the SAS SSD connected to it is set up in JBOD mode.

Motherboard: Supermicro
Raid controller: MegaRAID SAS 2208 [Thunderbolt]
Raid mode: JBOD (passthrough)
Boot disk: Hitachi Ultrastar Enterprise SAS SSD (connected directly to motherboard)
Drive that shows: WD Ultrastar 18TB HDD (connected to SATA backplane)
Drives that do not: WD Ultrastar 12TB HDD (connected to same SATA backplane)
Install disk: USB thumb drive
Method of creating bootable flash drive: dd
Secondary method of creating bootable flash drive: etcher for Linux in GUI (same result)

[jaboutboul]

Hi. Thanks for your report. RHEL (and therefore us) removed support for certain SAS controllers. You can install the drivers for these controllers from ELRepo (https://elrepo.org). They also provide driver update disks (DUD) to enable these controllers at install time. You can find more info on their site, and also here: https://elrepo.org/bugs/view.php?id=927

[jamesdev01]

There is no problem with the SAS drive.
It can see one of the SATA WD drives, and the SAS drive just fine.

The problem is that it cannot see two of the SATA drives.

All 3 of the SATA drives are connected to the same SATA backplane and connected to SATA on the motherboard, but Almalinux can only see one of the three.

[jamesdev01]

Additionally, there is another weird bug which I was sure was a problem during install.

When booting using CentOS 7, it shows a certain amount of storage.
But when bootin the AlmaLinux 8 or 9 install disc, it shows 30 GB less, which just so happens to be the size of the flash drive, and it shows the flash drive in the partition as if it was another partition.

Same problem was replicated on another system NOT using a RAID controller.

[jamesdev01]

On mine:

Left side is CentOS 7 install disc, right side is AlmaLinux 9 install disc (same hardware).
Below is different hardware.

On another system (no RAID controller)

This might be a different bug, or possibly the same bug.

The iso9660 is the USB drive. It does not show on the CentOS 7 install, but it does show on the AlmaLinux install and also reduces the available space by the amount of the USB drive (30GB). This is not normal behavior. There is a problem.

The first time I installed it asked me if I wanted to destroy partitions, I clicked yes, but it destroyed the partitions on the flash drive corrupting both the flash drive and the install! It wouldn't boot. I had to recreate the flash drive and reinstall, and this time not check to destroy existing partitions.

The install disc should not show up as a partition! It was NOT selected during the disk selection screen. I double and triple checked. Definitely did not select it, so it should not be showing.

My datacenter admin tried himself on a different system (the second picture), and same result. That USB drive should not be showing up as a partition.

[jamesdev01]

I actually have a cloudlinux subscription (Almalinux is made by the Cloudlinux team) so maybe it will solve the issue to simply upgrade the server to Cloudlinux?

Do you have a CloudLinux ISO I can burn to the flash drive and test the boot to see if the drive shows during the CloudLinux install? Or does CloudLinux only convert the live system? Edit: I got the iso.

But at least you can solve this bug for the future. Still would like to know the answer, will report back whenever I have tested the cloudlinux iso to see if the issue is the same.

[jamesdev01]

Hi. Thanks for your report. RHEL (and therefore us) removed support for certain SAS controllers. You can install the drivers for these controllers from ELRepo (https://elrepo.org). They also provide driver update disks (DUD) to enable these controllers at install time. You can find more info on their site, and also here: https://elrepo.org/bugs/view.php?id=927

If this does actually happen to be causing the issue, would it not make sense to update the install with this driver?

And how do I install that driver? The link you gave me goes to CentOS 8, not Almalinux 9.

The link you gave me is also for a different controller. It still might not even be relevant and might not have anything to do with this issue.

Especially since the same issue occurred on different hardware without any raid controller.

Would appreciate if someone can tell me if this page https://access.redhat.com/solutions/3751841 shows my controller or not.

Keep in mind I installed Almalinux just fine on my controller. It is 2 of the SATA drives that are not showing in Almalinux.

[jamesdev01]

CloudLinux 9.1 also cannot see the same two disks. I just burned the latest CloudLinux ISO and showed the same bugs:

Also the same weird bug where the USB drive which is not selected as shown in the above screen, shows as a partition and reduces the available space for the install by that much (30GB in this case for this 30GB drive):

Remember I am booting from a USB stick. And CentOS 7 is able to see all the drives. So I don't think this is a controller issue.

Also CentOS doesn't have this weird partition issue where the flash drive I'm booting on shows as a partition on the partitioning screen. Both Almalinux and CloudLinux both have both these same bugs.

This seems to be a bug.

[andrewlukoshko]

Hello @jamesdev01
Most likely it's hardware support regression in RHEL8 and RHEL9 kernels so comparison with CentOS 7 is not enough to diagnose the problem.
Could you please check the same with RHEL 8/9 and we'll have clear answer if it's AlmaLinux or upstream issue?
After that you could also check latest CentOS Stream 8/9 ISO which is basically pre-8.8/9.2 version of RHEL and if problem is still here we can report this upstream.
Thanks.

[jamesdev01]

I downloaded and burned the RHEL9 iso and I'm booting from it now.

[jamesdev01]

Could you please check the same with RHEL 8/9 and we'll have clear answer if it's AlmaLinux or upstream issue?

Ok, RHEL 9.1 doesn't see the two drives either.

And it has the same bug also about the USB drive that it is booting from is added as another partition and reducing the available space by the size of the USB flash drive.

So it does seem to be an upstream bug. Maybe you can send this up the chain to RHEL. I don't have a subscription with RedHat.

After that you could also check latest CentOS Stream 8/9 ISO which is basically pre-8.8/9.2 version of RHEL and if problem is still here we can report this upstream.

Ok, I'll also check CentOS Stream next.

[jamesdev01]

CentOS 9 Stream with same bugs. Cannot see other 2 drives and showing USB as partition and reducing available space by the size of the flash drive.

[jamesdev01]

CentOS 8 Stream same bugs, cannot see the two disks, USB partition

[jamesdev01]

RHEL 8.0 (8.0.0, not latest) still cannot see drives - but it does not have the USB bug like later versions

[jamesdev01]

As expected, the last working OS is 7.9. Tested RHEL 7.9 and it shows all drives and also shows correct space, unlike RHEL 8+.

[jaboutboul]

@andrewlukoshko lets go ahead and file this with Stream. It does indeed seem like a hardware support regression

[andrewlukoshko]

@jamesdev01 is this still happens with 8.9?

[AhmedShajee1]

Hello @jamesdev01, Since yesterday I have been trying to install Almalinux 8.6 and am facing the same issue. Have you found any solution for this by chance? Please do let me know!

[AhmedShajee1]

@andrewlukoshko @jaboutboul could you please guide me on this issue?