mmap bar1: Operation not permitted

[tejasgadhia]

I have all the necessary files:

root@test:/home/test/f80# ls -alh
total 24M
drwxrwxr-x 2 test test  4.0K Dec  2 18:50 .
drwxr-xr-x 4 test test  4.0K Nov 27 15:23 ..
-rw-rw-r-- 1 test test  1.8M Nov 27 15:06 ELP-4x200-4d-n_13.05.11.01.bin
-rw-rw-r-- 1 test test 1004K Nov 27 15:06 ELP.split.bin
-rw-rw-r-- 1 test test  2.2M Nov 27 15:06 NWD-BLP4-800_13.00.08.00.bin
-rw-rw-r-- 1 test test 1004K Nov 27 15:06 NWD.split.bin
-rwxrwxr-x 1 test test  880K Nov 27 15:06 ddcli
-rw-r--r-- 1 root root   16M Dec  2 18:49 f80_backup_new
-rwxrwxr-x 1 test test   22K Nov 27 15:06 lsirec
-rwxrwxr-x 1 test test  490K Nov 27 15:06 lsiutil
-rw-rw-r-- 1 test test  2.9K Nov 27 15:06 sbrtool.py

However when attempting to execute lsirec to make a backup, I'm getting the following error:

root@test:/home/test/f80# ./lsirec 0000:01:00.0 readsbr sbr_backup.bin
mmap bar1: Operation not permitted

For reference I've included an output of lspci -Dvvnn below for information about my device:

0000:01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SSS6200 PCI-Express Flash SSD [1000:007e] (rev 03)
        Subsystem: Oracle/SUN Nytro ELP4x200_4d_n [108e:050a]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: I/O ports at 4000 [disabled] [size=256]
        Region 1: Memory at a6cc0000 (64-bit, non-prefetchable) [size=64K]
        Region 3: Memory at a6880000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at a6800000 [disabled] [size=512K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x8 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=0000e000
                PBA: BAR=1 offset=0000f800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 04000001 00000003 01010000 2113a8b7
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 007e
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000a6cd0000 (64-bit, non-prefetchable)
                Region 2: Memory at 00000000a68c0000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

My machine is running Ubuntu 20.04.1 LTS and Kernel is Linux 5.4.0-56-generic if that helps.

[motechsolutions]

I have the same problem with Ubuntu 18.04 desktop x64

[baconwaifu]

This is an issue with the kernel's CONFIG_STRICT_DEVMEM setting. "Recently" (since 4.16) it started defaulting to "on" which enables a whole host of restrictions on /dev/mem: a program can only mmap() MMIO regions, and those MMIO regions must not be mapped by a driver. Kernel driver in use: mpt3sas indicates that your card is "in use" and thus those regions are considered "mapped" by the kernel.

If you can reboot, setting iomem=relaxed on the kernel cmdline will allow it to map "used" MMIO regions (mpt3sas doesn't touch a lot of what this uses, so it's not really used) or if you can't reboot, using /sys to unbind the driver from the card will work.

I'm considering a feature request/pull request to the upstream kernel for some sort of driver-level interface to the debug registers, which would allow operation with strict_devmem, as well as precluding the need to disable the IOMMU, since the driver would be the thing doing all the DMA transfers (the current implementation "cheats" by mapping a giant buffer and getting the physical address for the card to DMA to; an IOMMU blocks that because userland RAM is not intended for DMA)