Livepeer binary releases do not work with Netint hardware

[Strykar]

Describe the bug
Livepeer binary releases do not work with Netint hardware

To Reproduce

1. Use latest Livepeer binary (0.5.35) with a Netint GPU
2. Use basic transcoder-only config
3. Livepeer appears to find the Netint device, but then complains Nvidia decoder not found, tries to use software and finally fails
4. Livepeer startup log:

Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: *-------------*--------------------------------------------*
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | CliAddr     | 127.0.0.1:7939                             |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | OrchAddr    | xx.xx.xx.xx:8935                         |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | Transcoder  | true                                       |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | OrchSecret  | /home/strykar/livepeer/orcp                |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | MaxSessions | 5                                          |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | Netint      | 0                                          |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | Monitor     | true                                       |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: | Datadir     | /home/strykar/.lpData/arbitrum-one-mainnet |
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: *-------------*--------------------------------------------*
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: I1024 14:40:05.707090 2976680 starter.go:337] ***Livepeer is running on the offchain network***
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: I1024 14:40:05.708011 2976680 db.go:363] Initialized DB node
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: I1024 14:40:05.708051 2976680 starter.go:403] Netint devices: [0]
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: I1024 14:40:05.708059 2976680 starter.go:407] Transcoding on these Netint devices: [0]
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: WARNING: decoder.c:290] Nvidia decoder not found; defaulting to software
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: ERROR: encoder.c:234] Unable to find encoder : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: ERROR: transcoder.c:238] Unable to open output : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: E1024 14:40:05.710395 2976680 ffmpeg.go:977] Transcoder Return : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: I1024 14:40:05.710436 2976680 transcoder.go:296] required capability "H.264" is not supported on device 0
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: WARNING: decoder.c:290] Nvidia decoder not found; defaulting to software
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: ERROR: encoder.c:234] Unable to find encoder : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: ERROR: transcoder.c:238] Unable to open output : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: E1024 14:40:05.711473 2976680 ffmpeg.go:977] Transcoder Return : Invalid argument
Oct 24 14:40:05 graf.sundara.tv livepeer[2976680]: F1024 14:40:05.711533 2976680 starter.go:412] required capability "H.264" is not supported on hardware
Oct 24 14:40:05 graf.sundara.tv systemd[1]: t408.service: Main process exited, code=exited, status=255/EXCEPTION

Expected behavior
Livepeer should start and transcode using the Netint hardware

Desktop

• OS: Debian 11.5 kernel 5.10.0-18-amd64 / Ubuntu 22.04 kernel 5.15.0-52-generic

Additional context
After speaking with the only other Orchestrator I know of who has Netint hardware, John | EliteEncoder / @eliteprox, it appears that ffmpeg is not compiled with Netint support.
Perhaps Livepeer could ship a binary release for Netint (3.1.0 firmware release) if that makes things easier, like livepeer-amd64-netint.tar.gz? AV1 support might be nice too, but not critical 😄
Also, if this repo - https://github.com/livepeer/FFmpeg could be updated with a Netint branch, it might make building Livepeer easier for mortals and be a huge step forward.

[stronk-dev]

AFAIK this is indeed caused since you need to compile ffmpeg with Netint support. Netint gives their fork out on request if you have a netint card, but even then it is a real pain to get their cards working (I gave up on it myself)

Ideally they can get their changes merged in the official FFMPEG release, otherwise +1 for releasing a go-livepeer binary which has Netint support built in

AV1 support might be nice too, but not critical 😄

(Sidenote: as a side project I am currently working on implementing Intel Arc and AV1 support, luckily Intel made sure that FFMPEG supports their hardware and AV1 encoding)

[Strykar]

AFAIK this is indeed caused since you need to compile ffmpeg with Netint support. Netint gives their fork out on request if you have a netint card, but even then it is a real pain to get their cards working (I gave up on it myself)

I do have a version from Netint, back when they were still working with Livepeer to get it going with ffmpeg.
It is for an ancient version of Livepeer, which works (for now) in a remote transcoder-only config with the old firmware it was shipped with.

Ideally they can get their changes merged in the official FFMPEG release, otherwise +1 for releasing a go-livepeer binary which has Netint support built in

Unsure how this will pan out, they straight up told me to contact Livepeer for support with the latest releases, since they had upstreamed the required changes from their side.
Apparently, upgrading the firmware will cause it to stop working with that Livepeer version so I have left it as is in spite of having the newer firmware on hand to flash.

(Sidenote: as a side project I am currently working on implementing Intel Arc and AV1 support, luckily Intel made sure that FFMPEG supports their hardware and AV1 encoding)

Are you developing on the A770's with 16 GB RAM?
This is exciting news indeed and I look forward to this!

[cyberj0g]

We just need to merge Netint Ffmpeg fork into Livepeer Ffmpeg, already got permission to make it open source from them AFAIK. The changes are in separate files, mostly, so it's not a hard task.

[Strykar]

Thanks @cyberj0g, that's music to my ears!
Any chance you are aware if updating card firmware will break ffmpeg and livepeer?

[cyberj0g]

This branch can be used to effortlessly build Livepeer with Netint support from sources (Netint driver sources should be obtained separately): #2659
The binary release is not planned at this point, as we need a permission to bundle Netint driver.

[eliteprox]

@cyberj0g What version of libxcoder is this version patched for? FW 3.0.1 uses libxcoder_logan name in the code and so libraries are not recognized.

[rickstaa]

Related to #2780.