Encoder hangs inside EbDeinitHandle (via FFmpeg libraries)

[oviano]

When attempting a real-time encode on hardware that can't handle it (or is a borderline case) I find that on closing the encoder it 99% of the time hangs with this callstack:

` [External Code]
avcodecd.dll!EbDestroyThread(void * threadHandle) Line 113 C

avcodecd.dll!EBEncHandleStopThreads(EbEncHandle_s * encHandlePtr) Line 438 C
avcodecd.dll!EbEncHandleDctor(void * p) Line 467 C
avcodecd.dll!EbH265EncComponentDeInit(EB_COMPONENTTYPE * h265EncComponent) Line 1595 C
avcodecd.dll!EbDeinitHandle(EB_COMPONENTTYPE * h265EncComponent) Line 1616 C
avcodecd.dll!eb_enc_close(AVCodecContext * avctx) Line 452 C
avcodecd.dll!avcodec_close(AVCodecContext * avctx) Line 1117 C
avcodecd.dll!avcodec_free_context(AVCodecContext * * pavctx) Line 180 C
emu-server.exe!ovav::VIDEO_SERVER_TRANSCODER::thread_clean_up() Line 446 C++
emu-server.exe!ovcore::THREAD_BASE::thread_func() Line 219 C++
[External Code]
�`

This happens with the latest release (1.4.3) and also the latest code. In the above example I have built SVT-HEVC and used the debug library with the FFmpeg libraries.

I'm aware of a few hanging issues reported, but none that I can see that corresponds to this particular callstack, so I've opened a new issue.

[tianjunwork]

Thanks reporting the issue. Based on our recent test of 500 times of running ffmpeg on a Xeon server, we couldn't reproduce the issue. There is possibility that hang may happen. Could you tell more details of your dev env? System, cpu, and what other programs are running during svt-hevc encoding, etc.

[oviano]

Right, so this is running on far more modest hardware, specifically an i5-9400F with 16GB RAM. The OS is Windows 10.

The encode threads is using the automatic setting. I'm using the encoding preset 7 and the source is 1080p50 RAW video coming from a Magewell capture card.

Nothing else important is running, but the application will have other threads operating for network I/O, and the video and audio capture although these use a small amount of CPU compared to the HEVC encode.

The issue seems related to when the CPU is under duress. If I increase the preset, or lower the resolution of the source, the issue becomes very rare. But it's easy to reproduce with lower presets and also simply using the debug build of SVT incurs enough of a speed penalty to make it happen more.

Also, do you run any tests with a more limited number of threads? Perhaps the threading architecture hasn't had much coverage with low number of threads. Also do you run your tests as fast as possible or do you simulate a real time encode? This is happening for real time encoding.

(for a brief overview of my project, see emustream.tv)

[tianjunwork]

The issue seems related to when the CPU is under duress. If I increase the preset, or lower the resolution of the source, the issue becomes very rare. But it's easy to reproduce with lower presets and also simply using the debug build of SVT incurs enough of a speed penalty to make it happen more.

Thanks for the hint here. Could you also paste your command line?
Usually we run our test with multi-core Xeon skylake server with Ubuntu/CentOS which is the setup for majority users. Yes, your setup is different.
Could you help me understand what is a real time encode?

[oviano]

Ok so as I mention I am running this by directly calling the FFmpeg API from inside my application, I am not using a standalone FFmpeg command.

However I am going to try and replicate the issue using standard a FFmpeg command and if I am successful I will paste the command here.

When I say real time encode I mean my application is capturing a live AV stream and passing the YUV frames into the FFmpeg libraries as frames are captured
Ie at the same frame rate.

Another theory is that maybe the SVT encoder doesn’t like being closed without being flushed first. I do not flush the encoder first as there should not be a need and it’s not needed for any other encoder, but I will try and see if that makes any difference.

[oviano]

Right, so I can workaround it by ensuring I flush the encoder before closing it. After flushing everything cleans up properly without hanging.

I could not reproduce it in the FFmpeg standalone tool perhaps because FFmpeg already does this flush when you interrupt it; I tried emulating a real-time input using the "-re" ffmpeg flag and then chose a preset of 3. The encode was of course too slow to keep up with the input, but when I halted the encode by pressing "q", FFmpeg paused for a while but did eventually exit cleanly. I then repeated with a preset of 0 and although shutdown was much slower it did eventually exit, which suggests to me that FFmpeg is indeed flushing the encoder and so won't show this issue.

So maybe this is not something important, but it is the only encoder I've come across so far that needs to be flushed before closing. I think it ought to be able to abandon whatever data it's in the middle of encoding if told to close the reason being if you'd chosen a really slow preset you need to be able to abort without having to wait for all the queued frames to be encoded, but that's for you guys to decide.

[tianjunwork]

Thank you @oviano for the explanation and doing the experiment.
What has done to flush the encoder?

[oviano]

So in FFmpeg you pass a NULL frame to avcodec_send_frame which I can see from your ffmpeg patch then causes EB_BUFFERFLAG_EOS to be sent in the header to EbH265EncSendPicture.

I’ve not come across needing to do this before in order to be able to exit cleanly from an encoder but it’s an edge case really for people linking directly to your libraries, so hopefully this thread can act as a little bit of info in case someone comes across it in future.

[tianjunwork]

SVT supports process-based parallelism, which involves the splitting of the encoding operation into a set of independent encoding processes. Each encoding process creates 1 to N threads. A thread may block on others waiting for some resource to proceed.
When encoding ends, they need to be notified with EB_BUFFERFLAG_EOS signal to break from the waiting status.
That's why you see the backtrace stops at: avcodecd.dll!EBEncHandleStopThreads(EbEncHandle_s * encHandlePtr) Line 438 C. To be more specific: WaitForSingleObject(threadHandle, INFINITE); when trying to destroy the thread.
With the extensive use of multi-threading and current design, EB_BUFFERFLAG_EOS is required when encoding ends.
Thank you for trying out different things and solving the issue by yourself. And appreciate that you always keep the mind of help others in the community👍

[oviano]

Thank you for the explanation.

My suggestion would be that in the ffmpeg plugin when it closes the encoder it could also issue EB_BUFFERFLAG_EOS if it has not already done so previously via an explicit flush.

I will test this out and try and produce a PR for this soon.

[tianjunwork]

Hi @oviano , does issue EB_BUFFERFLAG_EOS if it has not already done solve your issue? any update since we last discussed?