FU has inconsistent NAL type

[darkSuperman]

FU has inconsistent NAL type: Header { unit_type: IdrWRadl, nuh_layer_id: 0, nuh_temporal_id_plus1: 1 } then Header { unit_type: FdNut, nuh_layer_id: 0, nuh_temporal_id_plus1: 1 }

conn: 10.10.10.111:39778(me)->10.10.10.240:554@2025-10-23T09:10:12.613399339Z
stream: TCP, interleaved channel ids 0-1
ssrc: 228ac871
seq: 28142
pkt: 2636@2025-10-23T09:10:12.641913596Z

Is the h265 stream incorrect? But I can stream it normally using VLC.

[scottlamb]

The truth is that almost all RTSP cameras I've tried violate the spec in some way or another. Retina is saying the H.265 is improperly framed into RTP packets in a way that (from recent investigation in scottlamb/moonfire-nvr#344) seems to be harmless. Retina used to error out in this case; now it just logs a warning once per stream and goes on with life. VLC may not even check for the inconsistency.

If you don't get correct video though, let me know, and we can investigate anew.

[darkSuperman]

Thank you for your explanation. I really can't get the .h265 mp4, but the .h264 is normal.

These are basically from your code，The run method keeps returning the error mentioned above

async fn write_mp4(
    opts: &Opts,
    session: retina::client::Session<retina::client::Described>,
    audio_params: Option<Box<AudioParameters>>,
    stop_signal: Pin<Box<dyn Future<Output = Result<(), std::io::Error>>>>,
) -> Result<(), Error> {
    let mut session = session
        .play(
            retina::client::PlayOptions::default()
                .initial_timestamp(opts.initial_timestamp)
                .enforce_timestamps_with_max_jump_secs(NonZeroU32::new(10).unwrap())
                .unknown_rtcp_ssrc(opts.unknown_rtcp_ssrc),
        )
        .await?
        .demuxed()?;

    // Append into a filename suffixed with ".partial", then try to either rename it into
    // place if it's complete or delete it otherwise.
    const PARTIAL_SUFFIX: &str = ".partial";
    let mut tmp_filename = opts.out.as_os_str().to_owned();
    tmp_filename.push(PARTIAL_SUFFIX); // OsString::push doesn't put in a '/', unlike PathBuf::.
    let tmp_filename: PathBuf = tmp_filename.into();
    let out = tokio::fs::File::create(&tmp_filename).await?;
    let mut mp4 = Mp4Writer::new(
        audio_params,
        opts.allow_loss,
        opts.frames_to_extract.clone(),
        out,
        opts.imgs_path.clone()
    )
    .await?;
    let result = copy(opts, &mut session, stop_signal, &mut mp4).await;
    if let Err(e) = result {
        // Log errors about finishing, returning the original error.
        if let Err(e) = mp4.finish().await {
            log::error!(".mp4 finish failed: {}", e);
            if let Err(e) = tokio::fs::remove_file(&tmp_filename).await {
                log::error!("and removing .mp4 failed too: {}", e);
            }
        } else if let Err(e) = tokio::fs::rename(&tmp_filename, &opts.out).await {
            log::error!("unable to move completed .mp4 into place: {}", e);
        }
        Err(e)
    } else {
        // Directly return errors about finishing.
        if let Err(e) = mp4.finish().await {
            log::error!(".mp4 finish failed: {}", e);
            if let Err(e) = tokio::fs::remove_file(&tmp_filename).await {
                log::error!("and removing .mp4 failed too: {}", e);
            }
            Err(e)
        } else {
            tokio::fs::rename(&tmp_filename, &opts.out).await?;
            Ok(())
        }
    }
}

pub async fn run(opts: Opts) -> Result<(), Error> {
    if matches!(opts.transport, Transport::Udp(_)) && !opts.allow_loss {
        warn!("Using --transport=udp without strongly recommended --allow-loss!");
    }

    let creds = super::creds(opts.src.username.clone(), opts.src.password.clone());
    let stop_signal = Box::pin(tokio::signal::ctrl_c());
    let session_group = Arc::new(retina::client::SessionGroup::default());
    let mut session = retina::client::Session::describe(
        opts.src.url.clone(),
        retina::client::SessionOptions::default()
            .creds(creds)
            .session_group(session_group.clone())
            .teardown(opts.teardown),
    )
    .await?;
    let video_stream_i = if !opts.no_video {
        let s = session.streams().iter().position(|s| {
            if s.media() == "video" {
                let encoding_name = s.encoding_name();
                if matches!(encoding_name, "h264" | "h265" | "jpeg") {
                    log::info!("Using {encoding_name} video stream");
                    return true;
                }
                log::info!(
                    "Ignoring {} video stream because it's unsupported",
                    s.encoding_name(),
                );
            }
            false
        });
        if s.is_none() {
            log::info!("No suitable video stream found");
        }
        s
    } else {
        log::info!("Ignoring video streams (if any) because of --no-video");
        None
    };
    if let Some(i) = video_stream_i {
        session
            .setup(i, SetupOptions::default().transport(opts.transport.clone()))
            .await?;
    }
    let audio_stream = if !opts.no_audio {
        let s = session
            .streams()
            .iter()
            .enumerate()
            .find_map(|(i, s)| match s.parameters() {
                // Only consider audio streams that can produce a .mp4 sample
                // entry.
                Some(retina::codec::ParametersRef::Audio(a)) if a.mp4_sample_entry().build().is_ok() => {
                    log::info!("Using {} audio stream (rfc 6381 codec {})", s.encoding_name(), a.rfc6381_codec().unwrap());
                    Some((i, Box::new(a.clone())))
                }
                _ if s.media() == "audio" => {
                    log::info!("Ignoring {} audio stream because it can't be placed into a .mp4 file without transcoding", s.encoding_name());
                    None
                }
                _ => None,
            });
        if s.is_none() {
            log::info!("No suitable audio stream found");
        }
        s
    } else {
        log::info!("Ignoring audio streams (if any) because of --no-audio");
        None
    };
    if let Some((i, _)) = audio_stream {
        session
            .setup(i, SetupOptions::default().transport(opts.transport.clone()))
            .await?;
    }
    if video_stream_i.is_none() && audio_stream.is_none() {
        bail!("Exiting because no video or audio stream was selected; see info log messages above");
    }
    let result = write_mp4(&opts, session, audio_stream.map(|(_i, p)| p), stop_signal).await;
    if result.is_err() {
        log::info!(
            "writing MP4 failed; \
                    details will be logged with `Fatal:` after RTSP session teardown"
        );
    }

    // Session has now been dropped, on success or failure. A TEARDOWN should
    // be pending if necessary. session_group.await_teardown() will wait for it.
    if let Err(e) = session_group.await_teardown().await {
        log::error!("TEARDOWN failed: {}", e);
    }
    result
}

2025-10-24 13:49:56 [INFO] Using h265 video stream
2025-10-24 13:49:57 [INFO] Ignoring audio streams (if any) because of --no-audio
2025-10-24 13:49:57 [INFO] writing MP4 failed; details will be logged with `Fatal:` after RTSP session teardown

[scottlamb]

Oh, that was a fatal error? What revision are you at? This changed quite recently.

[darkSuperman]

After I updated from 0.4.13 to 0.4.14, the error turned into an output warning, but I still can't extract the image like .h264

Here is the simplified code I'm using inside the video function to process the keyframe. It demonstrates that I'm directly taking the frame.data() provided by retina, converting it to Annex B format, and feeding it to the decoder, which then fails.

async fn video(&mut self, stream: &retina::client::Stream, frame: retina::codec::VideoFrame) -> Result<(), Error> {
        //--- extract keyframe images
        if frame.is_random_access_point() {
            self.frame_count += 1;
            if self.frames_to_extract.contains(&self.frame_count) {
                if let Some(decoder) = self.decoder.as_mut() {
                     error!("Original keyframe data (len={}): {:02X?}", frame.data().len(), frame.data());
                    let full_keyframe_unit_annexb = avcc_to_annexb(frame.data());
                    error!(
                        "Data fed into the decoder (len={}): {:02X?}",
                        full_keyframe_unit_annexb.len(),
                        full_keyframe_unit_annexb
                    );

                    match decoder.decode(&full_keyframe_unit_annexb) {
                        Ok(Some(yuv_picture)) => {
                            let (width, height) = yuv_picture.dimensions();
                            info!("Success！Size: {}x{}", width, height);

                            let mut rgb_buffer = vec![0; width * height * 3];
                            yuv_picture.write_rgb8(&mut rgb_buffer);

                            match image::ImageBuffer::<Rgb<u8>, _>::from_raw(width as u32, height as u32, rgb_buffer) {
                                Some(image) => {
                                    let filename = format!("{}/keyframe_{}.jpg", self.imgs_path, self.frame_count);
                                    if let Err(e) = image.save(&filename) {
                                        error!("save image error: {}", e);
                                    } else {
                                        info!("save image success: {}", filename);
                                    }
                                }
                                None => error!("unable to create ImageBuffer from raw data"),
                            }
                        }
                        Ok(None) => {
                            warn!("need more data to decode keyframe");
                        }
                        Err(e) => {
                            error!("error decoding keyframe: {:?}", e);
                        }
                    }
                }
            }
        }

pub fn avcc_to_annexb(avcc_data: &[u8]) -> Vec<u8> {
    let mut annexb_data = Vec::with_capacity(avcc_data.len() + 10);  
    let start_code: &[u8] = &[0x00, 0x00, 0x00, 0x01];
    let mut cursor = 0;

    while cursor + 4 <= avcc_data.len() {
        let nalu_len = u32::from_be_bytes(avcc_data[cursor..cursor + 4].try_into().unwrap()) as usize;
        cursor += 4;

        if cursor + nalu_len > avcc_data.len() {
            log::warn!("AVCC data seems corrupted. NALU length exceeds remaining buffer.");
            break;
        }

        annexb_data.extend_from_slice(start_code);
        
        annexb_data.extend_from_slice(&avcc_data[cursor..cursor + nalu_len]);
        
        cursor += nalu_len;
    }

    annexb_data
}

2025-10-25 18:42:12 [INFO] Using h265 video stream
2025-10-25 18:42:12 [INFO] Ignoring audio streams (if any) because of --no-audio
2025-10-25 18:42:12 [WARN] FU has inconsistent NAL header: Header { unit_type: IdrWRadl, nuh_layer_id: 0, nuh_temporal_id_plus1: 1 } then Header { unit_type: FdNut, nuh_layer_id: 0, nuh_temporal_id_plus1: 1 }; will not log about this again for this stream
2025-10-25 18:42:12 [ERROR] Data fed into the decoder (len=49101): [00, 00, 00, 01, 40, 01, 0C, 01, FF, FF, 01, 60, 00, 00, 03, 00, ...]
2025-10-25 18:42:12 [ERROR] error decoding keyframe: Error { native: 16, decoding_state: 0, misc: None, backtrace: Some(<disabled>) }
2025-10-25 18:42:12 [INFO] new video params: VideoParameters { rfc6381_codec: "hvc1.1.6.L120.00", pixel_dimensions: (1920, 1080), pixel_aspect_ratio: None, frame_rate: Some((3600, 90000)), extra_data: Length: 114 (0x72) bytes
0000:   01 01 60 00  00 00 00 00  00 00 00 00  78 f0 00 fc   ..`.........x...
0010:   fd f8 f8 00  00 0f 03 a0  00 01 00 18  40 01 0c 01   ............@...
0020:   ff ff 01 60  00 00 03 00  00 03 00 00  03 00 00 03   ...`............
0030:   00 78 ac 09  a1 00 01 00  2d 42 01 01  01 60 00 00   .x......-B...`..
0040:   03 00 00 03  00 00 03 00  00 03 00 78  a0 03 c0 80   ...........x....
0050:   11 07 cb 8d  ae e4 e5 33  70 10 10 10  40 00 03 84   .......3p...@...
0060:   00 00 57 e4  28 40 a2 00  01 00 07 44  01 c0 f2 b0   ..W.(@.....D....
0070:   22 40                                                "@ }

[scottlamb]

Nothing jumps out to me as wrong there.

Some video streams don't include the parameters in-band (so you'd have to extract them from the .parameters() as the ffmpeg decoder example does here)...but I think yours does: I see the same 40 01 0C ... bytes fed in your Data fed into decoder line as shown in the new video params line starting from byte position 0x12. So that's not it.

Speaking of: does the ffmpeg decoder example work any better on this stream?

Otherwise, if you're able to gather a packet capture (e.g. with Wireshark), I could take a look at that.