When replacing timecode, is it supposed to only generate/add 1 sample?

[KweenLizzie]

Hi there, thanks so much for making this, it's helped me a lot to understand timecode creation.

I'm just a video editor trying to solve a problem, and so I don't know Swift super well, but I kept getting warnings about the .wait() in "AVAsset Timecode Write.swift" because it doesn't seem to work well with modern concurrency, or at least however I was trying to use it (I don't really know what I'm doing). Anyway it worked until I ran 1TB of files to replace timecode and then compress, and then eventually it would cause a crash, so I had to stop using TimecodeKit unfortunately.

However, I looked into what you were doing and what I could do and came up with a solution using your method of creating new timecode samples.

I was just wondering if your code is supposed to only add 1 sample? I noticed that my output files seem to be fine but the timecode end is always 4 frames after timecode start, whereas the original files have a timecode end matching the final frame of the video. If it's only supposed to add 1 sample it seems like you realised the timecode end doesn't actually matter?

I'm also not sure if your timecode is usually written in the timescale of the original timecode, so I've added the timescale to the writer just in case:

// called with: await writeTemporaryTimecodeTrack(from: sourceTimecodeTrack, toURL: tempfileURL) { }

func writeTemporaryTimecodeTrack(
    from sourceTimecodeTrack: AVAssetTrack,
    toURL tempfileURL: URL,
    completion: @escaping () -> Void
) async {
    // get settings from source asset
    let formatDescriptions = try? await sourceTimecodeTrack.load(.formatDescriptions)
    guard let formatDescription = formatDescriptions?.first else {
        print("Failed to load timecode track properties")
        return
    }
    let frameDurationDesc = CMTimeCodeFormatDescriptionGetFrameDuration(formatDescription)
    let frameDuration = frameDurationDesc.value
    let timeScale = frameDurationDesc.timescale
    let inputFrameDuration = CMTime(value: frameDuration, timescale: timeScale)
    
    // set up writer
    guard let writer = try? AVAssetWriter(url: tempfileURL, fileType: .mov) else {
        print("Writer initialisation failed")
        return }
    writer.movieTimeScale = timeScale
    
    let input = AVAssetWriterInput(mediaType: .timecode, outputSettings: nil)
    input.mediaTimeScale = timeScale
    
    guard writer.canAdd(input) else { print("Adding input to writer failed")
        return }
    writer.add(input)
    
    guard writer.startWriting() else {
        print("Failed to start writing")
        return }
    writer.startSession(atSourceTime: .zero)
    
    // fetch start timecode or set new one here
    guard let getStartTimecode = getInitialTimecode(for: sourceTimecodeTrack) else {
        print("Failed to get initial timecode frame")
        return
    }
    var startTimecode = getStartTimecode.bigEndian
    
    // write data
    guard let blockBuffer = try? CMBlockBuffer(length: MemoryLayout<UInt32>.size) else {
        print("Failed to initialise blockbuffer")
        return }
    do {
        try blockBuffer.fillDataBytes(with: 0x00)
        try withUnsafeBytes(of: &startTimecode) { startTimecodePointer in
            try blockBuffer.replaceDataBytes(with: startTimecodePointer)
        }
    } catch {
        print("Failed to write blockbuffer data")
        return
    }
    
    // append sample buffer
    guard let sampleBuffer = try? CMSampleBuffer(
        dataBuffer: blockBuffer,
        formatDescription: formatDescription,
        numSamples: 1,
        sampleTimings: [CMSampleTimingInfo(duration: inputFrameDuration, presentationTimeStamp: .zero, decodeTimeStamp: .invalid)],
        sampleSizes: [4]
    ) else {
        print("Failed to initialise samplebuffer")
        return }
    do {
        try sampleBuffer.makeDataReady()
    } catch {
        print("Failed to make samplebuffer ready")
        return
    }
    input.append(sampleBuffer)
    input.markAsFinished()
    
    // finish writing
    do {
        try await finishWritingAsset(writer: writer, frameDuration: inputFrameDuration)
        print("Asset writing completed successfully.")
    } catch {
        print("Error during asset writing: \(error)")
    }
}

func finishWritingAsset(writer: AVAssetWriter, frameDuration: CMTime) async throws {
    writer.endSession(atSourceTime: frameDuration)
    try await withCheckedThrowingContinuation { continuation in
        writer.finishWriting {
            if writer.status == .completed {
                continuation.resume(returning: ())
            } else if writer.status == .failed {
                let defaultError = NSError(domain: "com.yourdomain.error",
                                           code: -1,
                                           userInfo: [NSLocalizedDescriptionKey: "Unknown error during asset writing."])
                continuation.resume(throwing: writer.error ?? defaultError)
            } else {
                let defaultError = NSError(domain: "com.yourdomain.error",
                                           code: -2,
                                           userInfo: [NSLocalizedDescriptionKey: "Asset writing was cancelled or an unexpected error occurred."])
                continuation.resume(throwing: defaultError)
            }
        }
    }
}

// if successfully written then call: await replaceTimecodeInMovie(movie: movie, withNewTimecodeFrom: tempfileURL)

func replaceTimecodeInMovie(movie: AVMutableMovie, withNewTimecodeFrom tempfileURL: URL) async {
    // load new timecode asset
    let newTimecodeAsset = AVMutableMovie(url: tempfileURL, options: [AVURLAssetPreferPreciseDurationAndTimingKey: true])
    guard let newTimecodeTrack = newTimecodeAsset.tracks(withMediaType: .timecode).first else {
        print("Failed to load new timecode track.")
        return
    }
    let newTimecodeTrackDuration = newTimecodeTrack.timeRange.duration
    
    //remove existing timecode tracks from the mutable movie
    let existingTimecodeTracks = movie.tracks(withMediaType: .timecode)
    existingTimecodeTracks.forEach { track in
        movie.removeTrack(track)
    }
    
    // create new mutable timecode track
    guard let movieTimecodeTrack = movie.addMutableTrack(withMediaType: .timecode, copySettingsFrom: nil) else {
        print("Failed to add new mutable timecode track.")
        return
    }
    
    // add new timecode track
    do {
        let range = CMTimeRangeMake(start: .zero, duration: newTimecodeTrackDuration)
        try movieTimecodeTrack.insertTimeRange(range, of: newTimecodeTrack, at: .zero, copySampleData: true)
        let videoTracks = movie.tracks(withMediaType: .video)
        if videoTracks.isEmpty {
            print("No video tracks found.")
        } else {
            videoTracks.forEach { videoTrack in
                movieTimecodeTrack.addTrackAssociation(to: videoTrack, type: .timecode)
                print("Timecode associated with video track.")
            }
        }
    }  catch let error as NSError {
        print("Failed to replace timecode track: \(error)")
        print("Error localised description: \(error.localizedDescription)")
    }
}

[orchetect]

I kept getting warnings about the .wait() in "AVAsset Timecode Write.swift" because it doesn't seem to work well with modern concurrency, or at least however I was trying to use it

Do you remember what sort of errors? Do you mean when Thread Sanitizer is turned on?

it worked until I ran 1TB of files to replace timecode and then compress, and then eventually it would cause a crash

Sounds like a memory issue. Do you recall if that caused really large RAM usage? If so, then that's probably a small fix where a release pool might be needed.

I was just wondering if your code is supposed to only add 1 sample? I noticed that my output files seem to be fine but the timecode end is always 4 frames after timecode start, whereas the original files have a timecode end matching the final frame of the video. If it's only supposed to add 1 sample it seems like you realised the timecode end doesn't actually matter?

I'd have to look into it. I didn't run into any issues with test video files I was using so maybe it's an edge case, not sure. In theory you just need one timecode sample at the start of the track and have the track duration match the video track's (if I recall, been a while since I worked on this module).

[KweenLizzie]

Thanks, and sorry but I don't remember the exact error as it never printed to the console. There was just a purple warning in Xcode about waiting for a lower class task or lower priority task or something? Like I said I don't really know what I'm doing, but the gist was that the function that compresses the files could hang because of the .wait() when appending the timecode. I just ignored it but then it started crashing after running for a while and so I updated the code to use await instead and haven't had any crashes. I assume it's because everything else in the code is using the latest concurrency and doesn't play well with the .wait()

I've since done some tests with the samples and reconnecting Premiere projects with a mix of camera files and it does seem like generally there aren't any issues with just the one timecode sample, but I ended up adding an extra sample at the end for the final frame, and now the timecode in MediaInfo has the same start and end timecode as the original file and feels good regardless of if it's need or not. Hopefully it isn't a bad thing to have a random sample at the end! The whole reason I tried using TimecodeKit in the first place is because AssetWriter kept crashing when camera files had a timecode sample for every video frame, so replacing the timecode fixed that issue.

Anyway everything seems to be working for me now, thanks.

[orchetect]

Thanks for the details.

purple warning in Xcode about waiting for a lower class task or lower priority task or something

Yeah that might depend on how/where the methods are being called as I am not seeing these errors come when running the package's unit tests. But since DispatchGroup is being used that could definitely run interference in an async/await context.

The AVFoundation portion of TimecodeKit was written around the time when async/await was very new, so efforts were made to allow functions to be backwards compatible for apps with older OS requirements. At some point I'll refactor them to use the newer API since Swift Concurrency is fairly pervasive now. That should fix the issues you ran into.

Hopefully it isn't a bad thing to have a random sample at the end!

Technically it's harmless, but I'm not sure it's necessary, that's all. Start sample and duration should be all that's needed. But if it works then great!

everything seems to be working for me now

Great - glad you found some solutions for your use case.

I'm always interested if there's areas of the library that are causing friction or may be buggy, as I want to improve its usage where possible. Thanks for sharing the code snippet (I cleaned it up a little so it's formatted with syntax highlighting) - it may be useful for reference when I dive back into the codebase.

[orchetect]

Just FYI - I finally updated the library and migrated all of the AVFoundation methods to async/await. Released in 2.2.0.

If you end up doing refactors let me know how that goes. Otherwise you can stay on 2.1.0 if you're not ready yet.