Issue with switching among tracks of same kind, codec & clock rate with ReplaceTrack of RTPSender

[selvan]

Your environment.

• Version: github.com/pion/webrtc/v4 v4.1.1

What did you do?

Creating a local track from a TrackRemote (remoteX),

localX, _ := webrtc.NewTrackLocalStaticRTP(remoteX.Codec().RTPCodecCapability, remoteX.ID(), remoteX.StreamID(), webrtc.WithRTPStreamID(remoteX.RID()))

Creating a another local track from a another TrackRemote (remoteY),

localY, _ := webrtc.NewTrackLocalStaticRTP(remoteY.Codec().RTPCodecCapability, remoteY.ID(), remoteY.StreamID(), webrtc.WithRTPStreamID(remoteY.RID()))

remoteX and remoteY are of same kind (Audio), codec type (VP8) and clock rate (90000)

Trying to replace localX with localY on the RTPSender with ReplaceTrack method.

What did you expect?

The remote peer attached to the RTPSender should receive switched video tracks when localX is replaced with localY & vice versa.

What happened?

The remote peer's video just freeze, when trying to replace localX with localY on the RTPSender.

Sample code to reproduce the issue

Running the same code:
https://github.com/selvan/webrtc/blob/issue-3149/examples/swap-tracks-via-replace

[arjunshajitech]

The remote peer attached to the RTPSender should receive switched video tracks when localX is replaced with localY & vice versa.

May be you need to request a key frame via PLI.

_ = pc.WriteRTCP([]rtcp.Packet{
	&rtcp.PictureLossIndication{
		MediaSSRC: uint32(receiver.Track().SSRC()),
	},
})

[selvan]

@arjunshajitech Have created sample code to demonstracte the issue,

Running the same code:
https://github.com/selvan/webrtc/blob/issue-3149/examples/swap-tracks-via-replace/README.md

PLI before ReplaceTrack: https://github.com/selvan/webrtc/blob/d9734013bb06fa7476c8e1ad5b87858a42ce9b76/examples/swap-tracks-via-replace/main.go#L180

[arjunshajitech]

@selvan

1. The code doesn't read RTP packets from the remote track and write them to the local track created in the ontrack event. This needs to be updated first.

Update the code in ontrack after the trackCount++

go func() {
			rtpBuf := make([]byte, 1400)
			for {
				n, _, err := track.Read(rtpBuf)
				if err != nil {
					if err == io.EOF {
						fmt.Printf("Track %s closed\n", track.ID())
						return
					}
					fmt.Printf("Error reading from track: %v\n", err)
					return
				}

				if _, err := lt.Write(rtpBuf[:n]); err != nil {
					fmt.Printf("Error writing to local track: %v\n", err)
					return
				}
			}
		}()

2. The video stuttering/stuck issue occurs because the SSRC changes every 5 seconds. When replacing the track, Pion doesn't automatically update the SSRC, sequence number, and timestamp. These need to be manually adjusted for smooth video playback and more code is needed to fully implement this flow. Thank you.

[selvan]

@arjunshajitech

1. Thanks for pointing out. Have fixed it.
2. Looking at the test for RTPSender,

   webrtc/rtpsender_test.go

   Line 23 in 86e4719

   func Test_RTPSender_ReplaceTrack(t *testing.T) { //nolint:cyclop

   
   ReplaceTrack looks to be good when different codecs (of same kind) is involved, without explicit SSRC/sequence number/timestamp adjustment.
3. Browser's replaceTrack, works without explicit need to adjust SSRC/sequence number/timestamp when two tracks of same kind, having same codec.

Any thoughts ?.

[arjunshajitech]

3. Browser's replaceTrack, works without explicit need to adjust SSRC/sequence number/timestamp when two tracks of same kind, having same codec.

This feature works in browsers as it's implemented according to the specification https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack, but Pion has not yet implemented it: #1020

[selvan]

ReplaceTrack test code

webrtc/rtpsender_test.go

Line 23 in 86e4719

func Test_RTPSender_ReplaceTrack(t *testing.T) { //nolint:cyclop

tests whether RTP packet is being received after ReplaceTrack. But the received packet may not be good enough to render the video due to mismatch in sequence number/timestamp.

As noted by @cretz in #1020, RTPSender.ReplaceTrack doesn't offer much benefit at the moment as RTP packet manipulation is required to render swaped tracks.

[arjunshajitech]

Check this example https://github.com/pion/webrtc/tree/master/examples/swap-tracks

Thank you.