Getting Video Tracks setup with non libdatachannel Peer

[stephen-derosa]

Hi all! I am having some trouble getting video tracks setup when connecting to a non libdatachannels peer. When connecting to a libdatachannel peer, things seem very straight forward -- all i have to do is addTrack for both peers with the correct rtc::Description::Direction, make sure encoding is correct and that ssrc matches, and boom, video tracks work. When using a non libdatachannel peer things seem a bit more complex.

Does anyone have examples of doing this with a non libdatachannel peer where the libdatachannel is the one sending the video?

Thanks!

[paullouisageneau]

While you can call addTrack() on both peers, it's not the right way and it indeed prevents you from properly supporting other implementations. You should call addTrack() only on offering side, and receive offered tracks with the onTrack callback on the answering side. You can get, update, and replace the description of a track in the callback, for instance to list payload types or add a sent SSRC.

[stephen-derosa]

would the answerer have to do anything special in their answer to the offer? Or is this just business as usual?

And does anything special need to be done in the onTrack callback? In the current architecture design, the answerer (libdatachannel peer) is the one sending video

[paullouisageneau]

If you want the answerer to send, you have to process the offer. You must at least retrieve the offered payload type for the format you want to send and add the sent SSRC. If the format is not present, you have to fail the negotiation. Optionally, you may remove other formats. Something like that:

std::shared_ptr<rtc::Track> track;
uint32_t ssrc = MYSSRC;
pc->onTrack([&track, ssrc](std::shared_ptr<Track> offeredTrack) {
    auto desc = offeredTrack->description();
    if (desc.direction() == rtc::Description::Direction::RecvOnly)
        return;

    // Find a format we can send
    for(int pt : desc.payloadTypes()) {
        auto rtpMap = desc.rtpMap(pt);
        if (rtpMap.format == "H264") {
            // If you want to send this format, you must use payload type pt
            desc.addSSRC(ssrc, "mycname"); // Add sent SSRC
            offeredTrack->setDescription(std::move(desc));
            offeredTrack->setMediaHandler([...]); // For instance you can set a packetizer here
            std::atomic_store(&track, offeredTrack);
            break;
        }
   }
});

You really should offer on the peer sending video, especially if you know in advance what video format you want to send. It makes more sense and it results in simpler logic and shorter descriptions.

[stephen-derosa]

thanks for the insights! How does the packetizer play into getting, in my case, encoded frames from a port and sending them over the track? Previously what i was doing for libdatachannel peers was running essentially the following in its own thread, where we simply bound to the socket that gstreamer was outputting h264 frames on:

void receiveLoop()
{
  // Receive from UDP
  char buffer[MEDIA_SEND_BUFFER_SIZE];
  int len;
  while ((len = recv(socket_, buffer, MEDIA_SEND_BUFFER_SIZE, 0)) >= 0)
  {
    if (len < sizeof(rtc::RtpHeader))
    {
      std::cout << "Invalid RTP packet of size of " << len << std::endl;
      continue;
    }
    else if (!track_)
    {
      std::cout << "Track is null." << std::endl;
      continue;
    }
    else if (!track_->isOpen())
    {
      std::cout << "Track is not open." << std::endl;
      continue;
    }

    auto rtp = reinterpret_cast<rtc::RtpHeader *>(buffer);
    rtp->setSsrc(ssrc_);

    track_->send(reinterpret_cast<const std::byte *>(buffer), len);
    std::cout << "Sent RTP packet of size " << len << std::endl;
  }
}

Im sure im missing something, but i dont see any references to ports/sockets in the RtpPacketization/RtpPacketizationConfig.

Thank you in advance.

[paullouisageneau]

If you forward an existing RTP stream, indeed you don't need a packetizer. You must only ensure that your RTP stream matches the payload type for the correct format, for instance you can override it like the SSRC:

std::shared_ptr<rtc::Track> track;
atomic<int> payloadType = 0;
uint32_t ssrc = MYSSRC;
pc->onTrack([&track, &payloadType, ssrc](std::shared_ptr<Track> offeredTrack) {
    auto desc = offeredTrack->description();
    if (desc.direction() == rtc::Description::Direction::RecvOnly)
        return;

    // Find a format we can send
    for(int pt : desc.payloadTypes()) {
        auto rtpMap = desc.rtpMap(pt);
        if (rtpMap.format == "H264") {
            // If you want to send this format, you must use payloadType pt
            payloadType = pt;
            desc.addSSRC(ssrc, "mycname"); // Add sent SSRC
            offeredTrack->setDescription(std::move(desc));
            std::atomic_store(&track, offeredTrack);
            break;
        }
   }
});

[...]
// Use payload type and SSRC when sending
auto rtp = reinterpret_cast<rtc::RtpHeader *>(buffer);
rtp->setPayloadType(payloadType);
rtp->setSsrc(ssrc);
track->send(reinterpret_cast<const std::byte *>(buffer), len);