The opening is on the factory; its code stays untouched
Every WebRTC release means merging code again, and once changes go into its body, merging gets harder and harder; the project is large and one build takes a long time too. The opening it leaves is the factory: when building a PeerConnectionFactory, both the decoder and encoder factories can be specified as your own implementations.
My approach is to hang my own video decoder factory on when the factory is built; the decoders inside take names from the player's own registry by codec type: platform hardware decoding first, FFmpeg software decoding as fallback — the selection mechanism is the one described in the decoding article. On Windows it resolves to the D3D11VA implementation, on Android to MediaCodec, on Apple to VideoToolbox. That gives H265 a usable decode path — its body only goes as far as the protocol layer for H265, with no decoder, while this path uses the player's own decode chain from start to finish; the WebRTC side only does RTP unpacking and bitstream parsing. The matching rebuild arguments were covered in the compile article: turn off its built-in H264 so both sides do not link separate copies of ffmpeg.
Video decoder: two conversions on the way in and out
The adapter implements WebRTC's decoder interface and at the same time the player side's decode observer; it is the bridge between the two ends. Initialization is hidden inside: at Configure it builds a bitstream parser according to H264 or H265 (picking out the QP along the way — the quantization parameter that takes part in judging picture quality), takes a name from the registry by codec type and creates the decoder; when the input thread sees the configuration frames are complete it completes initialization by itself, so no extra initialization interface needs to be attached.
The encoded frames WebRTC hands over look a lot like ffmpeg packets: always annexb, four-byte start codes. One place needs to be taken apart: non-data units such as AUD and SEI may be followed by configuration frames, that is, parameter sets, so the packet has to be split and inspected — configuration frames are taken out and fed to the decoder separately, and the rest of the packet is handed to the decoder as usual, which handles merging and splitting inside the packet itself — configuration frames sharing a packet with other frames is common, and without splitting, implementations like MediaCodec that want parameter sets one at a time get stuck.
The decode result has to be translated back to its return codes: success, data not ready, error, timeout, unsupported or not opened, each has its own mapping; a packet that arrives before the configuration frames are complete returns "success but a keyframe is needed", and WebRTC, on receiving that, goes and asks the peer for an I frame.
Both ends of the timestamp have to line up
When a decoded frame is converted back to a WebRTC VideoFrame, both the pts and the RTP timestamp have to be set. I measured this: leave them unset or set them wrong and the log either reports too much data in the decode queue and drops, or reports render delay too high — it uses timestamps to judge whether decoding is keeping up.
The method: when an encoded frame comes in, record its presentation time together with the RTP timestamp in a queue (with the QP, length-limited so it does not pile up); when decoding produces a pts, look up the matching RTP timestamp in the queue and fill it back in. When the lookup occasionally misses, fall back to pts times 90 — the RTP clock is 90 kHz, and that conversion is fixed.
Frames taken over come in two forms
On the signaling side: the local SDP is sent to the server and exchanged for the peer's answer; after SetRemoteDescription both sides know what each supports; in the callback for a track being created, the remote video track is attached to a sink, and frames come in through the sink callback.
Frames that come in are of two kinds. Software-decoded ones have a CPU buffer and are assembled into the player's frame as yuv420p; hardware-decoded ones are marked as native, and the buffer wraps a platform object — on Apple an image buffer of the CVPixelBuffer family, on Android the external texture from SurfaceTexture. Native frames do not land on the CPU; they go straight into the render layer, which is the same understanding as the two destinations of hardware-decoded frames in ordinary playback: what is shared is that piece of video memory, and who presents it is another matter.
The interface mapping is: when a frame has to be converted to WebRTC's I420, CPU frames really give data — yuv420p is copied directly, Windows' NV12 is converted with libyuv; GPU frames return empty: rendering all happens in the player's own pipeline and does not go through this step. The first frame also builds the video track description and triggers source readiness, and downstream configures its render objects from the description.
AAC: plug into its existing factory instead of replacing the whole thing
The audio side cannot copy the video approach. Its built-in opus and G711 have an audio-data detection layer behind them; replacing the whole thing with your own loses that. The approach is to write an AAC decoder modeled on its existing decoders and hang it in the factory's type list — besides it, the list still holds the original Opus, G722 and G711.
This adapter does two things: declare which SDP formats it supports to the outside, and create decoders from the negotiation result. The declared combinations list six groups by sample rate and channels: 8000 mono, 16000 mono and stereo, 32000 mono and stereo, 48000 stereo, all named mpeg4-generic; 44100 is not put in the declarations, because it has a check on sample rate. Decoding forwards internally to the project's own AAC implementation, going through fdk-aac with FFmpeg as fallback, output fixed at S16.
Two pitfalls: first, the header. AAC packaging comes in two kinds, adts and asc; the one from the live-stream path carries adts, while WebRTC's RTP packets do not, so the two-byte asc has to be generated and fed in as configuration; the two ends must match or it errors forever. Second, packetization: one AAC frame may be split across several RTP packets, the packet header says how long the frame is in total, and a received packet is only one segment of it, so parsing is accumulation-based — keep receiving while it is not enough, and only when it is enough assemble a complete frame and send it to the decoder, with the buffer cleared and started over. On a jittery network a frame split across four or five packets is common, and without gathering them all you decode half a frame of noise. One more small thing: the audio length it returns is counted in 16-bit samples, so converting to bytes requires dividing by 2.
The top layer does not join the player's tracks
Ordinary playback is MediaPlayer leading a chain of source, tracks, decoding and rendering; the WebRTC path does not enter that set — the top layer is a standalone player object that holds its own renderers: the remote picture renders directly on the native window, and sound goes through the audio rendering layer (the 3A switch is the one described in the audio pre-processing article). The local half — preview of your own picture — reuses the same rendering; the push-side wiring is covered in the next article.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, with the decode factory, frame adaptation and AAC factory inside.