Building a Cross-Platform Player

A new series rewritten in construction order, seven chapters: the platform-independent player (framework and data sources, decode integration and decoder selection, rendering layers and Vulkan output, audio rendering and A/V sync); one chapter per platform (Windows, Android, Apple) with four articles each (hardware decode, the two destinations for decoded frames, capture devices, hardware encoding); WebRTC; player refinements (seek and speed, subtitles, track switching, HDR, embedding in a host); and recording (frame sources, software encoding, muxing and streaming) last.

Chapter 1: The platform-independent player

The player's thread skeleton and implementation registry1.1 The player's thread skeleton and implementation registry: which classes in the player are a thread, every replaceable implementation hangs off the registry, and logging has to be there on day one.about 1389 words
The player's command queue and playback state machine1.2 The player's command queue and playback state machine: the queue guarantees order but not completion, a successful enqueue does not mean the command runs, and the nine states are not a single chain.about 2195 words
The player's data sources and stream packets1.3 The player's data sources and stream packets: the source exposes only open, close, and a few queries; under one base class are two families of implementations with six legs; demuxing digests the container and the protocol and hands over single-frame stream packets; what a packet carries, and the two rules for config frames and the keyframe flag; the special case of live streaming is saved for the last section.about 2167 words
The player's packet-to-frame chain1.4 The player's packet-to-frame chain: what each of the two queue stages holds back and what its depth is measured against; then a walk along the chain in three segments (IO / decode / render), each covering what it does, which queue it moves, and what the two queues look like when something goes wrong; and finally, when the frame queue is empty, how to tell whether there truly is no data or decoding cannot keep up.about 2166 words
The player's decode integration and decoder selection1.5 The player's decode integration and decoder selection: there is only one decode interface, and frames land in one of two places, CPU or GPU; decoders are chosen and fallen back through a candidate chain — the preferred name, the alternate, the fallback name, and the first entry as the last resort.about 2503 words
The player's rendering layers and Vulkan output1.6 The player's rendering layers and Vulkan output: what the coordinator, pixel processing and presentation surface roles each handle, why the inheritance chain stacks bottom-up, and why a frame and the beat run in opposite directions; which chain each of the two output paths lands on, and on the Vulkan path its rhythm, the five steps to the screen and the swapchain values.about 2041 words
The player's audio rendering and device output1.7 The player's audio rendering and device output: the device sets the audio's rhythm and progress, two threads and two queue levels, four platform implementations at the device layer — three push from their own thread, one waits for the device to pull; format and resampling pass several gates before handover, and playback position is derived backwards from queue backlog.about 1703 words
The player's audio/video sync and clock1.8 The player's audio/video sync and clock: how the two chains' rhythms are aligned to system time (video by frame rate, audio by the device); what the clock stores, why the master clock defaults to audio, how the video side adjusts waits and frame drops while the audio side adjusts the amount of data sent out, and why sources that are misaligned from the start simply have sync turned off.about 1742 words

Chapter 2: Windows

The player's hardware decoding abstraction and the Windows D3D11 implementation2.1 The player's hardware decoding abstraction and the Windows D3D11 implementation: the two things hardware decoding leaves to the platform are the decode device and frame output — create the decode device, and at frame output wrap the native texture into a GPU frame; the four hardware decode lanes are not one approach, Android and Apple connect to the native interfaces themselves while Windows and Vulkan borrow FFmpeg's wrappers, and this Windows implementation builds the device through d3d11va and hands it to the render side to reuse the same one, copies that texture in the texture array into a self-built texture before dispatching, and has to block the false positive of first assignment on size changes.about 1812 words
Two destinations for hardware-decoded frames on Windows2.2 Two destinations for hardware-decoded frames on Windows: the platform pixel layer first converts NV12/P010 into a shared texture; the path imported into the Vulkan pipeline uses an NT handle and a shared fence, with the reader waiting on the CPU for the writer; the path that skips VK draws it as a fullscreen quad and presents through the window's own swapchain, with the rhythm staggered by vertical sync.about 1277 words
Windows capture devices: screen, camera and sound card2.3 Windows capture devices: screen, camera and sound card. Screen and window share one capture implementation, with WinRT Graphics Capture as the main path and GDI as the fallback; the camera goes through Media Foundation asynchronous reads and accepts only a few planar and packed YUV formats; audio has two sets (microphone event-driven, sound-card loopback poll-only). Frames produced: textures for the screen, CPU planes for the camera, PCM blocks for audio.about 1398 words
Windows hardware encoding: encoder selection and the input path2.4 Windows hardware encoding: encoder selection and the input path. The preferred names are the two in-house ones, but only shells exist behind them, so the candidate chain falls to the hardware encoders FFmpeg ships; at open it picks among QSV/AMF/NVENC by whether a hardware device can be created, and only when none can does it fall back to the Media Foundation wrapper. Frames must be read back into memory before they enter the encoder — the direct-texture path exists only as a design in the repo.about 1241 words

Chapter 3: Android

Android hardware decoding: MediaCodec and the Surface lifecycle3.1 Android hardware decoding: MediaCodec and the Surface lifecycle. The JNI boundary grabs all Java methods once and leaves one door for reverse callbacks; opening splits into create, configure and start, parameter sets must be converted to start-code form and only the local copy is changed; a Surface has two sources, invalidation goes through one reset flag, and on seek this implementation can flush in place while other hardware decoders must rebuild.about 1118 words
Two destinations for hardware-decoded frames on Android: GL and VK side by side3.2 Two destinations for hardware-decoded frames on Android: GL and VK side by side. The external texture is first converted into an RGBA texture by the platform renderer; one path builds its own EGLSurface and presents directly, the other hands this memory to Vulkan. What crosses over is not a texture object but an AHardwareBuffer, with one view in each API; the rebuild criterion has to include the EGL context.about 1122 words
Android capture devices: Camera2 and AudioRecord3.3 Android capture devices: Camera2 and AudioRecord. The camera's format whitelist has only one planar format, the output surface goes through an external texture (no frame-grab API); callbacks only set a flag, frames are taken on their own thread, and flag merging means implicit frame drops; audio lands on the Java side, read synchronously with no queue; neither path reconnects automatically.about 1161 words
Android hardware encoding: MediaCodec takes GPU textures3.4 Android hardware encoding: MediaCodec takes GPU textures. There are two input paths: planar frames copied into the queue, texture frames drawn into the encoder's input surface (which needs a newer system API and rules out queueing once used); the draw step must set a microsecond timestamp and picks the sampler by whether the frame carries shared graphics memory; the output carries a four-byte start-code prefix, and config frames come from the system callback.about 1219 words

Chapter 4: Apple

Apple hardware decoding: VideoToolbox and the ObjC boundary4.1 Apple hardware decoding: VideoToolbox and the ObjC boundary. The class stays C++ and only the implementation file is ObjC++; the one place an ObjC object literal appears is the property dictionary when creating the session; parameter sets are turned into a format description with a branch per codec; the output pixel format has two judgments that would rather give up the lane; and the root cause of that family of black-screen and corrupted-picture cases sits in the layer and the format.about 1165 words
Two destinations for hardware-decoded frames on Apple: Metal and MoltenVK4.2 Two destinations for hardware-decoded frames on Apple: Metal and MoltenVK. The texture cache takes two textures by plane, and both the borrowed textures and the retained buffer live until the command buffer completes; when handing to Vulkan what is shared is the surface rather than the texture object, with no explicit cross-API synchronization; on MoltenVK, empty shader modules, surface row alignment and the missing compatibility switch crash outright.about 1137 words
Apple capture devices: AVCapture audio and video4.3 Apple capture devices: AVCapture audio and video. Camera and microphone land on the same capture session, and attaching inputs takes a fixed six steps; the format whitelist has only one planar format, and the preset and the upper-layer resolution are two independent choices; callbacks sit on a serial queue and dropped frames are not reported; closing the stream does not stop the session (so camera switches are instant), and there is no interruption or device-removal listener.about 1135 words
Apple hardware encoding: VideoToolbox4.4 Apple hardware encoding: VideoToolbox. The compression session is configured with real-time mode, profile, frame reordering disallowed, average bitrate, expected frame rate and maximum keyframe interval; there are three input paths (shared surface, system pixel buffer, planar data in memory); the output does four things — take the bitstream, compute the timestamp, check for dropped frames, check for keyframes — and config frames have to be resent at the keyframe moment.about 1135 words

Chapter 5: WebRTC

Building WebRTC: environment, fetching the code and per-platform build scripts5.1 Building WebRTC: environment, fetching the code and per-platform build scripts: fetch with depot_tools, configure with GN, run with ninja; the STL namespace hurdle on Android; the few threads that wire it into the project (headers and libraries from an external directory, the MT runtime, openssl per platform).about 1371 words
Wiring WebRTC's 3A into the recording chain: format normalization and layering5.2 Wiring WebRTC's 3A into the recording chain: normalization rewrites rather than rejects (44100 to 48000, convert to S16, fold channels to two, accumulate into 10 ms segments); resampling and segmentation sink into the base class while the algorithm goes in the subclass; attached at the audio source and rendering.about 1223 words
Own decoders and rendering, wired into WebRTC5.3 Own decoders and rendering, wired into WebRTC: swap in your own decode factory, since it has neither H265 nor AAC; frames taken over come in CPU and native forms; both ends of the timestamp must be set or decoding is judged to be falling behind.about 1287 words
Cross-platform two-way video calls: five stages and unifying the bitstream format5.4 Cross-platform two-way video calls: the five stages of pull, decode, image processing, encode and push are each replaceable; annexb and avcc/hvcc are decided first and converted per downstream, with configuration frames ordered VPS→SPS→PPS.about 1218 words
Own capture and hardware encoding, wired onto WebRTC's tracks and encoders5.5 Own capture and hardware encoding, wired onto WebRTC's tracks and encoders: four adapters, one each; capture once, with preview and encoding each taking a copy; bitrate and keyframes are both fed back by WebRTC.about 1197 words

Chapter 6: Player refinements

In progress

Chapter 7: Recording: frame sources, software encoding, muxing and streaming

The player's raw frame sources and the external texture entry7.1 The player's raw frame sources and the external texture entry: raw frame sources and encoded-data sources sit on two separate branches, and a source hands out frames in only three forms — CPU frame, GPU texture and audio data block; captured frames land in memory or in a texture and downstream routes them by type; external textures come in through the render layer's entry, while the capture-card branch is built as a whole plugin that registers its own slot in the device table.about 1696 words
The player's recording chain and software-encoder integration7.2 The player's recording chain and software-encoder integration: the recording chain assembles three pieces — a decoding source, offscreen processing and an encoder; a frame arrives through a callback, goes through offscreen processing, is read back into a CPU frame and enqueued; when the frame queue is full it blocks decoding instead of dropping frames; the software encoder follows the registry and the candidate chain, switching lanes before the first frame only if it cannot open; the pixel format is decided on the output side, and timestamps are passed through unchanged.about 1508 words
The player's muxing layer: three levels, the write thread and timestamps7.3 The player's muxing layer: three levels, the write thread and timestamps — the interface, a packet-only level, and a level with encoding added; writing out has only one queue and one thread, and timestamps are first zeroed on the first packet and then converted to the stream's own time base; the two other uses are remuxing without decoding and single-track direct copy, plus frame capture and audio extraction when no file is written.about 1228 words
The player's streaming output: the same muxer, with the target changed to a line of address7.4 The player's streaming output: the same muxer, with the target changed to a line of address — recording and streaming share the same muxing implementation and the same flow, differing only in that output line of address; the muxing format is guessed from the suffix first and mapped from the protocol when that fails (the rtmp family maps to flv); failures at stream start and on the write are handled on the spot, and how fast streaming goes is decided by the source and by the write speed.about 1205 words