Take a Dolby Vision film. To play it correctly the player has to answer three questions: what is in the source (container and metadata), what screen is in front of you, and how the middle path runs (how a decoded frame becomes pixels). The cross product is a dozen-plus cases: HDR10, HDR10+, Dolby Vision P5/P8 and HLG, multiplied by an SDR screen, a true HDR screen, and screens that “numerically look like HDR”.
This article opens up how Panvox does it. It is driven by the avox engine; every conclusion below can be found in the code.
Screenshots of HDR passthrough came out with the colours completely distorted, so there is no passthrough screenshot; the tone-mapped one looks normal.
Ground rules: two channels, two kinds of screen
Start from the source side. HDR's luminance curve is PQ (SMPTE ST 2084): code values map directly to absolute luminance from 0 to 10000 nits, unlike SDR's relative values. HDR10's static metadata (ST.2086) is one set per film; HDR10+'s dynamic metadata rides in T.35 SEI. Of the two common Dolby Vision profiles, P5 is “bare” — the picture data carries no HDR10-compatible layer, and without parsing its RPU the colours will be wrong; P8 has the compatibility layer, so an older player can play it as HDR10 and get roughly the right picture. What dynamic metadata has in common is that it carries brightness targets per frame (or per scene), which means the processing granularity should also be per frame.
On the display side, not every screen that reports itself as HDR is actually bright. In Apple's EDR scheme, built-in non-XDR displays always report headroom 2.0 — HDR numerically, but not in luminance. I set the threshold for “true HDR screen” on macOS at headroom strictly greater than 2.0; below that it is treated as SDR. Otherwise HDR films come out grey and flat, and the user blames the player.
The channel side is my most important architectural decision: lanes are split by output method, not by decode method. On the HDR passthrough path, the hardware-decoded frame goes straight to the platform's native renderer and nothing in the chain touches the image; wherever processing is needed (for example tone-mapping an HDR film onto an ordinary screen), both hardware and software decoded frames enter the Vulkan pipeline, and enhancement, filters and super-resolution exist only inside Vulkan. The two channels are structurally identical on every platform.
Why passthrough bypasses Vulkan
The Vulkan pipeline handles everything that needs processing: tone mapping on SDR screens, filters, subtitles — hardware and software decoded frames go through the same chain. HDR passthrough bypasses it so that a high-bit-depth signal never crosses the processing chain. Three reasons.
First, interop cost. A 10-bit (P010) hardware-decoded frame entering Vulkan must first be converted by the platform renderer into an 8-bit interface surface — the high bit depth is lost at step one. An early pitfall sat exactly here: HDR processing inside Vulkan had no effect on hardware-decoded frames, because the input had already been converted to 8-bit by the platform renderer. So on the HDR presentation path, decoded frames must go native.
Second, the payload is undetectable. A tone-mapped SDR frame and a PQ stream are both numbers between 0 and 1 at the byte level; the data cannot tell you which chain produced it. So the render lane can only be declared explicitly by the host (called HdrMode in the engine), and the render layer is forbidden to guess — a wrong guess produces a washed-out image with nothing to trace.
Third, where per-frame processing lives. Enhancement, effects and super-resolution are optional and exist only in Vulkan; passthrough is required for presentation and should take the shortest path — pixels from decoder to screen, with no “while we're here, let's tweak it” in between.
HDR screens: passthrough output (Windows / iOS / macOS)
On Windows, D3D11VA decodes to P010, a compute shader processes it, and the result is written as R10G10B10A2 PQ codes; the swap chain declares the PQ colour space with SetColorSpace1 and hands the ST.2086 static metadata to the system with SetHDRMetaData. Display capabilities are probed through IDXGIOutput6. One structural choice: intent and actual flip are separated — the UI thread only sets an intent bit, and the render thread flips the swap chain's HDR state on the next frame, rolling back on failure. That structure later blocked several classes of race, as the display-switching section below shows.
On Apple, the CVPixelBuffer from VideoToolbox goes straight into a Metal shader without an 8-bit intermediate; the presentation layer is an EDR-enabled CAMetalLayer (extended linear ITU-R 2020), the shader outputs linear values where 1.0 is SDR white (100 nits), highlights can exceed 1.0, and the system compositor pushes real luminance according to the display's headroom. On iOS the capability is probed through UIScreen's potentialEDRHeadroom with a threshold above 1.
On Android, Panvox currently only tone-maps; there is no passthrough output. The system supports it — the HDR presentation path exists and the ecosystem uses SurfaceView for the hardware route; what is missing is my render bridge: EGL has no HDR presentation surface yet, to be added later.
One detail people ask about passthrough: what happens to subtitles? Subtitles follow the lane — in passthrough the subtitle canvas switches to the platform's native render leg (DX11 / Metal) and is composited as a transparent overlay, without joining or interrupting the video signal; in the processing chain subtitles composite together with the picture inside Vulkan. “Nothing in the chain touches the image” constrains the video signal; overlays ride each lane's own presentation leg.
SDR screens: per-frame tone mapping
Most people watch on SDR screens, so this chain decides how it looks. The core is to make the “how bright to compress” decision per frame: peak luminance prefers Dolby's per-frame L1 value, then falls back to HDR10's maxCLL, then maxLuminance, and finally 1000 nits. The algorithm takes PQ EOTF into linear, applies an anchor-based soft compression (approximately linear pass-through, Reinhard extension above the anchor against the content peak), converts BT.2020 down to BT.709, and applies OETF for the output. The peak is fed to the shader per frame through a UBO — in implementation, “per-frame consumption” means each frame's knee is decided by that frame's own metadata, so fast light-dark transitions do not lose out.
Dolby Vision: P5, P8 and the RPU
The RPU is a small per-frame “recolour instruction” (a reshaping curve plus matrix) carried by Dolby Vision, and both P5 and P8 rely on it. P5 must reshape back to standard colour by itself; the reshaping semantics follow libplacebo's processing chain: pivots, polynomials, MMR, plus ycc_to_rgb, done inside the shader before colour conversion.
One engineering hurdle: FFmpeg does not export Dolby RPU parsing symbols. I vendored the relevant source into the repo and built it myself; metadata extraction became a decoder-agnostic side path (pulling NAL 62 RPU and SEI 137/144 where packets are fed in), giving the hardware and software legs equal standing — and that design made every later dynamic-metadata source (HDR10+ included) take the same road.
A typical pitfall: P5's container carries no colour tag, and the first version trusted the container default, so the whole film was under-tone-mapped — dark and oversaturated. The fix is to always consume the Dolby chain's output as PQ regardless of container tags.
How do you verify that “per-frame consumption actually works”? Eyeballs are unreliable. I built discriminating test material with identical pixels and only L1 varying: scene A's L1 peak equals the built-in static metadata (a control that should produce no visible difference), scene B lowers L1 to 250 nits (which must be applied). The decision is a per-frame mean-luminance difference over a threshold, not a look at the picture. The two hardware legs (Windows DX11VA and macOS VideoToolbox) measured bit-identical end to end.
HDR10+, and what is not done yet
HDR10+ dynamic metadata rides in T.35 SEI (ST 2094-10); I parse the bit syntax and consume its L1 peak/mean, taking the same road into tone mapping as Dolby L1. Not done yet: ST 2094's per-window trimming and saturation semantics; Dolby P7 (FEL) and VVC are not supported.
Switching screens and lanes mid-playback, and the intent bit
HDR is runtime state, not an install-time setting. Moving the window to another monitor, unplugging the HDR screen, or turning HDR off in the system all have to be handled. The structure above already prepared for it: the host sets an intent bit, the render thread performs the flip after passing the capability gate; rejection, early exit, rebuild — every path must clear the intent bit; a self-healing resend runs every three seconds. Switching between the Vulkan and Metal render lanes uses “tear down and re-mount”: destroy the surface, change lanes, re-mount, about 120 ms in practice, playback uninterrupted.
Three real battle scars, all stranger than expected:
One: resizing the window during HDR passthrough turned the picture grey and silently dropped the colour bars from 255 to 191. The cause was not the display: when the swap chain is rebuilt (ResizeBuffers), the 10-bit format, PQ colour space and HDR metadata are silently reset by the driver. The fix is to re-apply the full HDR swap-chain state immediately after a resize.
Two: moving an HDR screen's window to an SDR screen flashed once, then stayed a constant “magenta wash”. The chain: rebuilding the device cleared the actual state but left the intent bit; the self-healing resend was rejected by the capability gate; the rejected path did not clear the intent bit; the next round mounted a PQ swap chain on an SDR screen — SetColorSpace1 is only a declaration, and an SDR screen does not stop it. The end state was a PQ surface with tone mapping applied: luminance compressed, gamut not, hence magenta. The lesson: a declaration succeeding does not mean the screen listened — the intent bit must clear together with the rejection.
Three: on iPhone, a cold start playing an HDR film showed a black screen and healed itself after rotating once. The cause: when MoltenVK tears down the Vulkan swap chain it writes the host CAMetalLayer's drawableSize to 1×1, and because the Metal leg never builds a chain on that layer, nobody writes it back — a 3840×2160 frame drawn into a 1×1 drawable. The fix is to make the host re-run layout immediately when a lane change finishes. Finding it relied on one log line: target:1x1.
SDR films on an HDR screen
The default is to present them as SDR; there is no SDR-to-HDR up-mapping (inverse tone mapping). That is a deliberate trade, not an omission: the engine has zero implementation today, and a questionable-quality gamble is not worth taking.
Afterword
HDR playback is not hard because of “support or not”; it is hard because of per-frame metadata consumption, the differences between platform presentation paths, and races in runtime state. Splitting passthrough and the processing chain into two structurally identical channels, managing intent and actual state separately, and turning “really per-frame” into a testable assertion with discriminating material — those few things are what four-platform HDR rests on. For the details, see the implementation document written with AI assistance: avox/doc/plan/gpu/多平台HDR统一重构方案.md.
The avox engine is open source on GitHub; the pipeline, shaders and swap-chain management described here are all in it. The rest of the series: the product introduction, From the open-source engine avox to a product: the cross-platform player Panvox.