The player's HDR: passthrough, reshaping and tone mapping

Chapter 6: Player refinements·about 1190 words

Sections and series

Chapter 1: The platform-independent player

Chapter 2: Windows

Chapter 3: Android

Chapter 4: Apple

Chapter 5: WebRTC

Chapter 6: Player refinements

Chapter 7: Recording: frame sources, software encoding, muxing and streaming

Two lanes, split by output

HDR passthrough output is a presentation requirement: the hardware-decoded frame goes straight to the platform's native rendering, the whole chain applies no image processing, and the high-bit-depth signal never passes through the processing chain. Every other case that needs processing — tone mapping an HDR title on a standard screen, adding filters, applying enhancement — sends both hardware-decoded and software-decoded frames into the Vulkan pipeline, and the image-enhancement set exists only inside that pipeline. The two lanes are isomorphic on every platform.

There is a hard reason passthrough must bypass Vulkan: a 10-bit hardware-decoded frame entering the pipeline first has to be converted by the platform renderer into an 8-bit interface surface, so the high bit depth of HDR is lost at step one. There is another layer: the payload is undetectable. A tone-mapped standard frame and a PQ stream are both just numbers between 0 and 1 at the byte level, and cannot be told apart. So the lane can only be declared explicitly by the host; in the engine it is called HdrMode, with three values: follow, force standard, force passthrough. The render layer never guesses from the data.

Two render lanes: passthrough goes native, everything else enters the pipeline

Before passthrough: the gates on each platform

Before passthrough is accepted, three gates must pass: the display has to be capable (on Windows I probe through the newer output interface; if the PQ tier is not supported there is no passthrough; on Apple I read the screen's headroom), the system's HDR switch has to be on, and the host has to have stated its intent. Only when all three pass does the flip happen; if it cannot, the frame stays on the pipeline lane and color correctness is covered by tone mapping.

On Windows, the hardware-decoded P010 is written by a compute shader into 10-bit RGB PQ code; the swap chain first declares the PQ color space, then hands the static metadata to the system. The crux here is keeping intent and the actual flip apart: the host only sets an intent bit, and the render thread performs the real flip of the swap chain's HDR state on the next frame, rolling back on failure. Rejected, early exit, rebuild — every one of those paths must clear the intent bit. One more lesson lives in the comments: declaring a color space does not mean the screen actually obeyed. Display capability has to be determined first, otherwise the declaration succeeds and the picture is wrong.

On the Apple side, the hardware-decoded pixel buffer goes straight into the Metal shader without landing in an 8-bit intermediate format; the presentation layer is a Metal layer with EDR enabled, the shader outputs linear values directly, 1.0 is defined as standard white (about 100 nits), highlights may exceed 1, and the system compositor pushes the real brightness up to the headroom of the screen in front of it. Accepting passthrough has a threshold: the screen headroom must be greater than 1, the same threshold on both Apple platforms, and iOS additionally requires an OS version that supports it; anything below goes the standard-screen route.

On Android there is only tone mapping today, no passthrough. The system itself supports HDR presentation; what is missing is the render-bridge layer — EGL has no HDR presentation surface implementation yet.

Per-frame tone mapping: where the peak comes from and how the chain runs

Standard screens: per-frame tone mapping

Most people watch on standard screens, and this chain decides how it looks. The core is turning "how bright to compress" into a per-frame decision: the peak brightness prefers the per-frame Dolby L1 value, falls back to HDR10's maxCLL (maximum content light level), then to the maximum luminance, and finally bottoms out at one thousand nits. The processing chain takes PQ into linear space, applies a soft shoulder around the anchor (near-linear passthrough below it, soft compression past it by content peak), then compresses from the wide gamut to the standard gamut, reapplies the output curve and produces the picture. The peak is fed to the shader per frame through a uniform buffer — scenes that cut fast between bright and dark take no hit.

Dolby Vision and HDR10+

Dolby's dynamic metadata is a small per-frame color-reshaping instruction; FFmpeg does not export its parsing symbols, so the relevant source comes into the repo and is built there. Metadata extraction is a decoder-agnostic bypass: at the packet feed I recognize the relevant NAL and SEI types myself — this step needs no decoding, so the hardware-decode leg and the software-decode leg receive the same metadata, neither favored. The reshaping semantics of Dolby P5 follow the processing chain of an open-source implementation; reshaping completes in the shader and then enters color conversion. There is a snag: the P5 container carries no color tags, and handling it by the container default under-tone-maps the whole title, so the Dolby chain's output is always consumed as PQ, regardless of container tags. HDR10+'s dynamic metadata rides T.35 SEI; parsed bit by bit, the L1 peak is taken and enters tone mapping on the same path as Dolby L1.

Whether per-frame consumption really is per-frame cannot be judged by eye; it takes discriminative material: the same picture with only the L1 peak changed, one set equal to the built-in static metadata (should show no difference), one set with the peak pushed very low (must be compressible); the verdict is a per-frame difference in mean luminance crossing a threshold, not a look at the picture. Both hardware-decode legs measured end-to-end identical, and only then does "per-frame" stand confirmed.

Screen changes mid-play: intent clears on failure

HDR is a runtime state, not an install-time configuration: moving the window to another display mid-playback, unplugging the HDR screen, turning HDR off in the system — all must be handled. Switching between the two render lanes does not rebuild the whole pipeline: the presentation surface is detached from one leg and re-attached to the other, playback uninterrupted. The intent-versus-actual-state separation exists exactly for this, together with a self-healing resend every few seconds. The failures all lived in races. Rebuilding the swap chain silently reset bit depth, color space and metadata in the driver and the picture turned gray; the fix re-mounts the full state set immediately after a rebuild. After a cross-screen migration, device rebuild cleared the actual state but not the intent bit, the resend was refused by the capability gate, and the passthrough surface was propped onto a standard screen — brightness compressed but gamut not, the whole frame magenta. As long as the rejected path did not clear the intent bit, that loop could not close. One more incident turned the whole screen black: when a lane switch wrapped up, the host presentation layer's size was written as one pixel; the fix landed on the host side — nudge the host to run layout once more when the switch finishes.

Standard titles on an HDR screen present as standard by default, with no up-conversion. That is a deliberate trade-off: the engine has zero implementation on this path, and a picture-quality gamble with questionable payoff is better left undone.

The engine avox is open source on GitHub: github.com/xxxzhou/avox, where both lanes and all three processing paths live.