The player's subtitle canvas: format, scaling and multiple backends

Chapter 6: Player refinements·about 1157 words

Sections and series

Chapter 1: The platform-independent player

Chapter 2: Windows

Chapter 3: Android

Chapter 4: Apple

Chapter 5: WebRTC

Chapter 6: Player refinements

Chapter 7: Recording: frame sources, software encoding, muxing and streaming

The canvas moves from single-channel to RGBA

The earliest text subtitles went like this: glyphs were rasterized into grayscale and blitted into a single-channel canvas, uploaded only when the content changed, and blended on the GPU by one fixed pass with a single global color and opacity. What the single channel holds is the coverage of the glyph edges — the anti-aliasing ring of values; color is not in the canvas, it is one global monochrome. For plain text with one size and one color on screen, that is enough.

Where it falls short is just as clear. Styled subtitles draw the outline in a color different from the fill, and fade in and out per event; bitmap subtitles go further, where every pixel can be a different color. A single channel cannot express either one. The canvas moved to RGBA across the board: every producer emits a premultiplied canvas directly, cropped to the content bounding box — one line of dialogue usually occupies a single band at the bottom, not the full frame.

A single channel holds only coverage; per-pixel color requires RGBA

Premultiplication once hid an inconspicuous pitfall: changing opacity touched only alpha and left the colors unmultiplied. Under premultiplied semantics the color components have already been multiplied by alpha, so touching alpha alone makes anything semi-transparent glow white; changing opacity means multiplying the color components along with it. Upload volume rose with it: partial recognition results arrive about once every 100 ms, and one RGBA bottom band is several times the single-channel era. The saving grace is that text only re-layouts and re-uploads when the content really changed, so idle segments upload nothing.

Which side the scaling lands on

Sharpness is decided entirely by the size at which the scaling is computed. Text takes the CPU path: the actual font size is derived from the frame height and the user scale, glyphs are rasterized directly at that target size, and the one-pixel anti-aliasing coverage at the edges is recomputed for the new pixels. Scale up two times and the outline is still one clean one-pixel edge.

Text re-layouts on the CPU at target pixels; tracks invert the mapping into sample points on the GPU

Tracks (ASS, PGS) can only take GPU resampling. The canvas size is set by the source and the typesetting engine and cannot be re-laid-out the way text can; a bitmap subtitle has no "do it all over again at the new size" option to begin with. The approach is to compute a mapping that pulls the canvas content to the target position and size, then invert it into sample points; when shrinking, sample points can land outside the canvas, and out-of-bounds samples must be discarded explicitly.

The two sides must not both do it. The text slot has already folded scale, offset and opacity into font size, position and alpha, so the layer side only receives unit values; doing it on both sides means scaling twice. The pivots differ too: text scales around its alignment anchor while tracks use the frame center — but the two slots are never on screen at the same moment, so it never shows.

What is left of a backend is a thin layer

The layer interface is backend-neutral: update the canvas, clear it, push transforms, report content invalidation. Layer pointers change whenever the graph rebuilds, but the content must not be lost with them. A stable front end holds a few things: the authoritative copy of the transforms, updates that arrive during graph building parked until later, and the content-invalidation signal. The moment a rebuild happens, the layer pointers are replaced, the front end re-sends the parked transforms to the new layers, and the invalidation signal makes the view re-upload the canvas, so subtitles never drop out. The backends all share this one front end.

What remains per platform is three things: a texture upload call, a blending shader of twenty to thirty lines, and the pipeline hook points — one backend comes to one or two hundred lines. The platform leg does exactly one thing, blending the canvas onto the picture; it never needs to know any subtitle format.

Two boundaries have to hold. First, no subtitles may be stacked on the lane that feeds Vulkan, or subtitles double up, get blended before enhancement, and are resampled as part of the picture. Second, graph rebuilds are routine on this chain — a resolution change, a software/hardware decode switch, toggling the watermark, attaching a layer for the first time all trigger one, and a rebuild throws away the layer nodes together with their CPU and GPU resources. So the transforms must live in the resident front end and be re-sent after the rebuild, and the content needs an invalidation signal so the view re-uploads, otherwise subtitles stay gone until the next content change. The canvas layer itself is sticky: once attached it stays for the whole playback, and with no subtitles only the content is cleared.

Upload timing matters too. Every canvas update carries a content sequence number, and the view deduplicates by it: if the number has not moved, nothing is re-uploaded. One line of dialogue sitting on screen for three seconds uploads exactly once, on the first frame. Different subtitle sources use different number domains, and switching slots resets the count and starts over.

The canvas in HDR passthrough

In ordinary output, the canvas is composited with the picture inside the Vulkan pipeline, in the standard color space and standard gamma: subtitles and picture take the same pass, and blending happens on the pipeline's layers. In HDR passthrough output, the video signal bypasses the pipeline and goes straight to platform-native rendering, and the canvas switches with it to the platform leg. There the three domains get three treatments. The standard-gamma domain is the usual canvas. The PQ code-value domain uses a canvas pre-encoded on the CPU: the RGBA canvas is quantized against PQ on the CPU side, standard white maps to its code value in the PQ domain, the alpha channel skips the quantization table, and the blending code is exactly the one from the standard domain. The 16-bit floating-point linear domain linearizes in the shader first and then multiplies by the reference white. The reference white answers how bright linear 1.0 is: 203 nits, the quantized position of standard white in the PQ domain, while the Vulkan side counts 1.0 as 80 nits and the Metal side counts it as 100 nits — one contract, each platform remembering its own white, and subtitles follow whichever lane they ride. When the output domain flips, one content invalidation is raised to force a re-upload.

The blending code a platform leg has to write is about twenty lines no matter the domain: sample the canvas, blend it on with the premultiplied formula. Every domain difference is folded into the canvas's form and the first few conversion lines at the top of the shader; the contract itself does not move.

The engine avox is open source on GitHub: github.com/xxxzhou/avox, where the canvas, the stable front end and the four backends all live.