The player's seek: landing points, precise seek and switching decoders

Chapter 6: Player refinements·about 1270 words

Sections and series

Chapter 1: The platform-independent player

Chapter 2: Windows

Chapter 3: Android

Chapter 4: Apple

Chapter 5: WebRTC

Chapter 6: Player refinements

Chapter 7: Recording: frame sources, software encoding, muxing and streaming

Three landing tiers: true IDR, recovery point, drop to the next entry

First, what failure looks like. On some remuxed sources the picture freezes right after a seek and the frame queue stays empty; on others the screen stays black for a few seconds and the error log floods the moment it recovers. The root cause is always that first step: the only frames in a video stream that can be decoded on their own are IDR frames, and restored-edition transcodes stretch the GOP past twenty seconds to save space — in the extreme, the whole stream carries only open-GOP recovery points and not a single true IDR. When the landing point sits in the middle of a group, the decoder starts from a missing-reference state, the frame-number gap carries all the way through, and the count of valid decoded frames stays at zero.

So after a seek you cannot rush packets into the decoder; first look at what the landing point actually is. I built this as a gate on the source side: during the protection window every packet passes through it, and it picks the entry point by priority.

The three entry tiers of the seek landing gate: truncate, pass the recovery point, drop

The first tier is the cleanest: the group contains a true IDR, so truncate the data to it and start from there. The first NAL of a group cannot be trusted — the new-frame heuristic can misjudge an IDR slice as "no new frame started" and merge it into the previous group — so what it does instead is scan the whole group for random-access NAL units and truncate the group to the first one found. The leading B and P slices that get cut off are missing references anyway; feeding them in would only produce gray frames.

The second tier is a keyframe group the container marks, that is, an open-GOP recovery point; it is passed through and used as the entry directly. It does not guarantee a clean picture, but with the two preconditions below the recovery window is short, and a brief stretch of artifacts at the start is acceptable. Sources with no true IDR anywhere in the stream can only take this tier.

The third tier is a missed landing: the point falls between P and B frames, usually because the container index itself is broken. Here the gate drops first, and keeps dropping until the next usable entry appears. Dropping is far cheaper than decoding, but it cannot go on forever: if a stream truly has no keyframe at all, the gate lets packets through once five thousand have been dropped, so the screen is not black forever.

The criterion is the container index's keyframe flag, but containers differ. In mkv the keyframe flags in the index are seekable points as the muxer judged them, recovery points included, and can be trusted; some containers stamp every packet as a keyframe, and taking that as the criterion scrambles everything. So the gate is only enabled for H264 and H265, where the container flag is re-checked against NAL semantics in the bitstream, and only a trustworthy index like mkv's is taken at face value.

Beyond the three tiers sit two preconditions, the baseline that makes these sources usable at all. One: the decoder is allowed to output corrupt frames — frames with missing references are emitted too and heal themselves against the normal reference frames that follow, instead of stalling the whole stream. Two: audio has a protection window; audio packets are held back until the video's first I frame shows up. After a seek, audio running ahead while the picture is still frozen feels badly out of joint, and this rule pins the two ends together.

Precise seek is an option that defaults to off

The three tiers answer where decoding starts. There is a second question: should the first displayed frame land exactly on the target position?

Fast-forward style displays from the landing I frame; precise mode drops frames to the target

The default is the fast-forward style: display starts at whichever keyframe the landing point sits on, so the picture may begin some distance from the target, but not one extra frame is decoded. Precise mode is the reverse: decode silently starting from the entry frame, drop every frame whose timestamp is earlier than the target, and display the first frame that is not earlier. It is off by default, turned on when needed, and the command-line player keeps a matching parameter for it.

The implementation lives at two ends. On the video end, dropping happens before a frame is enqueued: frames whose timestamp has not reached the target never enter the queue, the first frame that qualifies clears the drop automatically, and a flush resets it. On the audio end, packets from before the target are dropped on the source side; packets with no timestamp at all must pass — audio timing does not have to be carried by packets in the first place, and untimed packets are advanced by the number of decoded samples, so dropping them would dig a hole.

The cost shows up at the other end: with a twenty-second GOP, precise seek under software decoding stays black for one to three seconds before reaching the target, and that stretch is silent decoding that has not yet had its turn to be displayed. This is an inherent cost of this kind of source, not a defect anywhere. One more boundary comes from the container index: when the index is sparse, the landing point may already lie past the target, there is no moving back, the drop condition never holds, and playback can only start from the landing point, degrading into the fast-forward style. The index sets the ceiling; the player cannot make up for it.

To verify that the two modes really land on different frames, the evidence cannot come from the UI: the position in the screenshot's filename gets pinned to the target value during a seek. What is reliable is the timestamp of the first frame actually decoded in the decode log. Same source, same seek, target a little past twelve seconds, landing point around ten: the fast-forward style's first frame timestamp is ten seconds, precise mode's is a little past twelve — two milliseconds after the target.

Switching decoders: rendering rebuilds along with it

Swapping software decoding for hardware mid-playback is done by closing the decoder and opening it again; the timing and the rebuild itself belong to the decoding article. Here I only add the two things on the rendering side.

First, the render objects have to be rebuilt together with the decoder. Software decoding hands over frames in memory, hardware decoding hands over textures; the frame queues on the two sides do not even share a shape, and the render side holding the old set cannot catch the new one. This also demands that decoding, rendering and window stay three separate things — the window only does display, otherwise every change drags the window along with it.

Second, when the decode context is rebuilt, the device itself does not move. The hardware decode device is one device shared by decoding and rendering — the Windows article covered how it is created and handed over for reuse; when the context is rebuilt, the existing hardware buffers are hung back on it and the device stays as it is, so the shared-texture copy path between the two sides does not break.

When the stream's resolution or sample rate changes midway, what is needed is also a rebuild, and it reuses this same path — no separate one is written.

The engine avox is open source on GitHub: github.com/xxxzhou/avox, where the landing gate, precise seek and this rebuild path all live.