RTSP VOD seek: the server gives the picture, the timeline you fix yourself

Chapter 6: Player refinements·about 1189 words

Sections and series

Chapter 1: The platform-independent player

Chapter 2: Windows

Chapter 3: Android

Chapter 4: Apple

Chapter 5: WebRTC

Chapter 6: Player refinements

Chapter 7: Recording: frame sources, software encoding, muxing and streaming

Local files and VOD streams are not the same seek

With a local file the player holds a complete index and knows where every frame sits in the file; dragging the bar is just moving the read pointer. Timestamps are written in the file and stay continuous whether you seek or not.

RTSP VOD has no index, only a session connected to the server. All the player can do is send a play command carrying a target time — the command carries normal play time, a position on the stream's own timeline; the whole recording-playback family works this way. On receiving it, the server positions the picture at the nearest keyframe around the target and starts sending the stream from there. This step sets the ceiling on seek precision, and it plants the trouble that follows: the server guarantees the picture arrives at the mark, not that the time labels arrive with it.

Why is continuation the most common behavior? Timestamps are stamped on the server side all along the way; the seek only moved the read position and never touched the stamping logic, so packets keep counting up from the old numbers. Some servers do restamp on seek, but the point they restamp from is not necessarily the duration actually skipped — so both continuation and wild jumps are seen in the wild, with no telling beforehand which one you get.

Local seek moves the read pointer; VOD seek relies on the server to position

Why the packet timestamps cannot be trusted

The server lined the picture up with the target, but its timestamp generator was not necessarily reset. The packets that come back behave in one of two ways, and there is no telling beforehand which: continuation, where timestamps keep climbing from where they stood before the seek — once I sought to fifteen seconds and the packets coming back were still ticking up around six; and a jump, where the time does jump but by an amount that does not equal the duration actually skipped, and video and audio may each jump on their own.

The result is that the picture and the time labels come unhooked: the picture is already at twenty seconds while the packets say twelve. The labels themselves are not broken; they are labeling the wrong thing.

Flowing downstream, these packets poison a whole chain: packet time becomes decoded-frame time, then rendered-frame time, feeds the clock, the clock supplies the current playback position, and the playback position is exactly where the next seek starts. One wrong number up front and every link behind it is wrong; the further it drifts, the further the next seek lands off.

The only number on this chain that can be trusted is the target time the player itself sent out. It corresponds to the position the user actually wants to watch and has nothing to do with how the server stamps time. The fix is built around it.

Fix it at the packet entry, anchor on the first packet

Dirty data comes in through the packet entry, so it gets corrected at the packet entry, and the decoding, rendering and clock behind it never need to know anything happened. The placement was chosen, not inherited: the other candidate was to correct after retrieval, but that means every consumer corrects its own copy and the logic scatters across call sites; fixed at the entry, this leg delivers repaired time and not one line downstream changes. The fix ended up in this VOD leg's own IO implementation: local files run on a different leg and live streams never seek, so the shared logic is untouched.

The first packet as the anchor, shifting the whole stretch of packet time onto the target

The method is three steps. At seek time, record the target time; when the first packet after the seek arrives, subtract the packet's own time from the target to get an offset; from then on every packet gets the offset added. Whatever the server stamped on the first packet, twelve seconds or twenty-five, it is pulled to twenty, the packets behind it shift along with it, and the gaps between packets stay as they were. The anchoring does not distinguish continuation from jumping — neither can be predicted, so both are anchored the same way on the first packet; a continuing stream comes out right after one anchor, and a jumping stream holds just as well.

The increment used for the shift is the time difference between adjacent packets in the same stretch of stream. That difference is stamped by the encoder and is reliable; no wall-clock estimation, since wall clock drifts under network jitter.

Video and audio each keep their own anchor, but anchor to the same target time. Keeping separate targets is what would let the two sides drift further and further apart; the offsets are kept separate because the two first packets do not arrive at the same moment and the server may stamp the two streams wrong in different ways — one offset cannot serve both. With both anchored to the same target, the gap between them stops growing, and the source-side audio-video alignment check passes.

Nothing accumulates: the offset is recomputed at every seek and only lives between two seeks. If one seek lands a little off, the next one starts over from scratch and the error does not carry forward. The seek target is taken from the current playback position, which is render-side time, never touching the distorted numbers inside the IO packets.

Error and limits

Precision is bounded by the server's positioning granularity and floats around one keyframe interval. The server can only look for the nearest keyframe each time; the target may sit closer to the previous one or to the next, and this error is independent on every seek — it does not accumulate. If the packets arriving right after the seek are mixed with leftovers from the previous stream, the anchor lands slightly off, so a seek first clears the packets already cached in the player; with re-anchoring on every seek on top, that little residue usually has no effect.

One more premise to state plainly: once packet time is distorted, there is no way to tell from the packets themselves which frame the server actually lined the picture up with, so the error cannot be repaired to anything finer than a keyframe interval. The information itself is missing; it is not a matter of twisting the algorithm a little harder.

The path that got deleted

On this road I tried one direction: when a large timestamp jump showed up, let the jump pass straight through and trust the downstream to adapt. It did not work — it could not save the streams that never made one clean jump but only jumped erratically; and once enabled, normal old streams actually accumulated drift, because the anchor was only fixed after the first packet and the offset came out too large. In the end I deleted the whole thing, from the server-side submodule through the player wrapper to the command-line entry, and new and old streams now take the same road: continuation or jump, anchor on the first packet.

The engine avox is open source on GitHub: github.com/xxxzhou/avox, where the packet time rewriting on this VOD leg lives.