The source exposes only open, close, and a few queries
The interface an encoded data source offers upward is surprisingly small: open, close, set speed. Set speed is a rate semantic for live streaming and unused in VOD; day to day only two actually turn — open and close. Everything else is a query: can it seek, how long is it in total, where is it playing now — none of these change the source's state.
Open receives an address, possibly a local path, possibly a network URL, and how to fetch it is the implementation's own judgment. The source also carries a mode bit that distinguishes local playback, download-while-playing, and live — every policy downstream follows it: how thick the cache is, how high the buffering watermark, whether seek is possible. With a cache of the same thickness, VOD can rewind at will, while live has no such thing as "rewind" at all.
The interface is kept this tight on purpose. The layer above wants "give me packets"; whether they come off a disk or down the network is something it should not know.
What implementations this data source layer has
This layer has just one base class: track descriptions live on it — video width, height, frame rate, and bitrate; audio sample rate and channel count; subtitle tracks; total duration. These are handed over after open and before any packet comes out, so the layer above only has to read the track descriptions; it never needs to ask first "are you a file or a camera".
The layer that does the work directly under the base class is called AVSource, the encoded data source: what it emits is compressed stream packets — H.264, H.265, AAC and the like. The common parts are implemented in it — PTS alignment, packet splitting and merging, config frame management; how exactly to fetch data is left to subclasses. There are six subclasses for now: ffmpeg and zlmediakit are compiled into the engine, while network shares, magnet and BT, and disc images are brought in by plugins — install the plugin and the registry grows a leg; without it, the address falls to the default leg. The player does not know plugins, only the registry.
Upward there is a set of interfaces and an observer. The action side is open, close, set speed; the information side is track count and each track's description, whether it can seek; data comes back through the observer — ready, sync reference, packet, error, end, each a callback. The layer above implements one observer, and this layer shuts all the differences at the other end of the address outside.
Demuxing: file and network on one end, clean packets on the other
Once the address is open, the container is digested: packaging schemes like mp4, mkv, and ts end at this layer. The output of demuxing is very specific — clean, single-frame stream packets.
The two words "single frame" take a fair amount of work. Data from the network often does not arrive cut on frame boundaries: a single P frame usually forms one packet, config frames like to be sent combined; an I frame is large and often split across several packets. Some streams start with a three- or four-byte start code, others with a length prefix, and that has to be recognized before it can be restored. All of this has to be restored into a shape the decoder can consume directly.
Timestamps get straightened out here too. Time bases inside containers vary wildly; outgoing packets uniformly carry presentation time and decode time, so downstream never has to ask "what unit is this number in". A reference has to be set as well: video uses its first I frame as the reference, audio with video present follows the video's I frame, and if the two diverge too far they each keep their own rather than being forced together; a mid-stream timestamp jump resets the reference; extreme out-of-range values are blocked outright, so they do not drag the decoder down.
The two sides of the boundary know nothing of each other. Upward, the player does not know whether packets came from a file or the network; downward, whoever fetches the data does not know whether the packets end up rendered or recorded. Hold this boundary and later changes — swapping IO schemes, adding recording — need no change to this contract; the reverse direction is nearly free: hand the packets to a muxer and write them out, and that is recording.
The packet is the smallest handoff unit, with every field in one go
"Packet" is the handoff unit because every handoff on the chain uses it: entering queues, crossing threads, dropping by frame, calling back by frame — the granularity is always the packet. So the fields have to come complete in one go, and downstream never has to ask again.
A packet carries: type — video, audio, subtitle, three track kinds, each with a data packet and a config packet tier; presentation timestamp and decode timestamp, which differ when B frames are present, rendering using the former and decoding the latter; packet duration, which carries audio forward when it has no timestamp; the keyframe flag; and the stream format prefix length — three or four bytes at the front for h264/h265, the first seven for aac possibly an ADTS header, by which the decoder recognizes the format. Besides 0 (ordinary) and 1 (keyframe), the keyframe flag uses 2 and up for hint frames signaling a config change.
The buffer grows by reuse: if an allocated block is larger than the data it must hold this time, it is kept for later use, so packets avoid reallocation as much as possible. The cost is that packets grow larger toward the end; what it buys is fewer allocations.
Config frames: first into the queue, and a copy before every keyframe
A few kinds of packet in the stream contain no picture: H.264's SPS and PPS, plus a VPS for H.265, and AAC's sample rate and channel information. They are the decoder's manual, and without the manual the decoder cannot start — H.264 configuration needs at least two NALUs, H.265 needs three, and one short means parsing fails outright.
So two rules: config frames enter the queue first, and before the decode thread starts its loop the cached configuration is fed to the decoder; and it must be sent again before every keyframe. The second was added later: once parameter sets are in the decoder they stay there, and re-injecting them is an invariant I set for myself, which solves three scenarios at once: a viewer joining mid-stream gets a keyframe as the first packet, with the configuration right in front of it; switching a track back, the old track's queue was cleared, and the cached copy is re-injected; switching decoders on downgrade, the new decoder likewise stands up on replayed configuration. AAC has a small pitfall too: some streams have no configuration data at all, so the first ADTS header is promoted to a config frame to complete the manual.
The keyframe flag: marked at the IO layer, consumed in five places downstream
Every packet carries a keyframe flag, marked at the IO layer: the container's flag comes in first, then h264/h265 re-check it against NALU semantics — some containers mark everything as a keyframe, and without the re-check every piece of logic behind it would be misled.
This one field is consumed in five places downstream: detecting I-frame mode (two all-I-frame streams at different times in a row automatically switch to I-frame mode); discarding after seek (drop non-keyframes until a keyframe is hit, with a sanity cap); decoder reset (the reset does not actually act until just before a keyframe packet); when downgrading hardware decoding and switching lanes, packets are dropped until a keyframe absorbs them before decoding starts; above 4x speed, non-keyframe packets are dropped right on the read thread. One field runs the whole chain and does five different jobs — which is also the benefit of making the packet the handoff unit: every downstream reads the same fact.
Six IO routes: the default, upper-layer rules, and automatic hits
The engine side gives only a default: open and you take the ffmpeg leg.
The rules are really set above. The player leaves an entry point for switching, and a host that wants its own address rules — which protocol takes which leg, which suffix takes which — writes them at this layer. That is what the command-line tool does: it maintains its own table, sending real streaming protocols like rtsp, rtmp, srt, and streaming suffixes like m3u8, flv, ts to zlmediakit, and the rest to ffmpeg. This mapping lives in the tool's code; another host can have another set of rules.
One rule needs no attention from above: at the moment of opening, the address picks the lane. Direct rtsp and rtmp streams go to the zlmediakit leg; magnet links or .torrent, network share addresses starting with smb:// or dav://, and .iso Blu-ray images go to their corresponding plugin legs — a hit only happens if the plugin has registered; without it, the leg chosen above is still used. This affects only this open, and does not override the choice the upper layer made.
The six legs implement the same interface and hand up the same packets and the same events, so the player cannot tell at all which leg it stands on.
Taking apart RTSP and RTMP protocol stacks yourself is heavy work — control information and media data are mixed on the same connection, and block format, serialization, ordering, and retransmission each have their details. I did not chew through this layer myself; I brought zlmediakit in as another data source implementation instead.
Reading data takes an independent thread, but the thread name is not a role name: "ffmpeg io parse" is the name this leg gives itself. Swap the leg and both the thread name and the thread itself change — the zlmediakit leg starts no separate read thread, packets are called back on its socket thread; the plugin legs each change the name to their own. The rule is on the interface side: whoever implements a data source is responsible for managing its own thread and giving it a name the logs can recognize.
What is special about the live path
Three points. First, live has no notion of position: a VOD seek is "jump to near where I last watched", and in live that verb does not exist; after a break you can only continue from the current live point. Second, the latency bill is paid in buffering: the thicker the buffer, the more jitter it absorbs and the further behind the viewer's picture; keep it thin and a single jitter stalls it. Third, failures look different: a VOD error is mostly "cannot read", a live error is "the stream broke", and a broken stream has to be able to come back on its own, and if it cannot there has to be an account of it — reconnecting is a full restart, fetching the description again, waiting for config frames again, and both the count and the time must be capped; for a live stream that is truly dead and has no owner, unlimited retries only fail and retry forever.
These three points mean live cannot be just "a different way of opening"; it needs a rhythm of its own. Three places in the protocol are easiest to misread: sorting (control and media packets mixed on one connection, separated by their leading bytes), frame assembly (one network packet is not one frame; config frames are sent combined, large frames are cut into dozens of packets, and only a complete set can be assembled), and ordering (the network does not guarantee arrival order while the decoder demands order, so everything passes through a sorter first). Timestamps are a trap too: the field in the header is a counter, not time — video advances 90,000 counts per second, and it has to be converted to milliseconds before it can enter the clock.
Startup cannot play one frame as it arrives either — a single network jitter stalls it right after starting; the approach is to accumulate past a threshold before showing picture, a compromise between low latency and no stalling. When the connection comes up, the server sends not picture but a description first: how many media streams, which codec, which sample rate; channels are built only after the description, and it counts as truly ready only when the config frames are complete. Waiting for readiness has a budget — some devices are simply slow to produce the first frame, and giving up too early kills them wrongly, while waiting too long means the viewer is long gone.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, and this data source layer, the six IO implementations, and the live-path parameters are all in there.