Two queue stages on one chain: packets and frames
This chain is three roles joined together: the read thread turns an address into packets, the decode thread turns packets into frames, the render thread turns frames into picture and sound. A queue sits between each stage — the packet queue before decoding, the frame queue after. The gap between speeds is absorbed by these two queues: the faster side finishes its work and piles it in the queue, the slower side takes it at its own pace.
The queues hang off tracks: one packet queue per track, one frame queue per track, video and audio each on their own. This decides everything that follows — pressure is accounted per track, not spread across the chain.
The depth of each queue is measured against the downstream weakness. The packet queue is left deep to leave room for audio and video not marching in step: at startup the two tracks' data does not arrive in step, and their timestamps can be offset by a stretch; too shallow and it spins waiting for the other side. The video frame queue, conversely, is left shallow — hardware-decoded frames sit on GPU resources, and queued too long the resources may already be reclaimed by the time rendering reaches them, and the picture comes out corrupted. The audio one is left wide, because the sound line dreads a break most.
The queue has only one rule of its own: once full, an enqueue does not return and does not drop data, but waits on a condition variable until there is room. The only thing that releases it early is the queue being closed — closing sets a closed bit and wakes everyone waiting; without that, a thread could wait forever for room when it should exit. Pressure therefore travels back one stage at a time: the downstream slows, its upstream queue fills first, that stage's producer is blocked, then the next one up... all the way to the read thread. Nothing on the chain has to notify anything else; the blockage itself is the signal. The reverse direction is debt: when upstream cannot supply, the stages below run empty in turn and rendering gets no frames.
With the chain laid out, the three segments are walked through from the start. Each is told the same way: what it does, which queue it moves, and what the two queues look like when something goes wrong.
The IO segment: turning an address into packets
The read thread does one thing: call the low-level read-packet interface, turn the address into packets, and push them into the packet queue. It does not know what downstream is doing and does not ask — if it does not fit, it waits at the enqueue.
The cost of waiting depends on where it gets held up. A local file costs nothing: reads are fast, and the yield at the end of the loop only gives up a time slice without sleeping, so the packet queue fills quickly and the read thread is soon stuck at the enqueue, still burning CPU. A network source is different: once the queue is full and the read thread stops calling the read-packet interface, TCP's receive window fills with unread data and the sender is held back; stay stopped and the peer or some device in between may judge this silent connection dead — TCP will not do that itself; it takes an application-level write timeout or an idle timeout on an intermediate device. The read timeout cannot save this situation: it protects the call that is "reading but getting nothing back", while a thread stuck at the enqueue is not reading at all, so the timeout has no chance to fire. This layer has no safety net: a read thread stuck at the enqueue indefinitely, and the socket thread connected to the server may be shut down.
The two legs even block in different places. The ffmpeg leg has a read thread of its own, and that is what blocks; the zlmediakit leg starts no separate thread, and packets are called back on its socket thread, so a full queue blocks that socket thread.
When IO goes wrong, things actually get simpler. The read loop looks like it has many branches, but they come down to three tiers: keep reading, reached the end, cannot read back. A read call returning "try again" or interrupted by a seek both fall in the first tier — continue to the next round and nothing happens on the queue side; the last two tiers are the ones that need wrapping up.
Reaching the end is normal: tail packets may still have tens of seconds left to play, so the source does not exit the read thread; it notes "end reached", reports the end once, and stays parked there — so that a later seek still has someone to read. On the queues it looks like this: no new packets come in, packets already there keep decoding and playing, the packet queue and frame queue drain slot by slot, and only when all four queues bottom out does the player enter completed.
An error is another matter: the read loop exits outright, no more packets are produced, and the player changes no state at all — the error is merely recorded and called back up, and the layer above decides what to do. The real consequences are indirect and all land on the two queue stages: no more packets, so the packet queue drains slot by slot, decoding finds nothing to decode and the frame queue empties too; rendering cannot get a frame, and both packet queues are empty at this point, so by the render segment's criterion it enters buffering, then the watchdog times out, and playback is closed. So what a user sees from a dropped network is not "an immediate error" but "it froze, and after a while playback stopped".
One more tier sits earlier: the source never opened at all. That happens before the read loop and reports up the same error path; it is just that the queues were never filled, so there is no draining process. Reporting end and error separately is deliberate: the error is for the layer above to decide on, and the player does not decide for it.
The decode segment: turning packets into frames
The decode thread takes packets from the packet queue, decodes frames, and pushes them into the frame queue. When the frame queue is full it jams at the frame-push step, and the rate of taking packets drops accordingly — backpressure in this segment takes the form of "stop taking".
When decoding goes wrong it first patches things up at its own level, not rushing to alarm the layer above. When not a single frame has come out yet, a few consecutive failures judge this decode path unusable — failing to create a hardware context or a driver that never produces frames both count. At that point it switches to the next decoder within the track, replays the cached config frames, and decodes from the start again. Switching lanes does not touch the packet queue: packets keep being supplied, neither closed nor cleared, so during this segment the frame queue empties and rendering spins, but the player does not report buffering — packets are still piled up, decoding just cannot keep up. At worst the user sees the picture reconnect.
Only after every decoder has been tried and still nothing works does it give up: close this track's packet queue, drain the residual packets inside, report the error to the player, and exit the decode thread. The order of this step is directly tied to the queues. By the time it gets here the packet queue is full — the read thread has been stuffing it all along, just waiting for decode to take from it; without closing and clearing first, the read thread would block forever at the enqueue, the other track's packets would starve too, and the whole pipeline would freeze. That audio can keep going when the video line dies depends on this step releasing first, so the queue must be sealed before the error is reported and the thread exits — not the other way around.
After reporting to the player, handling is the same as an IO error: the player changes no state itself and passes the error up. There is exactly one exception — a video track decode timeout makes it close and reopen outright. Other decode failures still wrap up through the buffering path: no more frames, the frame queue drains, and the packet queue has already been cleared, so both are empty and it enters buffering, the watchdog times out, and playback closes.
Failure is judged finely: a packet that is itself bad and yields no frame does not count as a failure — skip it and keep decoding; an occasional frame-collection failure after frames have already come out just writes a log line, neither dropping packets nor exiting. Only consecutive failures before any frame has appeared count as this lane being dead.
This path's watchdog is also only in force before the first frame comes out: only a steady feed of packets that produces no frame for a long time judges it dead. Once frames have appeared the watchdog is withdrawn, and if the decoder afterwards quietly stops producing frames without reporting an error, nobody is specifically watching it — its remaining hope is the render segment's criterion, and that criterion cannot see this case.
The render segment: turning frames into picture and sound
The render thread runs on the frame-rate beat: take a frame when due, and spin when there is none — the picture shows nothing, but it does not stop and wait; the next beat still comes around for a frame. It is the last link in the chain and the only role that looks back at the queues.
When it is slower than frame production, pressure starts here: the frame queue tops out first, decode is blocked at the frame-write step and its packet-taking rate drops, the packet queue fills in turn, and the read thread blocks at the enqueue. The two tracks top out at different speeds: the video frame queue is shallow and tops out at the slightest slowdown; the audio one is much deeper and takes a while to fill.
When it is faster than frame production, the frame queue empties and it spins. "Frame queue empty" alone settles nothing: it has already appeared several times in the previous two segments — slow IO does this, an IO error does this, switching decode lanes does this; spinning is just the same shadow left on the render side by several different causes. To conclude anything it has to look back at the two packet queues: only when both the audio and video packet queues are empty does it count as "there really is no data behind this" and enter buffering; as long as the packet queues hold stock, it is only decoding falling behind, and nothing is done. The criterion takes the packet queues rather than the frame queue for exactly this reason — an empty frame queue cannot distinguish which of the above it is.
So the same glance can lead to very different outcomes. Slow IO and an IO error both enter buffering, and the user sees the same thing — the picture freezes, and after a while playback is gone; the end-of-stream path looks no different on the queues, and is picked out by the "IO ended" flag outside them, going straight to completed. The decode level is worse: when it stalls the packet queue is full instead, and this glance cannot see it.
The audio path is not looked at in frames: its beat belongs to the device, and all that can be asked is how much the device still has backed up and unplayed; the duration converted from the backlog does not match PTS over the long run, so what is adjusted is the amount of data sent out, not the frame queue. The whole set of beat alignment and dropping frames when behind is in the audio-video sync article; here only its shape on the queues is described.
What happens after entering buffering, how much cache must accumulate before recovery, and how a watchdog timeout is handled were all covered in the command queue article and are not repeated here. What needs adding is the one thing this directional criterion measures: it governs "production stopped", not "production is still going and consumption stopped" — when decoding stalls the packet queue is full, which looks like normal playback to the criterion, while decoding's own watchdog was withdrawn once the first frame appeared. This case currently has no dedicated safety net.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, and this chain, the two queue stages, and this wrap-up path are all in there.