The three muxing levels: the interface, the packet-only level, and the level with encoding
The three levels stack. At the top is the interface, describing “open, close, current state” plus a few settings (whether to use hardware encoding, which codec to encode). The middle tier does muxing only: audio and video packets are stuffed in by type, and it does not care where they came from. The bottom tier adds two encoded streams on top of that, one video and one audio, taking raw frames — YUV planes, GPU frames, PCM data blocks — and handing them up a level once encoded.
Each of the two streams owns three decisions for its track: how the description is set (what size and format the encoder wants), which encoder to use (the candidate chain, the same set described in the recording-chain article), and where the encoded packets go.
Opening the stream starts by preparing both tracks' parameter sets
Without a configuration frame for video, this output cannot open at all — configuration frames have to be merged into a segment and put into the stream's description. On the audio side, AAC needs a decoder-configuration header computed on the spot. Both have to be ready before the header is written.
Writing out has only one queue and one thread
The output side holds an output context and a queue, and is a thread of its own. Audio and video packets do not get two queues; they all go into the same first-in-first-out queue, which has a cap and blocks the caller that enqueues when full.
Interleaving is not sorted out by comparing timestamps ourselves: packets are handed one by one to the muxing library's interleaved write, which orders them by the two streams' timestamps. The write thread does three things in its loop: accumulate access units, drop packets that should not land, and hand packets to the muxing implementation.
Access units are accumulated because one sample in the container is one frame: several packets with the same decode timestamp are slices of the same frame and must be merged into one access unit before writing, flushing the previous one as soon as the timestamp changes. Configuration packets do not take part in merging; they pass straight through.
When configuration changes midway there is one more step: the new configuration frame is injected a little before that keyframe, so the decoder has the manual before it uses it.
Three kinds of packets are dropped. One is the separator packet used for padding (AUD), which often shares a timestamp with the real data frame and would break the sample if kept; one is a pure-metadata access unit with no image data, which, if written, would leave a lone access unit at the start of the file; and one is a packet with a decode timestamp below zero — a frame whose timestamp itself is wrong, which would flash at a wrong point in time when written to the file.
Stream initialization waits here too: the stream is created and the header written only once the first keyframe arrives, so the first access unit of a file is a keyframe.
Winding down has two parts: on exit, flush the last accumulated access unit; after closing, the packets still in the queue have to be drained — in fast offline remuxing, the last few dozen frames are often still piled in the queue, and dropping them truncates the video. This draining has a time limit as a backstop, so shutdown does not get stuck on a target that cannot be written.
Timestamps: zero on the first packet, then convert to the stream's time base
Timestamps travel the whole chain in milliseconds, and the encoder is millisecond-based too. At the write step two things happen: the first packet becomes the base and is subtracted from every packet's timestamp; then there is one conversion to the target stream's own time base.
The base is the first packet's decode timestamp, not its presentation timestamp. In a stream with B-frames, the first frame's decode timestamp may be smaller than its presentation timestamp; with the presentation timestamp as the base, packets ordered by decode timestamp would come out negative, and those negative packets are exactly what the drop rule above catches — the keyframe at the start gets dropped, the file has no keyframe at its beginning, and the player shows green or garbled frames for the first few seconds until the next keyframe. With the decode timestamp as the base, no packet is below zero.
Audio has one extra piece of work. Audio packets coming from the source may have no timestamp, which is more common in live streams. The playback side does not rely on packet timestamps: the decoder advances along the sample axis by itself. The recording chain has no decode axis, and a timestamp-less packet coming straight in would poison the base, so a timestamp is synthesized here — packets with valid timestamps serve as anchors, the step is learned between anchors, and the first packet starts from zero. This is also why progress only follows video timestamps: those synthesized audio timestamps jump back and forth, and using them to report progress would make the progress bar bounce around.
Two no-decode uses: whole-stream remuxing and single-track direct copy
A sibling hangs off the same muxing path: remuxing. It neither decodes nor encodes; the encoded packets from the source are forwarded into the muxer as they are, and speed depends on how fast reading and writing go; it has no extra thread and no frame-processing segment.
The two recorders are dispatched by the same factory on one boolean: “decode and re-encode” or “remux only”.
The re-encoding path can mix in one more mode: a track is not re-encoded and its packets are copied over directly (video re-encoded, audio copied, for example). Direct copy has to ask whether the target container can hold that track's codec — if it cannot, the track cannot be copied and must be re-encoded; the audio check hangs at stream-open time, the same criterion as switching mp4 to mov earlier.
Both chains end up on the same muxing implementation, and the output target in both cases is one line of address.
Empty output: capturing frames and extracting audio
When the output is left empty, the chain does not build a muxer at all; frames reach the outside only through the processing segment's callback. Frame capture uses exactly this path: grab a single image, or hand every frame to outside processing.
Audio has another way out: a tap hangs on the processing segment, with its own queue; when full it drops the oldest and does not block.
These two capture paths use the opposite queue policy from recording. Recording would rather stall decoding than drop a frame — it is responsible only for the file's completeness; the tap path would rather drop than block the thread feeding it — it serves the real-time side, where slower recognition or a bit more resampling must never make the sound stall.
Here the packets land on disk. Change the same output line to a network address, and how the format is chosen and what a write failure means is the next article.
The avox engine is open source on GitHub: github.com/xxxzhou/avox. The muxing layer, the write thread and those two timestamp steps are all in there.