Assembling the recording chain: decoding source, offscreen processing and encoder
The recorder holds only three things: a decoding source, a stretch of offscreen processing, and a muxer that can encode.
The decoding source is AMediaSource; selection, the candidate chain and mid-stream lane switching from the playback side all live inside it (the decoding strategy), and the recording chain does not write a second copy. The processing segment uses WindowRender, the coordinator class from playback, except that recording sets it offscreen: no window, and no thread turning with the screen refresh; whoever feeds it decides when a frame gets processed. The third piece is the encoding muxer RawMuxer, which hangs two encoded streams below it, one video and one audio (the muxing layer has several levels, covered in detail in the muxing article).
Frames come in through callbacks, not through a fetch interface. Decoded CPU frames, hardware-decoded GPU frames and audio frames each have their own callback; there is a fourth entry for the “no decoding, forward the source packets straight into the muxer” use case. All four entries are implemented by the recorder itself — it is an observer of the raw frame source described in the frame-source article.
The frame's processing path: offscreen processing and CPU readback
The two video callbacks bring in two different things: a CPU frame is YUV planes in memory, a GPU frame is a hardware-decoded texture. Either way it goes into the offscreen processing segment first, and when processing is done a CPU frame is taken back out of it.
The hardware-decode path has one extra step: the texture is first converted to RGBA by the platform pixel layer, then mapped into the Vulkan pipeline, the same way in as during playback; the shared-texture article went into that part. What happens in the processing segment is the same as during playback: scaling, color, watermarks, and optionally quality-enhancement filters.
Readback is the cost of this step: the image is finished on the GPU but not presented — it is pulled back into memory and into the encoding queue, a third destination besides the two output paths in the render-layering article. The Vulkan leg copies the image into a buffer, maps it and copies it out; the native legs on each platform use their own methods. This segment also decides what format is handed to the encoder afterwards — the encoder does not choose for itself; it reads planes according to the description from upstream.
Frame-queue backpressure: blocking the decoding side when the queue is full
Encoding does not happen on the decoding thread. A processed frame is deep-copied into a frame queue — one for video, one for audio, each holding only a few frames; the encoding thread takes from the queue.
The deep copy is not gratuitous: the processing segment's output lives in a shared buffer, and the next frame overwrites the previous one as soon as it arrives; without copying, the frame cannot be safely thrown into the queue.
When the queue is full, frames are not dropped; the enqueue blocks instead: the decoding thread stops at the enqueue, and the pressure travels back through the decode callback all the way to the IO layer. That keeps memory bounded — if encoding is slow, upstream has to be slow too. The price is that the whole chain's tempo is set by its slowest end: a per-frame inference filter like quality enhancement hangs on the encoding side, and once it slows down and the queue fills, the whole line goes at its pace.
On close, a producer stuck in the enqueue has to be woken up, or shutdown gets stuck there waiting on the queue.
The encoding side's loop is a single thread and watches both the video and audio queues: each round it takes one video frame, then one audio frame, and rests a moment if there is nothing. When to finish is judged here too — only when the source has ended and both queues are empty does it wind down and close the encoder. Seeking also lands on this thread, so it does not fight the IO thread over the same state.
Software-encoder integration: registration, selection and open-time configuration
Encoders go through the registry, the same routine as sources and decoders. At build time the entries in ffmpeg marked as encoders are enumerated, hung in the table by codec id, with ffmpeg's own encoder name in the description. Which one is chosen goes through the candidate chain: the ordering and the exit path are the same set as on the decoding side.
The preferred name comes from a platform macro — one default name each for software and hardware — and can also be injected from outside at build time.
Opening the encoder configures these. The time base is set to milliseconds, consistent with the rest of the chain, so frame timestamps do not need converting back and forth. GOP is converted from seconds to a frame count rather than hard-coded as a frame count. B-frames are turned off. The thread count is left to ffmpeg to decide based on the machine. Bitrate and preset are dispatched by encoder name: the two software entries use CRF plus a fast preset, the vendor-suffixed entries use their own preset and quality levels — one piece of configuration code going into different branches by name.
If the resolution changes midway, the encoder is rebuilt rather than scaled: it is reopened at the new size, and scaling is the processing segment's job.
Switch lanes only when the encoder will not open
Connecting an encoder involves only two things: a description, and a packet-output callback. Whether it will open only becomes visible at the first frame — an encoder that constructs but fails configuration only shows it the moment the first frame is fed in.
So this side has a lane switch too, but a much narrower one than decoding: if the first frame fails to open, it switches to the next candidate for the same codec and re-encodes this frame. It dares to switch at the first frame because not a single packet has come out yet — the muxer header is not written, the encoding description has not been committed, and switching implementation needs no cleanup behind it. Once packets have come out it no longer switches; one switch would mean the muxer header has to be redone.
Earlier this code simply discarded the return value of frame encoding. The price was that when hardware encoding failed to open, the whole file had no video, with not a single line in the log; the only way to find out was to notice by eye that the file had no picture.
What the encoder gets: pixel format and timestamps
The pixel format the encoder wants is not negotiated by the encoder itself. The processing segment fixes it by target when it produces the image: software encoding gets YUV420P, hardware encoding gets NV12, and the encoder reads planes in the format from the description when opened. There is no conversion layer on the video side — a format mismatch is not corrected, and the encoder reads according to the layout it assumes. So the format must be aligned on the output side; on this line upstream has the final say.
Audio is the opposite: the encoder accepts only a fixed set of sample formats and channel layouts, so the chain hangs a resampling layer in between — a frame comes in, is resampled to the target format, then handed to the encoder.
Timestamps are easy on this side: a frame's pts goes into the encoder unchanged (the chain uses milliseconds throughout), and the packets the encoder produces are rebased once. From packet to file there is another conversion to the stream's own time base, and that step belongs to the next article.
Winding down: natural end, termination during stream start, and a dead decoding lane
There are three cases for when this chain is done and when it has failed. If the source ends mid-recording (the peer drops the stream, the file is read to the end), it winds down as complete: the encoding thread drains both queues, then closes the muxer. If it is terminated by the source during stream start with not a single frame written, that counts as failure — most likely no data was ever obtained. If the decoding lane dies entirely, that is always a failure: at that point the product contains no frames at all, and reporting it as complete would hand the layer above a broken picture.
Failure reports only the error callback and does not send the complete callback, so the layer above does not take a failure for a success.
At this point the frames have become packets. How packets become a file — opening the stream, zeroing the base, interleaved writing — belongs to the muxing article.
The avox engine is open source on GitHub: github.com/xxxzhou/avox. The assembly, queues and encoder configuration of this recording chain are all in there.