There is only one decoder interface
There is only one video decoder base class, and it inherits AVDecoder and an observer interface. The layer above calls just three things — set the context, push config frames, feed one packet; decoded frames come out through a callback rather than being fetched by the layer above.
The base class leaves six hooks. After the decoder description is obtained, onVaild runs, and the subclass initializes its own data there; onPreDecoder configures the decoder, on the premise that config frames are ready, returning noConfig when there is no configuration; decode is the one that actually consumes a packet and returns a DecodeResult; flush is seek semantics, dropping frames held internally, which belong to the old position before the jump; onInputEnd is end-of-stream semantics, releasing the last few frames held in the reorder buffer — without releasing them they would never be displayed; onClose clears up. flush and onInputEnd look alike but want opposite things: one asks for invalidation, the other for release — earlier there was only flush, and the cluster at end of stream stayed stuck.
The callback surface is wider than "two kinds of frame" too: onVideoDesc reports once after the decoder actually opens, onPacket reports before each packet is fed, onVideoComplete reports the end, and two more carry metadata — HDR static metadata and Dolby Vision reshaping data, called back only when the value changes.
The two landing places for frames
The interface does not distinguish implementations, but frames land in two places. These are not two decode lines but two products of the same line, decided by the decoding method: CPU decoding yields a block of YUV memory, platform hardware decoding yields a texture in video memory that never returns to system memory.
The layer above writes no branch for this; it only needs to know "a frame arrived". The one place that really has to distinguish is the renderer, which picks one that can take this kind of frame based on decoder type. There are five decode types in all: cpu, vulkan, iosVT (VideoToolbox), androidMC (MediaCodec), dx11 (FFmpeg's dx11 decoding on Windows). This chapter runs cpu.
Wiring in FFmpeg's decoder
FFmpeg's decoder inherits both VideoDecoder and a base class that manages FFmpeg, and opening is the standard three steps: find the decoder, configure the context, open.
There are two entry points for finding a decoder: by codec ID with avcodec_find_decoder, by name with avcodec_find_decoder_by_name. Registration names come in three tiers: the FFmpeg leg uses native names (h264, hevc, vp9); platform hardware decoders carry an ff_ prefix plus a platform suffix (ff_h264_vulkan, ff_h264_dx11, ff_h264_vaapi); Android and iOS native decoders use descriptive names with spaces. "ff_h264 is the software decoder name" is a common misunderstanding — ff_ only grows on the hardware side. AV1 has an extra pitfall: av1 in FFmpeg is a wrapper that only carries hardware acceleration, and with no accelerated context it reports unsupported on every packet; a build with dav1d renames it to libdav1d, and a build without dav1d has to cut this shell out entirely, so the candidate table does not keep a lane that can never decode anything.
The context is routine work: allocate, parse config frames, fill pixel format and dimensions, attach the decode environment (in FFmpeg's implementation this is empty at this step; platform hardware decoding creates a D3D11 or Vulkan device here), assemble extradata, open. Two values are decided here: the decode thread count is set to 0 so FFmpeg decides by core count; "tolerate bad frames" is turned on, so corrupt frames still come out and one bad packet does not jam the whole stretch. Sending and receiving packets is two steps: send a packet, take a frame; each frame taken is called back once, and the keyframe flag is filled in here — set when the packet's frame type is 1.
Candidates are drawn from every implementation under one codec
There is no factory here of "register a decoder, create an object by name". Decoders hang on one table indexed by codec ID, with several implementations under the same codec — FFmpeg's, dx11, vulkan, platform-native each hang their own, and registering means appending "the function that makes the object" under that codec's name. The candidate table takes the full set of this table under that codec.
A candidate records only the registry index, not a copy of the object-making function: the function's type-erasure information is generated at the registration point, and for a plugin registration it lives in the plugin's dynamic library; copied elsewhere it would dangle the moment the plugin is unloaded.
How the candidate table is ordered: preference only decides who is tried first
The candidate table is ordered in four segments: the override name, the hint name for the preferred pole, the hint name for the other pole, and the rest of the registry in registration order as the fallback. The override name is for troubleshooting — name a lane and it is forced, placed first; troubleshooting wants to verify whether the named one works at all, so no vulkan alternate hangs behind it.
Hint names come from a table keyed by platform and codec, and they only decide who is tried first within the same pole; whether something is software or hardware is not judged by the hint name but by the hardware flag in the registry. The hint name on the software side is empty, so the other pole cannot be hard-coded to some name — hard-coded, choosing software would leave no fallback. The candidate table takes the registry's full set directly, so the other pole always has candidates. The vulkan family is placed last as a whole: it exists only for h264 and h265, and setting its context probes the runtime once per platform, so placing it early would make other candidates wait for nothing.
In use they are tried one by one: initFunc builds the instance, and both wiring and setting the context count toward "can this lane be used"; if it fails, the name is noted and the next is taken. The old lane is still alive while a new one is tried — only when all candidates are exhausted is the track judged dead.
The two exit paths of a lane
Once a lane is running, two situations make it exit and look for the next one.
One is failing to open. Opening is lazy — the session really opens on the first packet feed, and a failure returns openFailed, switches to the next candidate on the spot, replays the config frames and keeps decoding, without judging the video track dead.
The other is opening but producing no frames, caught by the first-frame watchdog. It has two rules: it only credits actual output — it watches for a frame actually arriving in the frame queue, not the decode return value, because feeding duplicate parameter-set packets spins and returns success, and trusting the return value would be fooled; and it only times while input keeps coming — if the packet queue is empty and the read thread is stuck on parameter probing and emits no packets, that is starvation waiting, and the timer resets. Miss this rule and at the instant before a flood of data arrives, the hardware lane gets wrongly killed.
If supply continues for a stretch and still no frame appears, the lane is judged dead and the next one is taken. If the local stream has already been read to the end by then, the playback position also has to be rewound to the start so the new lane can begin decoding from a keyframe; if all candidates are exhausted and still nothing works, a decode timeout is reported and the video track closes up.
When switching lanes, config frames come along
Switching lanes has a handover: when the new lane comes up, the config frames are copied over from the previous lane and fed again, then it goes on consuming packets. On the first selection there is no previous lane, so the config frames can only wait for IO to deliver them.
There is one more action after switching: if the new lane starts decoding mid-GOP it only produces bad frames missing references, so packets are dropped until the next keyframe; but this is done only while the first frame has not appeared yet. Switching lanes during playback with picture already showing, moving the landing point would cause a frame jump, so the position is left alone.
The decode thread's loop
Start a thread, take packets from the video track's packet queue, decode them and hand them to the video track — that is the decode task's entire responsibility. The thread shell comes from the base class, and the loop body is a pure virtual function, so swapping implementations does not touch the thread mechanism.
The loop body splits two ways by packet: a config frame is pushed to the decoder and does not enter the decode step; anything else is fed in to decode. Pushing a config frame also depends on the result: if the content is the same as what is already stored, the packet is skipped outright; if the parameter-set size changed, a flag is recorded for the cases that need the decoder rebuilt.
The first successful return is the signal that "the decoder opened": at that point the description is dispatched once, reporting pixel format, dimensions, frame rate, and render type. On receiving the completion flag it dispatches the end and exits the loop; at speeds between 2x and 4x the loop runs faster and decodes only keyframes.
When no packet is available, or the parameter sets are not complete yet, the thread blocks and waits for the next round instead of spinning; the rounds that actually decode data, by contrast, must run full speed and must not sleep. When the frame queue is full the frame-write step blocks, and backpressure works as usual, the same as the read thread jamming at the enqueue. The decode thread does not need to know whether downstream is rendering or recording.
Decode failure: video stops, audio continues
A failed decoder configuration returns a negative number, and the handling is the least conspicuous and most important place in this article: the packet queue must be released before exiting. Exit with the packet queue full and the read thread blocks forever at the enqueue, which in turn jams demuxing, starves the audio packets too, and freezes the whole pipeline together — this was actually hit with a high-bitrate source paired with a lossless audio track. The design intent is for the video path to stop while audio keeps playing.
After the queue is released, the error code is handed to the video track, which carries it upward: the player digests it first, updates the state to where it should be, then calls the decode error back up. The error itself occupies no state; it goes through a callback.
Two checks before opening: config frame count and packet format
There are two checks before the decoder is really opened; failing either reports "missing config" and does not force it open.
One is the config frame count: H.264 needs at least two (SPS, PPS), H.265 at least three (VPS, SPS, PPS). Counting recognizes only the parameter-set NAL types, skipping SEI and AUD that come mixed in — without skipping, a single SEI tucked into a stretch would fail the judgment, the parameters would fall back to the container's dimensions, and that would clash with the SPS size previously trusted.
The second is that packet format must match config frame format. Whether packets are AVCC or AnnexB is judged once when the first packet arrives, and the result decides how extradata is assembled. Some platforms' native decoders accept only one (Apple wants AVCC, Android wants AnnexB), so a conversion runs before the decoder; in that conversion even AnnexB's three-byte header is unified to four bytes, since the decoder may not recognize the former.
This layer only flattens malformed timestamps
Time base conversion is not done at this layer. Converting pts and dts to milliseconds happens on FFmpeg's IO side, treating audio, video, and subtitles alike; a B frame may have no pts, and dts stands in. The playback reference is also set upstream, taking the time of the first video keyframe, and only falling back to the first audio packet when the whole stream has no video track.
The decode layer does exactly one timestamp job: flattening. Old containers often have a cluster of packets squeezed around the same time point, with timestamps malformed beyond recognition, and sending them straight to the decoder causes trouble. The flattener changes only the pts handed to the decoder; the container's and the exported ones are left untouched. The method computes one frame's interval from the frame rate and offsets the packets of the same cluster by that interval; for a normal stream a single-packet cluster shifts by zero. The cluster at the tail is released only after input is drained, otherwise the last cluster stays stuck in hand.
Both implementation switching and seek wait for a keyframe
When the layer above changes "whether to use platform hardware decoding", it does not swap the decoder on the spot; it issues a reset command. The command thread, on receiving it, sets a reset flag on the video track, and what moves is the running decode thread: after the flag is set it does not exit immediately but waits for the next keyframe (or for the packet queue to empty) before breaking out of the loop. Both implementations accept only keyframes as an entry point; switching midway would feed a P frame to a new decoder with no reference frames at all, and waiting for an I frame gives a clean start.
seek takes another road but reaches the same conclusion. It was verified on the hardware lane: calling only the decoder's flush cannot continue — flush cannot clear the frame number and picture order count, and continuing from a non-IDR keyframe landing point sends the frame number spiraling down to zero; on the software side flush does clear them, but both lanes take the same rebuild path. So after a seek the decoder is rebuilt as a whole; the only exception is Android's MediaCodec, which has an in-place flush mode that preserves the instance and the texture chain behind it. There is one more ordering point: seek's flush does not call the decoder directly on the command thread — this function runs on the command thread, while the decode thread may be stuck in the window between "allocating the context" and "opening", and that window is empty inside, so a synchronous flush landing there is guaranteed to crash; so it only sets a flag and lets the decode thread handle it inside its loop.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, and this decode line and that candidate table are both in there.