The device sets the audio's rhythm and progress
Video has a frame rate; the player computes from the PTS when each frame should appear, so on the rendering leg it is its own metronome. Audio has no such thing. Samples are continuous, how long a piece of data plays is decided by the sample rate and the byte count, and when it comes out of the speaker is decided by the device. The player's only job on this chain is to hand the data over on time.
The unit of handover is fixed length. A continuous stream is cut into pieces whose length is one frame (a frame in time, not the picture frame of video); the byte count of one piece is derived from sample rate, sample format and channel count together. This length is the unit of measure for every buffer later on — how large the device buffer must be, how much is still queued, all use it as the ruler.
For video, the player knows which frame it is on; for audio that number is in the device's hands, and the device will only tell you how much is still queued and unplayed.
Two threads, two levels of queue
The audio track carries its own two queue levels and two threads. The packet queue comes first, where the packets produced by the read thread land per track; the audio decode task pulls packets from the packet queue, decodes them, and writes frames into the frame queue; the audio render task pulls frames from the frame queue and sends them into the device. Both threads inherit the same thread base class, and how they start and stop is contained in that one base. The decode thread starts first; only after the decoder settles the output format can the render thread be created — you have to know what format the device will open with first.
The audio frame queue is deeper than the video one. On the video side the worry is that a long queue lets GPU resources be reclaimed first; on the audio side the fear is a gap — the ear notices a break in sound immediately, so it is better to use a little more memory.
When the packet queue fills, the pressure is pushed back to the read thread; audio and video share the same mechanism at this layer.
Two layers of audio rendering: AudioRender and AudioOutput
In the code, audio rendering is split into an upper and a lower layer. The upper layer is AudioRender, held directly by the layer above: the processing chain before the device (audio pre/post-processing such as echo cancellation), tapping audio off to the side, and loudness equalization all live here, and the result is dispatched downward. The lower layer is AudioOutput, where device-specific things become pure virtual methods — the three lifecycle hooks onInit / onRender / onClose, plus the queue and clock controls empty / getQueueMS / full / pause / flush / speed.
Four platform implementations fill these methods in: the Windows one goes through WASAPI, the Android one through AudioTrack on the Java side, the Apple one through CoreAudio's AudioUnit, plus a PulseAudio one for Linux. Which one is chosen is settled at compile time: when the factory asks for the default audio output it returns a backend type per platform; the backends register into the registry at startup and the factory fetches by type.
The point of this layer is that the layer above does not need to know which kind of device is taking the data below.
Handing over data: push yourself or wait for the device to pull
Of the four implementations, three push from a thread of their own.
The Windows leg first asks the device for a buffer much wider than the default — several times the length of one frame, since the system default is too narrow. After that every frame follows the same routine: ask how many samples are still queued in the buffer, get the writable span, copy the data in, then release it. If the device is changed (the user plugs in headphones, changes the default output), this leg gets a notification and rebuilds the device from scratch.
Android hands data to AudioTrack on the Java side, with the buffer loosened a bit beyond the system recommendation, written in streaming mode. The write call itself blocks, so in this implementation the “buffer full” check is always false: when it cannot fit, the thread simply stops inside the write call.
The Linux leg is PulseAudio; the buffer is also a few frame lengths, and writes are pushed through a relative-positioning interface.
The Apple leg is the reverse: it does not push. It registers a render callback with AudioUnit and the device pulls when the time comes. The callback runs on a real-time thread and cannot wait on a lock, so this leg builds a ring buffer of its own first (capacity a power of two, modulo by mask); the render thread writes into the ring, the callback takes from it, and fills with zeros when there is not enough. Historically the callback used a mutex, and failing to take the lock meant whole-block silence — 2.35 percent of playback time, and the direct source of the choppy sound back then; later it was changed so the callback only touches atomic indices.
Both approaches reduce to the same set of actions at the abstraction layer: “can I fit more right now” and “how much is still queued”. The three pushing legs treat thread blocking as backpressure; the pulling one treats the ring buffer as backpressure. The upper layer sees the same thing.
Device selection and failure handling
Two things remain at the device layer, and the Windows implementation shows them most clearly.
One is which device to pick. This implementation keeps no device list; it asks for the system's default playback endpoint. It opens in shared mode, not exclusive — exclusive mode requires the format to match the device exactly, and cannot open once the system changes its configuration.
The other is how to come back after the device fails. A failed data write or a failed watermark query is treated as device failure: mark a rebuild, fall back and retry, with a backoff between retries so it does not spin once every device has been unplugged; a rebuild while paused stays paused after it is done and does not start playing on its own. Of the device notifications received, only “the default playback device changed” is accepted; plug and unplug at the port level is covered by the failure path.
There is a lesson here learned the hard way: draining the buffer is an audible pop. The watermark for “can I fit more” used to be two audio segments' length, and the render thread side jitters by tens of milliseconds, which eats the margin, so the watermark was raised to no less than half the device buffer. This implementation also has no native “clear” action; clearing the buffer means stopping and starting again.
Format, resampling and volume before the device
The format the device wants is usually not the format in the stream. On Windows, opening the device gives you the system mix format; on Apple it is the other way round and you query the device's real stream format; the Android leg always reads 32-bit integer bit patterns as floats — so the engine side has to change 32-bit integer to float first, and without that the whole track is noise. When the two do not match, resampling happens once on the chain: sample rate, sample format and channel count settled in one go; when there are more channels than the device, they are downmixed as well, with the mix level taken from a preset parameter.
Android has one more fallback: if a multichannel track cannot be created, drop to two channels. Otherwise nothing consumes the track, the frame queue fills up and the whole playback pipeline jams.
The decode side has to cooperate too: planar formats are converted to interleaved before going up, because both the device layer and the speed plugin take only interleaved.
Volume lands on this layer too. Windows and PulseAudio multiply per sample in software; iOS hands it to AudioUnit's output volume parameter, Android to AudioTrack's stereo volume. The player side also has loudness equalization, measuring loudness per BS.1770 and adjusting gain with slow convergence.
Speed change is done before handing data to the device
The device itself does not understand playback speed: speed is an empty implementation in all four backends. So the speed change happens before the data is handed over, and how depends on whether a plugin is present. With a speed plugin it goes through that, changing speed without changing pitch; without one it degrades to changing the sample rate — divide the sample rate by the speed factor and the same data plays back faster, pitch rising with it.
Above 4x, below 0.2x, or with I-frame-only mode on, the audio path goes silent: frames still leave the queue and the clock still advances, they just are not rendered.
Where playback is now: derived backwards from queue backlog
The device does not report “which millisecond is playing”. The engine's algorithm is short: record the PTS of every frame handed over, ask the device how many milliseconds are still queued, and subtracting the two gives the current playback position.
“How much is still queued” has a different algorithm on each of the four legs: Windows counts samples in the buffer not yet played, Android counts frames waiting to play, Apple uses the distance between the ring buffer's read and write pointers, PulseAudio subtracts played bytes from written bytes. Different algorithms, the same quantity: time not yet played out.
This number is the audio track's clock. By default it is also the master clock, and the video side adjusts its render interval against it; how master and slave are decided, and which side is corrected when the drift grows, is the next article's subject.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, and the audio chain, the device layer and the four platform implementations are all in there.