Echo, noise and gain each get their own algorithm
Screen recording and audio recording started with just two stages, capture and encoding, and listening to the result right away sounded wrong: there was echo, there was ambient noise, and the volume kept going up and down. The three things each correspond to an algorithm. Echo is the sound from the speakers going around and being picked up by the microphone again; removing it relies on acoustic echo cancellation (AEC). Ambient noise goes to noise suppression (NS). The volume swinging up and down is the job of automatic gain control (AGC). The three together are called 3A, each an independent switch, and you can turn on only one or two of them.
WebRTC's processing has been iterating along with its call scenarios; rewriting it yourself is not realistic. Only two things need attention: the data fed to it must first be normalized into the shape it wants; and the processing chain must be placed at the right position in the recording chain.
Normalize the data before it enters processing
The PCM captured by devices comes in every shape; before processing it is normalized by a few rules.
Sample rate is the first item. 44100 — the value devices give most often — is exactly what APM handles least smoothly, so 44100 is rewritten to 48000 outright; 32000 and 48000 are fine as they are. Format is the second: processing works in 16-bit integers, so anything that is not S16 is converted to S16. Channels are the third: more than two channels is folded down to two. All three are rewrite rules, not rejection rules — whatever comes in is first normalized into the shape processing wants.
Planar data needs no separate worry: the resampling step goes through ffmpeg's swr, and planar formats become interleaved at that conversion step. What to watch is the pointers passed to swr_convert — planar input and output have to point at arrays of planes, two pointers for stereo; point them at the wrong place and the sound comes out broken.
The last item is length. APM processes in fixed durations, taking 10 ms at a time. A frame from a device can be any length, so one more layer accumulates PCM of arbitrary length and releases it one full segment at a time.
Resampling and segmentation sink into the base class; the algorithm goes in the subclass
Normalization and segmentation are things every 3A implementation has to do, independent of whose algorithm is used, so both sink into the base class: the base class is also a reshaper, and at initialization it builds the resampler from the input and output formats, works out how many bytes 10 ms corresponds to, and allocates a 10 ms buffer. Normalization rewrites not just the format used during processing — the description itself is updated and handed back to the caller: what downstream receives is the processing chain's output format, and configuring rendering by it will not go wrong.
The work of actually calling APM is in a subclass, which does only two things: configure APM at initialization; and when the buffer fills a segment, feed that segment in. After this layering, switching to another 3A implementation means replacing only the subclass; normalization, resampling and segmentation stay untouched. The subclass is not constructed directly either; it goes through a factory that builds implementations by name, the registered name being webrtc, and the business side just asks for a "processing chain".
How big 10 ms is can be worked out with a quick calculation: at 48000 Hz, stereo, 16-bit integers, 10 ms is 480 sample points, 480 entries per channel, two bytes each — 1920 bytes. The buffer is allocated at that size; no matter how long the data from the device is, one segment is released once that much has accumulated.
Configuring the three: echo cancellation, noise suppression, automatic gain
At initialization the subclass first runs the normalization above, then fills in three items in the configuration: echo cancellation on, mobile mode off — that set is for lowering the configuration on mobile; noise suppression on, level set high; automatic gain uses the new-interface path. All three switches are on by default. When filled in, an APM instance is built with the built-in processing constructor, then initialized with the input-stream and output-stream configurations.
Each time a segment fills, ProcessStream is called: input is the accumulated segment, output is written to another buffer, and when processing is done the frames are dispatched in the callback. All three items are switch plus level with no finer knobs; the algorithm parameters are not tuned in detail — default values first, polishing the effect is a later matter.
Attached at two places: the audio source and audio rendering
There are two places where 3A has to attach, and they are clearly separated by role. Position matters too: both are before encoding and after decoding — 3A feeds on waveforms, and once encoding is done the data is a packet in the compressed domain, leaving the algorithm no place to bite; so the capture side sits before data is handed to the encoder, and the playback side sits before data is handed to the device. These two are the only positions left on the chain where the data is still complete PCM. The audio source is the capture side — screen recording, audio recording and the far end of a call all come through here — and what matters most there is noise suppression and echo cancellation; audio rendering is the playback side, where what matters most is automatic gain.
The audio source approach: it does nothing until the first frame arrives, then builds the processing chain according to the format the device actually gives, and once built dispatches an audio description once (sample rate, channels and so on, from which downstream configures rendering); after that every frame goes to the processing chain first, and the completion callback dispatches the frame outward. On branches where the chain is not built or processing is off, frames pass straight through.
Audio rendering is the same: when the description is set, the processing chain is built; data to be sent does not go straight down to the device but goes through processing first, comes back from the callback and is then handed to the platform device. The difference between the two places is only which items are configured; the skeleton is the same.
The switches are also managed separately in the two places. The rendering side has a public switch: turn it on for calls, off for watching films — once off, data passes straight through to the device without a single processing pass; the film-watching path does not need echo cancellation, so the data goes through one fewer hand. When processing is off, the audio source leg likewise takes the pass-through branch and dispatches frames as they are, only without building a chain and without occupying that 10 ms buffer.
At this point the 3A on the recording chain is wired up. The next article moves to the pull side: WebRTC does not carry H265 or AAC decoding itself — how to run your own decoders through its factory.
The avox engine is open source on GitHub: github.com/xxxzhou/avox, with the base class, the subclass and both attachment points inside.