Device enumeration and the format whitelist on the camera side
At init the camera manager creates a system camera manager, gets the device id list, and creates a source object for each id. Above it sits the device-manager layer; listing devices and taking by index were covered in the frame-source article, so here I only say who fills the Android slot.
Creating a source object reads the device's capability info once and walks the available stream configurations. Two filters apply at this step: skip input-direction configurations (output only), and keep only one format — the 8-bit planar YUV option. Every other format (the various packed RGB ones, say) is dropped outright. So only that one format can enter the candidate table, though whether it is actually NV12 or another planar layout on a given device depends on what the device reports.
Which option to use is decided only at open: given the resolution and frame rate requested from above, pick the closest one in the candidate table. The pick is a score — first how far off the aspect ratio is, then how far off the long side is, then a penalty term for “prefer larger”, and finally the frame-rate difference. If the layer above gives none, there is a default: one set for front and one for back.
Output surface: straight to a texture, bypassing the frame-grab API
Camera frames can land in memory (taking one image at a time through the frame-grab API) or directly in a texture. Here the choice is a texture: before starting the stream, build a chain — start an EGL context, create a SurfaceTexture and give it an external texture, then convert it into a native window; that window is the output surface handed to the camera.
The reason for choosing a texture lies in the two places downstream: the encoding side wants an input surface that accepts textures, and the rendering side already handles external textures. Going through memory would add one more upload later.
Sessions are built plainly: use that native window to create a session output and a camera output target, add a template, then issue repeating requests for a continuous stream of frames. Changing the template or the output surface requires rebuilding the session.
How a frame reaches the layer above: two threads dividing the work
When the camera has a new frame, the system calls back on the Java side: the in-house wrapper object receives the notification and passes its native pointer back to C++. The C++ callback is short and does two things — set a “new frame” flag and wake the waiting thread.
The actual frame taking happens on the source object's own task thread: wait for the flag (with a timeout, to avoid waiting forever), and once woken, update the texture with the frame and assemble it into a frame object for the layer above.
This division produces two behaviors: the flag is a single boolean, so multiple notifications arriving during processing merge into one — the frames in between are simply gone; the queue itself is provided by the system texture object, the consumer takes only the latest image each time, and when the producer runs faster the old ones are dropped. Dropped frames are not reported at all; only the wait-timeout path writes a log line.
Frame form: the camera hands out texture frames, the format marked as the 8-bit planar option, and the buffer pointer in the frame is null — the data lives in that external texture and is fetched through the context.
After the device is gone
Disconnect and error go through two callbacks. Disconnect simply reports device loss; errors are classified by the system error code — in use, disabled, or something else — each mapped to an error type and reported, with a human-readable note attached. Session close and activation each have a callback too, used to maintain the open flag.
There is no automatic reconnect here: a disconnect is only reported, and getting the device back requires an explicit reopen by the layer above. A denied permission is not a separate class either; it simply shows up as an open failure.
Audio capture lands on the Java-side recorder class
Audio does not go through the pure-C recording API; it lands on the Java-side recorder class. The C++ side grabs this class and its few methods by name: construct, compute the minimum buffer, start and stop, read, release, get the session id.
The parameters are a few defaults: 16-bit signed, with one default set for sample rate and channel count. Opening the stream picks the mono or stereo channel mask by channel count, computes the minimum buffer once with a static method, and allocates twice that. If the computed value is invalid, it falls back to computing from the frame length.
Data returns to C++ through a preallocated byte array held by a global reference; the capture thread does one blocking read, copies the bytes read into a buffer, and assembles them into an audio frame for delivery. There is no application-level queue; the read is synchronous, one frame per delivery.
A read error (a negative return) reports a device error and exits the loop. Also, the “is it open” test uses whether the session id is greater than zero; on models where the default audio session id can be zero, this gives the wrong answer.
Neither path reconnects automatically; recovery is a reopen by the layer above
Put the two paths together and there is one shared trade-off: neither the camera nor the microphone reconnects automatically. The camera path stops after reporting device loss, the audio path exits the loop on a read failure; recovery in both cases means the layer above going through open again.
The cost of this trade-off is one more job for the layer above: subscribe to error callbacks and decide, after a device is gone, whether to reopen or switch devices. Why it went away, whether to reopen, and at what setting — those judgments are left to the layer with better context.
Compare the Windows side: screen capture skips frames rather than stopping when the window is invisible, and a camera disconnect is only reported. All three platforms handle this consistently — the capture layer is only responsible for reporting the fact that it is gone.
Both paths converge on the same entry point
The camera's texture frames and the microphone's audio blocks both end up at the same device-source entry point: the device source joins the two tracks into one, then passes it through the raw-frame-source layer to the observer above. The recording chain discussed later plugs in here, on the same template as the Windows side — the only difference between the two ends is which device is taken, and whether the output surface is self-built or host-supplied.
The avox engine is open source on GitHub: github.com/xxxzhou/avox; the camera chain and the audio-side JNI bridge are both in there.