Design for reduced latency

The Android 4.1 release introduced internal framework changes for a lower latency audio output path. There were minimal public client API or HAL API changes. This document describes the initial design, which has continued to evolve over time. Having a good understanding of this design should help device OEM and SoC vendors implement the design correctly on their particular devices and chipsets. This article is not intended for application developers.

Track creation

The client can optionally set bit AUDIO_OUTPUT_FLAG_FAST in the audio_output_flags_t parameter of AudioTrack C++ constructor or AudioTrack::set(). Currently the only clients that do so are:

The AudioTrack C++ implementation reviews the AUDIO_OUTPUT_FLAG_FAST request and may optionally deny the request at client level. If it decides to pass the request on, it does so using TRACK_FAST bit of the track_flags_t parameter of the IAudioTrack factory method IAudioFlinger::createTrack().

The AudioFlinger audio server reviews the TRACK_FAST request and may optionally deny the request at server level. It informs the client whether or not the request was accepted, via bit CBLK_FAST of the shared memory control block.

The factors that impact the decision include:

  • Presence of a fast mixer thread for this output (see below)
  • Track sample rate
  • Presence of a client thread to execute callback handlers for this track
  • Track buffer size
  • Available fast track slots (see below)

If the client's request was accepted, it is called a "fast track." Otherwise it's called a "normal track."

Mixer threads

At the time AudioFlinger creates a normal mixer thread, it decides whether or not to also create a fast mixer thread. Both the normal mixer and fast mixer are not associated with a particular track, but rather with a set of tracks. There is always a normal mixer thread. The fast mixer thread, if it exists, is subservient to the normal mixer thread and acts under its control.

Fast mixer

Features

The fast mixer thread provides these features:

  • Mixing of the normal mixer's sub-mix and up to 7 client fast tracks
  • Per track attenuation

Omitted features:

  • Per track sample rate conversion
  • Per track effects
  • Per mix effects

Period

The fast mixer runs periodically, with a recommended period of two to three milliseconds (ms), or a slightly higher period of five ms if needed for scheduling stability. This number was chosen so that, accounting for the complete buffer pipeline, the total latency is on the order of 10 ms. Smaller values are possible but may result in increased power consumption and chance of glitches depending on CPU scheduling predictability. Larger values are possible, up to 20 ms, but result in degraded total latency and so should be avoided.

Scheduling

The fast mixer runs at elevated SCHED_FIFO priority. It needs very little CPU time, but must run often and with low scheduling jitter. Jitter expresses the variation in cycle time: it is the difference between the actual cycle time versus the expected cycle time. Running too late will result in glitches due to underrun. Running too early will result in glitches due to pulling from a fast track before the track has provided data.

Blocking

Ideally the fast mixer thread never blocks, other than at HAL write(). Other occurrences of blocking within the fast mixer are considered bugs. In particular, mutexes are avoided. Instead, non-blocking algorithms (also known as lock-free algorithms) are used. See Avoiding priority inversion for more on this topic.

Relationship to other components

The fast mixer has little direct interaction with clients. In particular, it does not see binder-level operations, but it does access the client's shared memory control block.

The fast mixer receives commands from the normal mixer via a state queue.

Other than pulling track data, interaction with clients is via the normal mixer.

The fast mixer's primary sink is the audio HAL.

Normal mixer

Features

All features are enabled:

  • Up to 32 tracks
  • Per track attenuation
  • Per track sample rate conversion
  • Effects processing

Period

The period is computed to be the first integral multiple of the fast mixer period that is >= 20 ms.

Scheduling

The normal mixer runs at elevated SCHED_OTHER priority.

Blocking

The normal mixer is permitted to block, and often does so at various mutexes as well as at a blocking pipe to write its sub-mix.

Relationship to other components

The normal mixer interacts extensively with the outside world, including binder threads, audio policy manager, fast mixer thread, and client tracks.

The normal mixer's sink is a blocking pipe to the fast mixer's track 0.

Flags

AUDIO_OUTPUT_FLAG_FAST bit is a hint. There's no guarantee the request will be fulfilled.

AUDIO_OUTPUT_FLAG_FAST is a client-level concept. It does not appear in server.

TRACK_FAST is a client -> server concept.