Optimize boot times

This document provides partner guidance for improving boot times for specific Android devices. Boot time is an important component of system performance as users must wait for boot to complete before they can use the device. For devices such as cars where cold boot-up happens more frequently, having a quick boot time is critical (no one likes waiting for dozens of seconds just to input a navigation destination).

Android 8.0 allows for reduced boot times by supporting several improvements across a range of components. The following table summarizes these performance improvements (as measured on a Google Pixel and Pixel XL devices).

Optimize bootloader

To optimize bootloader for improved boot times:

• For logging:
  • Disable log writing to UART as it can take a long time with lots of logging. (On the Google Pixel devices, we found it slows the bootloader 1.5s).
  • Log only error situations and consider storing other information to memory with a separate mechanism to retrieve.
• For kernel decompression, considering using LZ4 for contemporary hardware instead of GZIP (example patch). Keep in mind that different kernel compression options can have different loading and decompression times, and some options may work better than others for your specific hardware.
• Check unnecessary wait times for debouncing/special mode entry and minimize them.
• Pass boot time spent in bootloader to kernel as cmdline.
• Check CPU clock and consider parallelization (requires multi-core support) for kernel loading and initializing I/O.

Optimize I/O efficiency

Improving I/O efficiency is critical to making boot time faster, and reading anything not necessary should be deferred until after boot (on a Google Pixel, about 1.2GB of data is read on boot).

Tune the filesystem

The Linux kernel read ahead kicks in when a file is read from beginning or when blocks are read sequentially, making it necessary to tune I/O scheduler parameters specifically for booting (which has a different workload characterization than normal apps).

Devices that support seamless (A/B) updates benefit greatly from filesystem tuning on first time boot (e.g. 20s on Google Pixel). An an example, we tuned the following parameters for the Google Pixel:

on late-fs
  # boot time fs tune
    # boot time fs tune
    write /sys/block/sda/queue/iostats 0
    write /sys/block/sda/queue/scheduler cfq
    write /sys/block/sda/queue/iosched/slice_idle 0
    write /sys/block/sda/queue/read_ahead_kb 2048
    write /sys/block/sda/queue/nr_requests 256
    write /sys/block/dm-0/queue/read_ahead_kb 2048
    write /sys/block/dm-1/queue/read_ahead_kb 2048

on property:sys.boot_completed=1
    # end boot time fs tune
    write /sys/block/sda/queue/read_ahead_kb 512
    ...

Miscellaneous

• Turn on the dm-verity hash prefetch size using kernel config DM_VERITY_HASH_PREFETCH_MIN_SIZE (default size is 128).
• For better file system stability and a dropped forced check that occurs on every boot, use the new ext4 generation tool by setting TARGET_USES_MKE2FS in BoardConfig.mk.

Analyze I/O

To understand I/O activities during boot, use kernel ftrace data (also used by systrace):

trace_event=block,ext4 in BOARD_KERNEL_CMDLINE

To breakdown file access for each file, make the following changes to the kernel (development kernel only; don't use in production kernels):

diff --git a/fs/open.c b/fs/open.c
index 1651f35..a808093 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -981,6 +981,25 @@
 }
 EXPORT_SYMBOL(file_open_root);
 
+static void _trace_do_sys_open(struct file *filp, int flags, int mode, long fd)
+{
+       char *buf;
+       char *fname;
+
+       buf = kzalloc(PAGE_SIZE, GFP_KERNEL);
+       if (!buf)
+               return;
+       fname = d_path(&filp-<f_path, buf, PAGE_SIZE);
+
+       if (IS_ERR(fname))
+               goto out;
+
+       trace_printk("%s: open(\"%s\", %d, %d) fd = %ld, inode = %ld\n",
+                     current-<comm, fname, flags, mode, fd, filp-<f_inode-<i_ino);
+out:
+       kfree(buf);
+}
+
long do_sys_open(int dfd, const char __user *filename, int flags, umode_t mode)
 {
 	struct open_flags op;
@@ -1003,6 +1022,7 @@
 		} else {
 			fsnotify_open(f);
 			fd_install(fd, f);
+			_trace_do_sys_open(f, flags, mode, fd);

Use the following scripts to help with analyzing boot performance.

• system/extras/boottime_tools/bootanalyze/bootanalyze.py Measures boot time with a breakdown of important steps in the boot process.
• system/extras/boottime_tools/io_analysis/check_file_read.py boot_trace Provides access information per each file.
• system/extras/boottime_tools/io_analysis/check_io_trace_all.py boot_trace Gives system-level breakdown.

Optimize init.*.rc

Init is the bridge from the kernel till the framework is established, and devices usually spend a few seconds in different init stages.

Run tasks in parallel

While the current Android init is more or less a single threaded process, you can still perform some tasks in parallel.

• Execute slow commands in a shell script service and join that later by waiting for specific property. Android 8.0 supports this use case with a new wait_for_property command.
• Identify slow operations in init. The system logs the init command exec/wait_for_prop or any action taking a long time (in Android 8.0, any command taking more than 50 ms). For example:

  init: Command 'wait_for_coldboot_done' action=wait_for_coldboot_done returned 0 took 585.012ms

  Reviewing this log may indicate opportunities for improvements.

• Start services and enable peripheral devices in critical path early. For example, some SOCs require starting security-related services before starting SurfaceFlinger. Review the system log when ServiceManager returns "wait for service" — this is usually a sign that a dependent service must be started first.
• Remove any unused services and commands in init.*.rc. Anything not used in early stage init should be deferred to boot completed.

Note: Property service is part of init process, so calling setproperty during boot can lead a long delay if init is busy in builtin commands.

Use scheduler tuning

Use scheduler tuning for early boot. Example from a Google Pixel:

on init
    # boottime stune
    write /dev/stune/schedtune.prefer_idle 1
    write /dev/stune/schedtune.boost 100
    on property:sys.boot_completed=1
    # reset stune
    write /dev/stune/schedtune.prefer_idle 0
    write /dev/stune/schedtune.boost 0

    # or just disable EAS during boot
    on init
    write /sys/kernel/debug/sched_features NO_ENERGY_AWARE
    on property:sys.boot_completed=1
    write /sys/kernel/debug/sched_features ENERGY_AWARE

Some services may need a priority boost during boot. Example:

init.zygote64.rc:
service zygote /system/bin/app_process64 -Xzygote /system/bin --zygote --start-system-server
    class main
    priority -20
    user root
...

Start zygote early

Devices with file-based encryption can start zygote earlier at the zygote-start trigger (by default, zygote is launched at class main, which is much later than zygote-start). When doing this, make sure to allow zygote to run in all CPUs (as the wrong cpuset setting may force zygote to run in specific CPUs).

Disable power saving

During device booting, power saving setting for components like UFS and/or CPU governor can be disabled.

Caution: Power saving should be enabled in charger mode for efficiency.

on init
    # Disable UFS powersaving
    write /sys/devices/soc/${ro.boot.bootdevice}/clkscale_enable 0
    write /sys/devices/soc/${ro.boot.bootdevice}/clkgate_enable 0
    write /sys/devices/soc/${ro.boot.bootdevice}/hibern8_on_idle_enable 0
    write /sys/module/lpm_levels/parameters/sleep_disabled Y
on property:sys.boot_completed=1
    # Enable UFS powersaving
    write /sys/devices/soc/${ro.boot.bootdevice}/clkscale_enable 1
    write /sys/devices/soc/${ro.boot.bootdevice}/clkgate_enable 1
    write /sys/devices/soc/${ro.boot.bootdevice}/hibern8_on_idle_enable 1
    write /sys/module/lpm_levels/parameters/sleep_disabled N
on charger
    # Enable UFS powersaving
    write /sys/devices/soc/${ro.boot.bootdevice}/clkscale_enable 1
    write /sys/devices/soc/${ro.boot.bootdevice}/clkgate_enable 1
    write /sys/devices/soc/${ro.boot.bootdevice}/hibern8_on_idle_enable 1
    write /sys/class/typec/port0/port_type sink
    write /sys/module/lpm_levels/parameters/sleep_disabled N

Defer noncritical initialization

Noncritical initialization such as ZRAM can be deferred to boot_complete.

on property:sys.boot_completed=1
   # Enable ZRAM on boot_complete
   swapon_all /vendor/etc/fstab.${ro.hardware}

Optimize boot animation

Use the following tips to optimize the boot animation.

Configure early start

Android 8.0 enables starting boot animation early, before mounting userdata partition. However, even when using the new ext4 tool chain in Android 8.0, fsck is still triggered periodically due to safety reasons, causing a delay in starting the bootanimation service.

To make bootanimation start early, split the fstab mount into two phases:

• In the early phase, mount only the partitions (such as system/ and vendor/) that don't require run checks, then start boot animation services and its dependencies (such as servicemanager and surfaceflinger).
• In the second phase, mount partitions (such as data/) that do require run checks.

Boot animation will be started much faster (and in constant time) regardless of fsck.

Finish clean

After receiving the exit signal, bootanimation plays the last part, the length of which can slow boot time. A system that boots quickly has no need for lengthy animations which could effectively hide any improvements made. We recommend making both the repeating loop and finale short.

Optimize SELinux

Use the following tips to optimize SELinux for improved boot times.

• Use clean regular expressions (regex). Poorly-formed regex can lead to a lot of overhead when matching SELinux policy for sys/devices in file_contexts. For example, the regex /sys/devices/.*abc.*(/.*)? mistakenly forces a scan of all /sys/devices subdirectories that contain "abc", enabling matches for both /sys/devices/abc and /sys/devices/xyz/abc. Improving this regex to /sys/devices/[^/]*abc[^/]*(/.*)? will enable a match only for /sys/devices/abc.
• Move labels to genfscon. This existing SELinux feature passes file-matching prefixes into the kernel in the SELinux binary, where the kernel applies them to kernel-generated filesystems. This also helps fix mislabeled kernel-created files, preventing race conditions that can occur between userspace processes attempting to access these files before relabeling occurs.

Tools and methods

Use the following tools to help you collect data for optimization targets.

Bootchart

Bootchart provides CPU and I/O load breakdown of all processes for the whole system. It doesn't require rebuilding system image and can be used as a quick sanity check before diving into systrace.

To enable bootchart:

adb shell 'touch /data/bootchart/enabled'
adb reboot

After boot up, fetch boot chart:

$ANDROID_BUILD_TOP/system/core/init/grab-bootchart.sh

When finished, delete /data/bootchart/enabled to prevent collecting the data every time.

1. Run the following commands:

         sudo apt install python-is-python3
         cd ~/Documents
         git clone https://github.com/xrmx/bootchart.git
         cd bootchart/pybootchartgui
         mv main.py.in main.py
       

2. Update $ANDROID_BUILD_TOP/system/core/init/grab-bootchart.sh to point to the local copy of pybootchartgui (located at ~/Documents/bootchart/pybootchartgui.py)

Systrace

Systrace allows collecting both kernel and Android traces during boot up. Visualization of systrace can help in analyzing specific problem during the boot-up. (However, to check the average number or accumulated number during the entire boot, it is easier to look into kernel trace directly).

To enable systrace during boot-up:

• In frameworks/native/cmds/atrace/atrace.rc, change:

    write /sys/kernel/debug/tracing/tracing_on 0
    write /sys/kernel/tracing/tracing_on 0

  To:

    #    write /sys/kernel/debug/tracing/tracing_on 0
    #    write /sys/kernel/tracing/tracing_on 0

This enables tracing (which is disabled by default).

• In the device.mk file, add the following line:

  PRODUCT_PROPERTY_OVERRIDES +=    debug.atrace.tags.enableflags=802922
  PRODUCT_PROPERTY_OVERRIDES +=    persist.traced.enable=0

• In the device BoardConfig.mk file, add the following:

  BOARD_KERNEL_CMDLINE := ... trace_buf_size=64M trace_event=sched_wakeup,sched_switch,sched_blocked_reason,sched_cpu_hotplug

For detailed I/O analysis, also add block and ext4 and f2fs.

• In the device-specific init.rc file, add the following:

  on property:sys.boot_completed=1          // This stops tracing on boot complete
  write /d/tracing/tracing_on 0
  write /d/tracing/events/ext4/enable 0
  write /d/tracing/events/f2fs/enable 0
  write /d/tracing/events/block/enable 0

• After boot up, fetch trace:

  adb root && adb shell atrace --async_stop -z -c -o /data/local/tmp/boot_trace
  adb pull /data/local/tmp/boot_trace
  $ANDROID_BUILD_TOP/external/chromium-trace/systrace.py --from-file=boot_trace