Native libraries part I - Common pitfalls

by Rolf Smit | 01 November 2016 |
AndroidNative-codeABINDK

Native libraries on Android are great, they allow developers to include native C/C++ code into an app. Native code is in general faster than Java code because the code is compiled directly to CPU instructions (instead of Java byte-code). This seems great, why should we even use Java if native code is faster? Well one of the reasons might be that not every CPU has the same architecture and therefore the same instruction set. This basically means you need to compile and package the native code for each CPU architecture independently, most likely increasing your APK size allot. This isn't always a major problem, but you might encounter more serious problems like the famous UnsatisfiedLinkError exception while trying to load the native libraries (binaries) from your Java code. In this blog post we will explore which CPU architectures Android supports, how Android picks the correct native libraries and how to avoid the famous UnsatisfiedLinkError by covering some common pitfalls.

CPU architectures

Nowadays there are quite some CPU architectures around, but for the Android system is boils down to a set of just three architectures: ARM, x86 and MIPS. Except that this isn't the complete picture. Almost every CPU architecture comes in two different families: 32-bit and 64-bit. Take for example the ARM CPU architecture, the 64-bit variant offers a 64-bit instruction set aside of the 32-bit instruction set. This means 32-bit ARM libraries are compatible with 64-bit ARM processors. However the opposite is not true, you can't run 64-bit ARM libraries on a 32-bit CPU (at least not by default, although it might be possible using some kind of visualization). On Android all 32-bit architectures are compatible with their 64-bit CPU counterparts. This means you don't need to include 64-bit binaries although this also means you don't take advantage of the 64-bit CPU (which is, for example faster when working with 64-bit numbers).

Android devices support different architectures which in turn support different instruction sets (e.g. 32-bit or 64-bit or a combination). In Android these combinations are known as ABI's (Application Binary Interface). Android currently supports 7 different ABI's (source):

ARM

  • armeabi (32-bit)
  • armeabi-v7a (32-bit extended instruction set)
  • arm64-v8a (64-bit)

x86

  • x86 (32-bit)
  • x86_64 (64-bit)

mips

  • mips (32-bit)
  • mips64 (64-bit)

So if you want the best possible performance on every Android device "out in the wild" you would need to compile your native sources for 7 different Application Binary Interfaces (ABI's).

Dive deeper: The ARMv8-A specification adds the 64-bit ARM instruction set, known as AArch-64 or A64. This instruction set is according to the specification optional. Take for example the Cortex-A32 CPU which supports the ARMv8-A specification but doesn't include the 64-bit instruction set. However on Android the arm64-v8a ABI requires the AArch-64 instruction set, so any ARMv8-A CPU without the AArch-64 instruction set is not compatible with Androids arm64-v8a ABI, but it might be compatible with the armeabi-v7a or armeabi ABI because the ARMv8-A architecture is provides compatibility with the ARMv7-A and old 32-bit ARM instruction set.

Multiple ABIs, one device

When Android installs an APK file (the PackageManager does this) it chooses which ABI it would like to use.

Yep, a device can support more than one ABI!

Most older Android devices only support a single ABI (armeabi or x86), but modern devices often support more than one ABI. Lots of ARM based devices support both the armeabi and armeabi-v7a ABI. Devices with 64-bit processors always support both the 32-bit ABI and 64-bit ABI. You can even encounter devices with support for three different ABI's: armeabi, armeabi-v7a and arm64-v8a (most 64-bit ARM devices).

You can obtain information about the supported ABI's using the following ADB command:

> adb shell getprop


[ro.product.cpu.abi]: [arm64-v8a]
[ro.product.cpu.abilist]: [arm64-v8a,armeabi-v7a,armeabi]
[ro.product.cpu.abilist32]: [armeabi-v7a,armeabi]
[ro.product.cpu.abilist64]: [arm64-v8a]

In code you can obtain information about the supported ABI's quite easily using the build constants:

Build.CPU_ABI //Deprecated
Build.CPU_ABI2 //Deprecated
Build.SUPPORTED_ABIS //Since API level 21
Build.SUPPORTED_32_BIT_ABIS //Since API level 21
Build.SUPPORTED_64_BIT_ABIS //Since API level 21

If you need to support older Android versions you might want to use these convenient little methods instead:

public static String[] getSupported32BitAbis(){
    if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP){
        return Build.SUPPORTED_32_BIT_ABIS;
    } else {
        //Android version below API 21, getSupportedAbis returns what we need.
        return getSupportedAbis();
    }
}

public static String[] getSupported64BitAbis(){
    if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP){
        return Build.SUPPORTED_64_BIT_ABIS;
    } else {
        //64-bit support has been added in Android 5.
        return new String[0];
    }
}

public static String[] getSupportedAbis(){
    if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP){
        return Build.SUPPORTED_ABIS;
    } else {
        //CPU_ABI2 can be empty
        if(Build.CPU_ABI2.isEmpty()){
            return new String[]{Build.CPU_ABI};
        } else {
            return new String[]{Build.CPU_ABI, Build.CPU_ABI2};
        }
    }
}

How Android chooses the ABI at install time

An APK file can contain native binaries for more than one ABI (using different folders), so how does Android choose which native binary it should use? The process is actually quite simple, Android starts by searching for the most preferred ABI (the first in the list returned by Build.SUPPORTED_ABIS) and so it continues until it finds a matching ABI folder in the APK. If Android fails to find such ABI, the installation process will fail. If not it copies all native binaries inside the folder to the installation directory. Which directory this is depends on your device setup, but in general:

  • Internal memory: data/app/[package]/libs/[abi]
  • SD card: mnt/asec/[package]/libs/[abi]

When things go wrong

Example 1 - its your fault

Imagine you're building a simple app which uses two native binaries: A and B and you only compile those binaries to support just one ABI, lets say armeabi. A setup like this works fine (although it doesn't support all Android devices available). But things go horribly wrong if you add support for a second ABI, lets say armeabi-v7a, but forget to include both libraries A and B in the armeabi-v7a folder.

The above setup results in the following directory structure inside the APK file (notice the missing binary inside the armeabi-v7a folder):

libs/
    armeabi/
            libA.so
            libB.so
    armeabi-v7a/
            libA.so

If you compile above setup and install the resulting APK on a device with support for two ABI's: armeabi-v7a and armeabi your app will crash when running it, throwing an UnsatisfiedLinkError for library B.

How did this happen? Although the device supports the armeabi ABI (which is "binary-complete") the the installer prefers armeabi-v7a native binaries. Because the installer did find the armeabi-v7a folder inside your APK it assumed the folder contained every native library needed for the app to run.

How do I fix this? Obtain a binary of library B compiled for the armeabi-v7a ABI and add it to your project so that the armeabi-v7a folder contains the same binaries as the armeabi folder.

Binary-complete: This is how I like to call an ABI which is fully present inside the APK, or in other words an ABI which doesn't miss any binary.

Example 2 - its still your fault

Most Android apps have at least some dependencies, a dependency might be a simple .jar file but can also be an .aar file. You don't need to worry about simple .jar dependencies (because those won't have their own native binaries) but if you start using .aar files things can become quite tricky. Take for example the library SQLCipher (SQLCipher for Android) this library uses native binaries which are packed into the .aar file. The current version (3.5.4) has support for three different ABIs armeabi, armeabi-v7a and x86.

Lets take the app from example 1 (but with the missing binary) and add the SQLCipher dependency to it. The compiled APK file now looks like this:

libs/
    armeabi/
            libA.so
            libB.so
            libsqlcipher.so 
    armeabi-v7a/
            libA.so
            libB.so
            libsqlcipher.so 
    x86/
            libsqlcipher.so 

If you install and run the resulting APK on a device supporting armeabi, armeabi-v7a or arm64-v8a or any combination those ARM based ABI's everything goes fine. But where previously your app would fail to install (on x86 devices) it now suddenly installs! But it crashes as soon as you run it.

How did this happen? Obviously this happened because the installer came across the x86 folder and assumed it contained every native library needed. But how did the x86 folder end up in your APK file? Well that has to do with the way the build process works, but I won't go into detail on this (more info).

How do I fix this? Obtaining x86 binaries for the missing binaries A and B would be the best solution, just like we did in example 1 for the missing binary inside the armeabi-v7a folder. But this isn't always possible, especially when you use pre-compiled libraries. In those cases you will need to exclude the x86 binaries from the resulting APK. This consequence of excluding the x86 binaries is that your app won't support x86 devices anymore (but it didn't anyway). If you exclude the x86 libraries from the APK the installer will fail to install the app, which is (at least in my opinion) a better user-experience than having a run-time crash.

You can filter which ABI's you want to include in the APK by adding the abiFilters option which is part of the ndk closure in your modules build.gradle file:

android {
    ndk {
        //Package only armeabi and armeabi-v7a libraries into the apk 
        abiFilters "armeabi", "armeabi-v7a"
    }
}

Depending on the version of the Android Gradle tools plugin you're using, you might also need to add the following statement inside the gralde.properties file (this doesn't seem to be needed in version 2.2.0):

android.useDeprecatedNdk=true

You can also manually exclude specific files from the APK using the Gralde exclude statement inside the packagingOptions closure:

android {
    packagingOptions {
         exclude 'lib/x86/libsqlcipher.so'
    }
}

Conclusion

Android developers are too often unaware of the native libraries provided by dependencies. Especially those who only use real devices to test on. Because these devices are often ARM devices the developers won't notice a missing x86 binary.

Here are some best-practices and tips to finish this blog post with:

  • Tip 1: Use the Native Libs Monitor app to check which ABI your app uses and which binaries are included;
  • Tip 2: Use the Android Studio 2.2 "Analyze APK" feature, you can find this feature under "build > Analyze APK...", this feature allows you to look directly inside the APK file;
  • Best practice: Get to know the native binaries used by your app and only include the ABI's into the final APK which are "binary-complete" (that's what I like to call it!) using the abiFilters option.

In part two of this blog post I will dive deeper into the install process and what can go wrong if you move your app to the sd-card, we will also explore some solutions to bugs in the PackageManager install process and how to fix them using the ReLinker library.

Happy coding!

Note I: I probably made some typos or I even included wrong information in this blog post (I'm not an ARM specification expert), I really appropriate feedback and corrections!

Note II: In some ultra-rare cases it's possible that a specific ABI has more binaries than some other ABI (most likely due to specific native dependencies which are only need for that specific ABI). This is however very unusual and you will probably never encounter a situation like this.