Merge "Revert "No special treatment for vendor"" into main
diff --git a/docs/mte.md b/docs/mte.md
new file mode 100644
index 0000000..3890283
--- /dev/null
+++ b/docs/mte.md
@@ -0,0 +1,244 @@
+# Arm Memory Tagging Extension (MTE) implementation
+
+AOSP supports Arm MTE to detect invalid memory accesses. The implementation is
+spread across multiple components, both within and out of the AOSP tree. This
+document gives an overview and pointers about how the various MTE features are
+implemented.
+
+For documentation of the behavior rather than the implementation, see the
+[SAC page on MTE] instead. For MTE for apps, see the [NDK page on MTE].
+
+The relevant components are:
+
+* [LLVM Project] (out of AOSP tree)
+ * Stack tagging instrumentation pass
+ * Scudo memory allocator
+* bionic
+ * libc
+ * dynamic loader
+* Zygote
+* debuggerd
+* [NDK]
+
+## MTE enablement
+
+The way MTE is requested and enabled differs between native binaries and Java
+apps. This is necessarily so, because Java apps get forked from the Zygote,
+while native executables get inintialized by the linker.
+
+### Native binaries
+
+Both AOSP and the NDK allow you to compile C/C++ code that use MTE to detect
+memory safety issues. The [NDK legacy cmake toolchain] and the
+[NDK new cmake toolchain] both support "memtag" as an argument for
+`ANDROID_SANITIZE`. NDK make has no specific support for MTE, but the
+relevant flags can be passed directly as `CFLAGS` and `LDFLAGS`.
+
+For the OS itself, [Soong] supports "memtag_[heap|stack|globals]" as
+`SANITIZE_TARGET and as `sanitize:` attribute in Android.bp files;
+[Android make] supports the same environment variables as Soong. This passes
+the appropriate flags to the clang driver for both compile and link steps.
+
+#### Linker
+
+* For **dynamic executables** LLD has support to
+ [add appropriate dynamic sections] as defined in the [ELF standard]
+* For **static executables** and as a fallback for older devices, LLD
+ also supports [adding the Android-specific ELF note]
+
+Both of the above are controlled by the linker flag `--android-memtag-mode`
+which is [passed in by the clang driver] if
+`-fsanitize=memtag-[stack|heap|globals]` is [passed in].
+`-fsanitize=memtag` [enables all three] (even for API levels that don't
+implement the runtime for globals, which means builds from old versions
+of clang may no work with newer platform versions that support globals).
+`-fsanitize-memtag-mode` allows to choose between ASYNC and SYNC.
+
+This information can be queried using `llvm-readelf --memtag`.
+
+This information is [picked up by libc init] to decide whether to enable MTE.
+`-fsanitize-heap` controls both whether scudo tags allocations, and whether
+tag checking is enabled.
+
+#### Runtime environment (dynamic loader, libc)
+
+There are two different initialization sequences for libc, both of which end up
+calling `__libc_init_mte`.
+
+N.B. the linker has its own copy of libc, which is used when executing these
+functions. That is why we have to use `__libc_shared_globals` to communicate
+with the libc of the process we are starting.
+
+* **static executables** `__libc_init` is called from `crtbegin.c`, which calls
+ `__libc_init_mte`
+* **dynamic executables** the linker calls `__libc_init_mte`
+
+`__libc_init_mte` figures out the appropriate MTE level that is requested by
+the process, calls `prctl` to request this from the kernel, and stores data in
+`__libc_shared_globals` which gets picked up later to enable MTE in scudo.
+
+It also does work related to stack tagging and permissive mode, which will be
+detailed later.
+
+### Apps
+
+Apps can request MTE be enabled for their process via the manifest attribute
+`android:memtagMode`. This gets interpreted by Zygote, which always runs with
+`ASYNC` MTE enabled, because MTE for a process can only be disabled after
+it has been initialized (see [Native binaries](#native-binaries)), not enabled.
+
+[decideTaggingLevel] in the Zygote figures out whether to enable MTE for
+an app, and stores it in the `runtimeFlags`, which get picked up by
+[SpecializeCommon] after forking from the Zygote.
+
+## MTE implementation
+
+### Heap Tagging
+
+Heap tagging is implemented in the scudo allocator. On `malloc` and `free`,
+scudo will update the memory's tags to prevent use-after-free and buffer
+overflows.
+
+[scudo's memtag.h] contains helper functions to deal with MTE tag management,
+which are used in [combined.h] and [secondary.h].
+
+
+### Stack Tagging
+
+Stack tagging requires instrumenting function bodies. It is implemented as
+an instrumentation pass in LLVM called [AArch64StackTagging], which sets
+the tags according to the lifetime of stack objects.
+
+The instrumentation pass also supports recording stack history, consisting of:
+
+* PC
+* Frame pointer
+* Base tag
+
+This can be used to reconstruct which stack object was referred to in an
+invalid access. The logic to reconstruct this can be found in the
+[stack script].
+
+
+Stack tagging is enabled in one of two circumstances:
+* at process startup, if the main binary or any of its dependencies are
+ compiled with `memtag-stack`
+* library compiled with `memtag-stack` is `dlopen`ed later, either directly or
+ as a dependency of a `dlopen`ed library. In this case, the
+ [__pthread_internal_remap_stack_with_mte] function is used (called from
+ `memtag_stack_dlopen_callback`). Because `dlopen`
+ is handled by the linker, we have to [store a function pointer] to the
+ process's version of the function in `__libc_shared_globals`.
+
+Enabling stack MTE consists of two operations:
+* Remapping the stacks as `PROT_MTE`
+* Allocating a stack history buffer.
+
+The first operation is only necessary when the process is running with MTE
+enabled. The second operation is also necessary when the process is not running
+with MTE enabled, because the writes to the stack history buffer are
+unconditional.
+
+libc keeps track of this through two globals:
+
+* `__libc_memtag_stack`: whether stack MTE is enabled on the process, i.e.
+ whether the stack pages are mapped with PROT\_MTE. This is always false if
+ MTE is disabled for the process (i.e. `libc_globals.memtag` is false).
+* `__libc_memtag_stack_abi`: whether the process contains any code that was
+ compiled with memtag-stack. This is true even if the process does not have
+ MTE enabled.
+
+### Globals Tagging
+
+TODO(fmayer): write once submitted
+
+### Crash reporting
+
+For MTE crashes, debuggerd serializes special information into the Tombstone
+proto:
+
+* Tags around fault address
+* Scudo allocation history
+
+This is done in [tombstone\_proto.cpp]. The information is converted to a text
+proto in [tombstone\_proto\_to\_text.cpp].
+
+## Bootloader control
+
+The bootloader API allows userspace to enable MTE on devices that do not ship
+with MTE enabled by default.
+
+See [SAC bootloader support] for the API definition. In AOSP, this API is
+implemented in [system/extras/mtectrl]. mtectrl.rc handles the property
+changes and invokes mtectrl to update the misc partition to communicate
+with the bootloader.
+
+There is also an [API in Device Policy Manager] that allows the device admin
+to enable or disable MTE under certain circumstances.
+
+The device can opt in or out of these APIs by a set of system properties:
+
+* `ro.arm64.memtag.bootctl_supported`: the system property API is supported,
+ and an option is displayed in Developer Options.
+* `ro.arm64.memtag.bootctl_settings_toggle`: an option is displayed in the
+ normal settings. This requires `ro.arm64.memtag.bootctl_supported` to be
+ true. This implies `ro.arm64.memtag.bootctl_device_policy_manager`, if it
+ is not explicitely set.
+* `ro.arm64.memtag.bootctl_device_policy_manager`: the Device Policy Manager
+ API is supported.
+
+## Permissive MTE
+
+Permissive MTE refers to a mode which, instead of crashing the process on an
+MTE fault, records a tombstone but then continues execution of the process.
+An important caveat is that system calls with invalid pointers (where the
+pointer tag does not match the memory tag) still return an error code.
+
+This mode is only available for system services, not apps. It is implemented
+in the [debugger\_signal\_handler] by disabling MTE for the faulting thread.
+Optionally, the user can ask for MTE to be re-enabled after some time.
+This is achieved by arming a timer that calls [enable_mte_signal_handler]
+upon expiry.
+
+## MTE Mode Upgrade
+
+When a system service [crashes in ASYNC mode], we set an impossible signal
+as an exit code (because that signal is always gracefully handled by libc),
+and [in init] we set `BIONIC_MEMTAG_UPGRADE_SECS`, which gets handled by
+[libc startup].
+
+[SpecializeCommon]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/jni/com_android_internal_os_Zygote.cpp?q=f:frameworks%2Fbase%2Fcore%2Fjni%2Fcom_android_internal_os_Zygote.cpp%20%22%20mallopt(M_BIONIC_SET_HEAP_TAGGING_LEVEL,%22&ss=android%2Fplatform%2Fsuperproject%2Fmain
+[LLVM Project]: https://github.com/llvm/llvm-project/
+[NDK]: https://android.googlesource.com/platform/ndk/
+[NDK legacy cmake toolchain]: https://android.googlesource.com/platform/ndk/+/refs/heads/main/build/cmake/android-legacy.toolchain.cmake#490
+[NDK new cmake toolchain]: https://android.googlesource.com/platform/ndk/+/refs/heads/main/build/cmake/flags.cmake#56
+[Soong]: https://cs.android.com/android/platform/superproject/main/+/main:build/soong/cc/sanitize.go?q=sanitize.go&ss=android%2Fplatform%2Fsuperproject%2Fmain
+[decideTaggingLevel]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/com/android/internal/os/Zygote.java?q=symbol:decideTaggingLevel
+[picked up by libc init]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_static.cpp?q=symbol:__get_tagging_level%20f:bionic
+[enables all three]: https://github.com/llvm/llvm-project/blob/e732d1ce86783b1d7fe30645fcb30434109505b9/clang/include/clang/Basic/Sanitizers.def#L62
+[passed in]: https://github.com/llvm/llvm-project/blob/ff2e619dfcd77328812a42d2ba2b11c3ff96f410/clang/lib/Driver/SanitizerArgs.cpp#L719
+[passed in by the clang driver]: https://github.com/llvm/llvm-project/blob/ff2e619dfcd77328812a42d2ba2b11c3ff96f410/clang/lib/Driver/ToolChains/CommonArgs.cpp#L1595
+[adding the Android-specific ELF note]: https://github.com/llvm/llvm-project/blob/435cb0dc5eca08cdd8d9ed0d887fa1693cc2bf33/lld/ELF/Driver.cpp#L1258
+[ELF standard]: https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#6dynamic-section
+[add appropriate dynamic sections]: https://github.com/llvm/llvm-project/blob/7022498ac2f236e411e8a0f9a48669e754000a4b/lld/ELF/SyntheticSections.cpp#L1473
+[storeTags]: https://cs.android.com/android/platform/superproject/main/+/main:external/scudo/standalone/memtag.h?q=f:scudo%20f:memtag.h%20function:storeTags
+[SAC page on MTE]: https://source.android.com/docs/security/test/memory-safety/arm-mte
+[NDK page on MTE]: https://developer.android.com/ndk/guides/arm-mte
+[AArch64StackTagging]: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/AArch64/AArch64StackTagging.cpp
+[scudo's memtag.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/memtag.h
+[combined.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/combined.h
+[secondary.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/secondary.h
+[__pthread_internal_remap_stack_with_mte]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/pthread_internal.cpp?q=__pthread_internal_remap_stack_with_mte
+[stack script]: https://cs.android.com/android/platform/superproject/main/+/main:development/scripts/stack?q=stack
+[Android make]: https://cs.android.com/android/platform/superproject/main/+/main:build/make/core/config_sanitizers.mk
+[store a function pointer]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_dynamic.cpp;l=168?q=memtag_stack_dlopen_callback
+[tombstone\_proto.cpp]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/libdebuggerd/tombstone_proto.cpp?q=tombstone_proto.cpp
+[tombstone\_proto\_to\_text.cpp]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/libdebuggerd/tombstone_proto_to_text.cpp
+[SAC MTE bootloader support]: https://source.android.com/docs/security/test/memory-safety/bootloader-support
+[system/extras/mtectrl]: https://cs.android.com/android/platform/superproject/main/+/main:system/extras/mtectrl/
+[API in Device Policy Manager]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/admin/DevicePolicyManager.java?q=symbol:setMtePolicy%20f:DevicePolicyManager.java
+[debuggerd\_signal_handler]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/handler/debuggerd_handler.cpp?q=f:debuggerd_handler.cpp%20symbol:debuggerd_signal_handler
+[enable_mte_signal_handler]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_static.cpp?q=symbol:__enable_mte_signal_handler
+[in init]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/init/service.cpp?q=f:system%2Fcore%2Finit%2Fservice.cpp%20should_upgrade_mte
+[crashes in ASYNC mode]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/handler/debuggerd_handler.cpp;l=799?q=BIONIC_SIGNAL_ART_PROFILER
+[libc startup]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_static.cpp?q=BIONIC_MEMTAG_UPGRADE_SECS
diff --git a/libc/Android.bp b/libc/Android.bp
index d9b3658..5ae8c4f 100644
--- a/libc/Android.bp
+++ b/libc/Android.bp
@@ -264,6 +264,7 @@
name: "libc_init_static",
defaults: ["libc_defaults"],
srcs: [
+ "bionic/libc_init_mte.cpp",
"bionic/libc_init_static.cpp",
":elf_note_sources",
],
@@ -1489,6 +1490,7 @@
srcs: [
"arch-common/bionic/crtbegin_so.c",
"arch-common/bionic/crtbrand.S",
+ "bionic/android_mallopt.cpp",
"bionic/gwp_asan_wrappers.cpp",
"bionic/heap_tagging.cpp",
"bionic/icu.cpp",
@@ -1507,6 +1509,7 @@
filegroup {
name: "libc_sources_static",
srcs: [
+ "bionic/android_mallopt.cpp",
"bionic/gwp_asan_wrappers.cpp",
"bionic/heap_tagging.cpp",
"bionic/icu_static.cpp",
diff --git a/libc/bionic/android_mallopt.cpp b/libc/bionic/android_mallopt.cpp
new file mode 100644
index 0000000..79e4072
--- /dev/null
+++ b/libc/bionic/android_mallopt.cpp
@@ -0,0 +1,146 @@
+/*
+ * Copyright (C) 2009 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <errno.h>
+#include <stdatomic.h>
+
+#include <platform/bionic/malloc.h>
+#include <private/bionic_globals.h>
+
+#include "gwp_asan_wrappers.h"
+#include "malloc_limit.h"
+
+#if !defined(LIBC_STATIC)
+#include <stdio.h>
+
+#include <private/bionic_defs.h>
+
+#include "malloc_heapprofd.h"
+
+extern bool gZygoteChild;
+extern _Atomic bool gZygoteChildProfileable;
+
+bool WriteMallocLeakInfo(FILE* fp);
+bool GetMallocLeakInfo(android_mallopt_leak_info_t* leak_info);
+bool FreeMallocLeakInfo(android_mallopt_leak_info_t* leak_info);
+#endif
+
+// =============================================================================
+// Platform-internal mallopt variant.
+// =============================================================================
+#if !defined(LIBC_STATIC)
+__BIONIC_WEAK_FOR_NATIVE_BRIDGE
+#endif
+extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
+ // Functionality available in both static and dynamic libc.
+ if (opcode == M_GET_DECAY_TIME_ENABLED) {
+ if (arg == nullptr || arg_size != sizeof(bool)) {
+ errno = EINVAL;
+ return false;
+ }
+ *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
+ return true;
+ }
+ if (opcode == M_INITIALIZE_GWP_ASAN) {
+ if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
+ errno = EINVAL;
+ return false;
+ }
+
+ return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
+ }
+ if (opcode == M_MEMTAG_STACK_IS_ON) {
+ if (arg == nullptr || arg_size != sizeof(bool)) {
+ errno = EINVAL;
+ return false;
+ }
+ *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
+ return true;
+ }
+ if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
+ return LimitEnable(arg, arg_size);
+ }
+
+#if defined(LIBC_STATIC)
+ errno = ENOTSUP;
+ return false;
+#else
+ if (opcode == M_SET_ZYGOTE_CHILD) {
+ if (arg != nullptr || arg_size != 0) {
+ errno = EINVAL;
+ return false;
+ }
+ gZygoteChild = true;
+ return true;
+ }
+ if (opcode == M_INIT_ZYGOTE_CHILD_PROFILING) {
+ if (arg != nullptr || arg_size != 0) {
+ errno = EINVAL;
+ return false;
+ }
+ atomic_store_explicit(&gZygoteChildProfileable, true, memory_order_release);
+ // Also check if heapprofd should start profiling from app startup.
+ HeapprofdInitZygoteChildProfiling();
+ return true;
+ }
+ if (opcode == M_GET_PROCESS_PROFILEABLE) {
+ if (arg == nullptr || arg_size != sizeof(bool)) {
+ errno = EINVAL;
+ return false;
+ }
+ // Native processes are considered profileable. Zygote children are considered
+ // profileable only when appropriately tagged.
+ *reinterpret_cast<bool*>(arg) =
+ !gZygoteChild || atomic_load_explicit(&gZygoteChildProfileable, memory_order_acquire);
+ return true;
+ }
+ if (opcode == M_WRITE_MALLOC_LEAK_INFO_TO_FILE) {
+ if (arg == nullptr || arg_size != sizeof(FILE*)) {
+ errno = EINVAL;
+ return false;
+ }
+ return WriteMallocLeakInfo(reinterpret_cast<FILE*>(arg));
+ }
+ if (opcode == M_GET_MALLOC_LEAK_INFO) {
+ if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
+ errno = EINVAL;
+ return false;
+ }
+ return GetMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
+ }
+ if (opcode == M_FREE_MALLOC_LEAK_INFO) {
+ if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
+ errno = EINVAL;
+ return false;
+ }
+ return FreeMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
+ }
+ // Try heapprofd's mallopt, as it handles options not covered here.
+ return HeapprofdMallopt(opcode, arg, arg_size);
+#endif
+}
diff --git a/libc/bionic/libc_init_mte.cpp b/libc/bionic/libc_init_mte.cpp
new file mode 100644
index 0000000..3c8ef7d
--- /dev/null
+++ b/libc/bionic/libc_init_mte.cpp
@@ -0,0 +1,325 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <android/api-level.h>
+#include <elf.h>
+#include <errno.h>
+#include <malloc.h>
+#include <signal.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/auxv.h>
+#include <sys/mman.h>
+
+#include "async_safe/log.h"
+#include "heap_tagging.h"
+#include "libc_init_common.h"
+#include "platform/bionic/macros.h"
+#include "platform/bionic/mte.h"
+#include "platform/bionic/page.h"
+#include "platform/bionic/reserved_signals.h"
+#include "private/KernelArgumentBlock.h"
+#include "private/bionic_asm.h"
+#include "private/bionic_asm_note.h"
+#include "private/bionic_call_ifunc_resolver.h"
+#include "private/bionic_elf_tls.h"
+#include "private/bionic_globals.h"
+#include "private/bionic_tls.h"
+#include "private/elf_note.h"
+#include "pthread_internal.h"
+#include "sys/system_properties.h"
+#include "sysprop_helpers.h"
+
+#ifdef __aarch64__
+extern "C" const char* __gnu_basename(const char* path);
+
+static HeapTaggingLevel __get_memtag_level_from_note(const ElfW(Phdr) * phdr_start, size_t phdr_ct,
+ const ElfW(Addr) load_bias, bool* stack) {
+ const ElfW(Nhdr) * note;
+ const char* desc;
+ if (!__find_elf_note(NT_ANDROID_TYPE_MEMTAG, "Android", phdr_start, phdr_ct, ¬e, &desc,
+ load_bias)) {
+ return M_HEAP_TAGGING_LEVEL_TBI;
+ }
+
+ // Previously (in Android 12), if the note was != 4 bytes, we check-failed
+ // here. Let's be more permissive to allow future expansion.
+ if (note->n_descsz < 4) {
+ async_safe_fatal("unrecognized android.memtag note: n_descsz = %d, expected >= 4",
+ note->n_descsz);
+ }
+
+ // `desc` is always aligned due to ELF requirements, enforced in __find_elf_note().
+ ElfW(Word) note_val = *reinterpret_cast<const ElfW(Word)*>(desc);
+ *stack = (note_val & NT_MEMTAG_STACK) != 0;
+
+ // Warning: In Android 12, any value outside of bits [0..3] resulted in a check-fail.
+ if (!(note_val & (NT_MEMTAG_HEAP | NT_MEMTAG_STACK))) {
+ async_safe_format_log(ANDROID_LOG_INFO, "libc",
+ "unrecognised memtag note_val did not specificy heap or stack: %u",
+ note_val);
+ return M_HEAP_TAGGING_LEVEL_TBI;
+ }
+
+ unsigned mode = note_val & NT_MEMTAG_LEVEL_MASK;
+ switch (mode) {
+ case NT_MEMTAG_LEVEL_NONE:
+ // Note, previously (in Android 12), NT_MEMTAG_LEVEL_NONE was
+ // NT_MEMTAG_LEVEL_DEFAULT, which implied SYNC mode. This was never used
+ // by anyone, but we note it (heh) here for posterity, in case the zero
+ // level becomes meaningful, and binaries with this note can be executed
+ // on Android 12 devices.
+ return M_HEAP_TAGGING_LEVEL_TBI;
+ case NT_MEMTAG_LEVEL_ASYNC:
+ return M_HEAP_TAGGING_LEVEL_ASYNC;
+ case NT_MEMTAG_LEVEL_SYNC:
+ default:
+ // We allow future extensions to specify mode 3 (currently unused), with
+ // the idea that it might be used for ASYMM mode or something else. On
+ // this version of Android, it falls back to SYNC mode.
+ return M_HEAP_TAGGING_LEVEL_SYNC;
+ }
+}
+
+// Returns true if there's an environment setting (either sysprop or env var)
+// that should overwrite the ELF note, and places the equivalent heap tagging
+// level into *level.
+static bool get_environment_memtag_setting(HeapTaggingLevel* level) {
+ static const char kMemtagPrognameSyspropPrefix[] = "arm64.memtag.process.";
+ static const char kMemtagGlobalSysprop[] = "persist.arm64.memtag.default";
+ static const char kMemtagOverrideSyspropPrefix[] =
+ "persist.device_config.memory_safety_native.mode_override.process.";
+
+ const char* progname = __libc_shared_globals()->init_progname;
+ if (progname == nullptr) return false;
+
+ const char* basename = __gnu_basename(progname);
+
+ char options_str[PROP_VALUE_MAX];
+ char sysprop_name[512];
+ async_safe_format_buffer(sysprop_name, sizeof(sysprop_name), "%s%s", kMemtagPrognameSyspropPrefix,
+ basename);
+ char remote_sysprop_name[512];
+ async_safe_format_buffer(remote_sysprop_name, sizeof(remote_sysprop_name), "%s%s",
+ kMemtagOverrideSyspropPrefix, basename);
+ const char* sys_prop_names[] = {sysprop_name, remote_sysprop_name, kMemtagGlobalSysprop};
+
+ if (!get_config_from_env_or_sysprops("MEMTAG_OPTIONS", sys_prop_names, arraysize(sys_prop_names),
+ options_str, sizeof(options_str))) {
+ return false;
+ }
+
+ if (strcmp("sync", options_str) == 0) {
+ *level = M_HEAP_TAGGING_LEVEL_SYNC;
+ } else if (strcmp("async", options_str) == 0) {
+ *level = M_HEAP_TAGGING_LEVEL_ASYNC;
+ } else if (strcmp("off", options_str) == 0) {
+ *level = M_HEAP_TAGGING_LEVEL_TBI;
+ } else {
+ async_safe_format_log(
+ ANDROID_LOG_ERROR, "libc",
+ "unrecognized memtag level: \"%s\" (options are \"sync\", \"async\", or \"off\").",
+ options_str);
+ return false;
+ }
+
+ return true;
+}
+
+// Returns the initial heap tagging level. Note: This function will never return
+// M_HEAP_TAGGING_LEVEL_NONE, if MTE isn't enabled for this process we enable
+// M_HEAP_TAGGING_LEVEL_TBI.
+static HeapTaggingLevel __get_tagging_level(const memtag_dynamic_entries_t* memtag_dynamic_entries,
+ const void* phdr_start, size_t phdr_ct,
+ uintptr_t load_bias, bool* stack) {
+ HeapTaggingLevel level = M_HEAP_TAGGING_LEVEL_TBI;
+
+ // If the dynamic entries exist, use those. Otherwise, fall back to the old
+ // Android note, which is still used for fully static executables. When
+ // -fsanitize=memtag* is used in newer toolchains, currently both the dynamic
+ // entries and the old note are created, but we'd expect to move to just the
+ // dynamic entries for dynamically linked executables in the future. In
+ // addition, there's still some cleanup of the build system (that uses a
+ // manually-constructed note) needed. For more information about the dynamic
+ // entries, see:
+ // https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#dynamic-section
+ if (memtag_dynamic_entries && memtag_dynamic_entries->has_memtag_mode) {
+ switch (memtag_dynamic_entries->memtag_mode) {
+ case 0:
+ level = M_HEAP_TAGGING_LEVEL_SYNC;
+ break;
+ case 1:
+ level = M_HEAP_TAGGING_LEVEL_ASYNC;
+ break;
+ default:
+ async_safe_format_log(ANDROID_LOG_INFO, "libc",
+ "unrecognised DT_AARCH64_MEMTAG_MODE value: %u",
+ memtag_dynamic_entries->memtag_mode);
+ }
+ *stack = memtag_dynamic_entries->memtag_stack;
+ } else {
+ level = __get_memtag_level_from_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct,
+ load_bias, stack);
+ }
+
+ // We can't short-circuit the environment override, as `stack` is still inherited from the
+ // binary's settings.
+ get_environment_memtag_setting(&level);
+ return level;
+}
+
+static void __enable_mte_signal_handler(int, siginfo_t* info, void*) {
+ if (info->si_code != SI_TIMER) {
+ async_safe_format_log(ANDROID_LOG_ERROR, "libc", "Got BIONIC_ENABLE_MTE not from SI_TIMER");
+ return;
+ }
+ int tagged_addr_ctrl = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);
+ if (tagged_addr_ctrl < 0) {
+ async_safe_fatal("failed to PR_GET_TAGGED_ADDR_CTRL: %m");
+ }
+ if ((tagged_addr_ctrl & PR_MTE_TCF_MASK) != PR_MTE_TCF_NONE) {
+ return;
+ }
+ async_safe_format_log(ANDROID_LOG_INFO, "libc",
+ "Re-enabling MTE, value: %x (tagged_addr_ctrl %lu)",
+ info->si_value.sival_int, info->si_value.sival_int & PR_MTE_TCF_MASK);
+ tagged_addr_ctrl =
+ (tagged_addr_ctrl & ~PR_MTE_TCF_MASK) | (info->si_value.sival_int & PR_MTE_TCF_MASK);
+ if (prctl(PR_SET_TAGGED_ADDR_CTRL, tagged_addr_ctrl, 0, 0, 0) < 0) {
+ async_safe_fatal("failed to PR_SET_TAGGED_ADDR_CTRL %d: %m", tagged_addr_ctrl);
+ }
+}
+
+static int64_t __get_memtag_upgrade_secs() {
+ char* env = getenv("BIONIC_MEMTAG_UPGRADE_SECS");
+ if (!env) return 0;
+ int64_t timed_upgrade = 0;
+ static const char kAppProcessName[] = "app_process64";
+ const char* progname = __libc_shared_globals()->init_progname;
+ progname = progname ? __gnu_basename(progname) : nullptr;
+ // disable timed upgrade for zygote, as the thread spawned will violate the requirement
+ // that it be single-threaded.
+ if (!progname || strncmp(progname, kAppProcessName, sizeof(kAppProcessName)) != 0) {
+ char* endptr;
+ timed_upgrade = strtoll(env, &endptr, 10);
+ if (*endptr != '\0' || timed_upgrade < 0) {
+ async_safe_format_log(ANDROID_LOG_ERROR, "libc",
+ "Invalid value for BIONIC_MEMTAG_UPGRADE_SECS: %s", env);
+ timed_upgrade = 0;
+ }
+ }
+ // Make sure that this does not get passed to potential processes inheriting
+ // this environment.
+ unsetenv("BIONIC_MEMTAG_UPGRADE_SECS");
+ return timed_upgrade;
+}
+
+// Figure out the desired memory tagging mode (sync/async, heap/globals/stack) for this executable.
+// This function is called from the linker before the main executable is relocated.
+__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte(
+ const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start, size_t phdr_ct,
+ uintptr_t load_bias) {
+ bool memtag_stack = false;
+ HeapTaggingLevel level =
+ __get_tagging_level(memtag_dynamic_entries, phdr_start, phdr_ct, load_bias, &memtag_stack);
+ if (memtag_stack) __libc_shared_globals()->initial_memtag_stack_abi = true;
+
+ if (int64_t timed_upgrade = __get_memtag_upgrade_secs()) {
+ if (level == M_HEAP_TAGGING_LEVEL_ASYNC) {
+ async_safe_format_log(ANDROID_LOG_INFO, "libc",
+ "Attempting timed MTE upgrade from async to sync.");
+ __libc_shared_globals()->heap_tagging_upgrade_timer_sec = timed_upgrade;
+ level = M_HEAP_TAGGING_LEVEL_SYNC;
+ } else if (level != M_HEAP_TAGGING_LEVEL_SYNC) {
+ async_safe_format_log(ANDROID_LOG_ERROR, "libc",
+ "Requested timed MTE upgrade from invalid %s to sync. Ignoring.",
+ DescribeTaggingLevel(level));
+ }
+ }
+ if (level == M_HEAP_TAGGING_LEVEL_SYNC || level == M_HEAP_TAGGING_LEVEL_ASYNC) {
+ unsigned long prctl_arg = PR_TAGGED_ADDR_ENABLE | PR_MTE_TAG_SET_NONZERO;
+ prctl_arg |= (level == M_HEAP_TAGGING_LEVEL_SYNC) ? PR_MTE_TCF_SYNC : PR_MTE_TCF_ASYNC;
+
+ // When entering ASYNC mode, specify that we want to allow upgrading to SYNC by OR'ing in the
+ // SYNC flag. But if the kernel doesn't support specifying multiple TCF modes, fall back to
+ // specifying a single mode.
+ if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg | PR_MTE_TCF_SYNC, 0, 0, 0) == 0 ||
+ prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg, 0, 0, 0) == 0) {
+ __libc_shared_globals()->initial_heap_tagging_level = level;
+
+ struct sigaction action = {};
+ action.sa_flags = SA_SIGINFO | SA_RESTART;
+ action.sa_sigaction = __enable_mte_signal_handler;
+ sigaction(BIONIC_ENABLE_MTE, &action, nullptr);
+ return;
+ }
+ }
+
+ // MTE was either not enabled, or wasn't supported on this device. Try and use
+ // TBI.
+ if (prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0) == 0) {
+ __libc_shared_globals()->initial_heap_tagging_level = M_HEAP_TAGGING_LEVEL_TBI;
+ }
+ // We did not enable MTE, so we do not need to arm the upgrade timer.
+ __libc_shared_globals()->heap_tagging_upgrade_timer_sec = 0;
+}
+
+// Figure out whether we need to map the stack as PROT_MTE.
+// For dynamic executables, this has to be called after loading all
+// DT_NEEDED libraries, in case one of them needs stack MTE.
+__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte_stack(void* stack_top) {
+ if (!__libc_shared_globals()->initial_memtag_stack_abi) {
+ return;
+ }
+
+ // Even if the device doesn't support MTE, we have to allocate stack
+ // history buffers for code compiled for stack MTE. That is because the
+ // codegen expects a buffer to be present in TLS_SLOT_STACK_MTE either
+ // way.
+ __get_bionic_tcb()->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, nullptr);
+
+ if (__libc_mte_enabled()) {
+ __libc_shared_globals()->initial_memtag_stack = true;
+ void* pg_start = reinterpret_cast<void*>(page_start(reinterpret_cast<uintptr_t>(stack_top)));
+ if (mprotect(pg_start, page_size(), PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN)) {
+ async_safe_fatal("error: failed to set PROT_MTE on main thread stack: %m");
+ }
+ }
+}
+
+#else // __aarch64__
+void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t, uintptr_t) {}
+void __libc_init_mte_stack(void*) {}
+#endif // __aarch64__
+
+bool __libc_mte_enabled() {
+ HeapTaggingLevel lvl = __libc_shared_globals()->initial_heap_tagging_level;
+ return lvl == M_HEAP_TAGGING_LEVEL_SYNC || lvl == M_HEAP_TAGGING_LEVEL_ASYNC;
+}
diff --git a/libc/bionic/libc_init_static.cpp b/libc/bionic/libc_init_static.cpp
index 7c46113..553741d 100644
--- a/libc/bionic/libc_init_static.cpp
+++ b/libc/bionic/libc_init_static.cpp
@@ -157,260 +157,6 @@
layout.finish_layout();
}
-
-#ifdef __aarch64__
-static HeapTaggingLevel __get_memtag_level_from_note(const ElfW(Phdr) * phdr_start, size_t phdr_ct,
- const ElfW(Addr) load_bias, bool* stack) {
- const ElfW(Nhdr) * note;
- const char* desc;
- if (!__find_elf_note(NT_ANDROID_TYPE_MEMTAG, "Android", phdr_start, phdr_ct, ¬e, &desc,
- load_bias)) {
- return M_HEAP_TAGGING_LEVEL_TBI;
- }
-
- // Previously (in Android 12), if the note was != 4 bytes, we check-failed
- // here. Let's be more permissive to allow future expansion.
- if (note->n_descsz < 4) {
- async_safe_fatal("unrecognized android.memtag note: n_descsz = %d, expected >= 4",
- note->n_descsz);
- }
-
- // `desc` is always aligned due to ELF requirements, enforced in __find_elf_note().
- ElfW(Word) note_val = *reinterpret_cast<const ElfW(Word)*>(desc);
- *stack = (note_val & NT_MEMTAG_STACK) != 0;
-
- // Warning: In Android 12, any value outside of bits [0..3] resulted in a check-fail.
- if (!(note_val & (NT_MEMTAG_HEAP | NT_MEMTAG_STACK))) {
- async_safe_format_log(ANDROID_LOG_INFO, "libc",
- "unrecognised memtag note_val did not specificy heap or stack: %u",
- note_val);
- return M_HEAP_TAGGING_LEVEL_TBI;
- }
-
- unsigned mode = note_val & NT_MEMTAG_LEVEL_MASK;
- switch (mode) {
- case NT_MEMTAG_LEVEL_NONE:
- // Note, previously (in Android 12), NT_MEMTAG_LEVEL_NONE was
- // NT_MEMTAG_LEVEL_DEFAULT, which implied SYNC mode. This was never used
- // by anyone, but we note it (heh) here for posterity, in case the zero
- // level becomes meaningful, and binaries with this note can be executed
- // on Android 12 devices.
- return M_HEAP_TAGGING_LEVEL_TBI;
- case NT_MEMTAG_LEVEL_ASYNC:
- return M_HEAP_TAGGING_LEVEL_ASYNC;
- case NT_MEMTAG_LEVEL_SYNC:
- default:
- // We allow future extensions to specify mode 3 (currently unused), with
- // the idea that it might be used for ASYMM mode or something else. On
- // this version of Android, it falls back to SYNC mode.
- return M_HEAP_TAGGING_LEVEL_SYNC;
- }
-}
-
-// Returns true if there's an environment setting (either sysprop or env var)
-// that should overwrite the ELF note, and places the equivalent heap tagging
-// level into *level.
-static bool get_environment_memtag_setting(HeapTaggingLevel* level) {
- static const char kMemtagPrognameSyspropPrefix[] = "arm64.memtag.process.";
- static const char kMemtagGlobalSysprop[] = "persist.arm64.memtag.default";
- static const char kMemtagOverrideSyspropPrefix[] =
- "persist.device_config.memory_safety_native.mode_override.process.";
-
- const char* progname = __libc_shared_globals()->init_progname;
- if (progname == nullptr) return false;
-
- const char* basename = __gnu_basename(progname);
-
- char options_str[PROP_VALUE_MAX];
- char sysprop_name[512];
- async_safe_format_buffer(sysprop_name, sizeof(sysprop_name), "%s%s", kMemtagPrognameSyspropPrefix,
- basename);
- char remote_sysprop_name[512];
- async_safe_format_buffer(remote_sysprop_name, sizeof(remote_sysprop_name), "%s%s",
- kMemtagOverrideSyspropPrefix, basename);
- const char* sys_prop_names[] = {sysprop_name, remote_sysprop_name, kMemtagGlobalSysprop};
-
- if (!get_config_from_env_or_sysprops("MEMTAG_OPTIONS", sys_prop_names, arraysize(sys_prop_names),
- options_str, sizeof(options_str))) {
- return false;
- }
-
- if (strcmp("sync", options_str) == 0) {
- *level = M_HEAP_TAGGING_LEVEL_SYNC;
- } else if (strcmp("async", options_str) == 0) {
- *level = M_HEAP_TAGGING_LEVEL_ASYNC;
- } else if (strcmp("off", options_str) == 0) {
- *level = M_HEAP_TAGGING_LEVEL_TBI;
- } else {
- async_safe_format_log(
- ANDROID_LOG_ERROR, "libc",
- "unrecognized memtag level: \"%s\" (options are \"sync\", \"async\", or \"off\").",
- options_str);
- return false;
- }
-
- return true;
-}
-
-// Returns the initial heap tagging level. Note: This function will never return
-// M_HEAP_TAGGING_LEVEL_NONE, if MTE isn't enabled for this process we enable
-// M_HEAP_TAGGING_LEVEL_TBI.
-static HeapTaggingLevel __get_tagging_level(const memtag_dynamic_entries_t* memtag_dynamic_entries,
- const void* phdr_start, size_t phdr_ct,
- uintptr_t load_bias, bool* stack) {
- HeapTaggingLevel level = M_HEAP_TAGGING_LEVEL_TBI;
-
- // If the dynamic entries exist, use those. Otherwise, fall back to the old
- // Android note, which is still used for fully static executables. When
- // -fsanitize=memtag* is used in newer toolchains, currently both the dynamic
- // entries and the old note are created, but we'd expect to move to just the
- // dynamic entries for dynamically linked executables in the future. In
- // addition, there's still some cleanup of the build system (that uses a
- // manually-constructed note) needed. For more information about the dynamic
- // entries, see:
- // https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#dynamic-section
- if (memtag_dynamic_entries && memtag_dynamic_entries->has_memtag_mode) {
- switch (memtag_dynamic_entries->memtag_mode) {
- case 0:
- level = M_HEAP_TAGGING_LEVEL_SYNC;
- break;
- case 1:
- level = M_HEAP_TAGGING_LEVEL_ASYNC;
- break;
- default:
- async_safe_format_log(ANDROID_LOG_INFO, "libc",
- "unrecognised DT_AARCH64_MEMTAG_MODE value: %u",
- memtag_dynamic_entries->memtag_mode);
- }
- *stack = memtag_dynamic_entries->memtag_stack;
- } else {
- level = __get_memtag_level_from_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct,
- load_bias, stack);
- }
-
- // We can't short-circuit the environment override, as `stack` is still inherited from the
- // binary's settings.
- get_environment_memtag_setting(&level);
- return level;
-}
-
-static void __enable_mte_signal_handler(int, siginfo_t* info, void*) {
- if (info->si_code != SI_TIMER) {
- async_safe_format_log(ANDROID_LOG_ERROR, "libc", "Got BIONIC_ENABLE_MTE not from SI_TIMER");
- return;
- }
- int tagged_addr_ctrl = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);
- if (tagged_addr_ctrl < 0) {
- async_safe_fatal("failed to PR_GET_TAGGED_ADDR_CTRL: %m");
- }
- if ((tagged_addr_ctrl & PR_MTE_TCF_MASK) != PR_MTE_TCF_NONE) {
- return;
- }
- async_safe_format_log(ANDROID_LOG_INFO, "libc",
- "Re-enabling MTE, value: %x (tagged_addr_ctrl %lu)",
- info->si_value.sival_int, info->si_value.sival_int & PR_MTE_TCF_MASK);
- tagged_addr_ctrl =
- (tagged_addr_ctrl & ~PR_MTE_TCF_MASK) | (info->si_value.sival_int & PR_MTE_TCF_MASK);
- if (prctl(PR_SET_TAGGED_ADDR_CTRL, tagged_addr_ctrl, 0, 0, 0) < 0) {
- async_safe_fatal("failed to PR_SET_TAGGED_ADDR_CTRL %d: %m", tagged_addr_ctrl);
- }
-}
-
-static int64_t __get_memtag_upgrade_secs() {
- char* env = getenv("BIONIC_MEMTAG_UPGRADE_SECS");
- if (!env) return 0;
- int64_t timed_upgrade = 0;
- static const char kAppProcessName[] = "app_process64";
- const char* progname = __libc_shared_globals()->init_progname;
- progname = progname ? __gnu_basename(progname) : nullptr;
- // disable timed upgrade for zygote, as the thread spawned will violate the requirement
- // that it be single-threaded.
- if (!progname || strncmp(progname, kAppProcessName, sizeof(kAppProcessName)) != 0) {
- char* endptr;
- timed_upgrade = strtoll(env, &endptr, 10);
- if (*endptr != '\0' || timed_upgrade < 0) {
- async_safe_format_log(ANDROID_LOG_ERROR, "libc",
- "Invalid value for BIONIC_MEMTAG_UPGRADE_SECS: %s", env);
- timed_upgrade = 0;
- }
- }
- // Make sure that this does not get passed to potential processes inheriting
- // this environment.
- unsetenv("BIONIC_MEMTAG_UPGRADE_SECS");
- return timed_upgrade;
-}
-
-// Figure out the desired memory tagging mode (sync/async, heap/globals/stack) for this executable.
-// This function is called from the linker before the main executable is relocated.
-__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte(
- const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start, size_t phdr_ct,
- uintptr_t load_bias, void* stack_top) {
- bool memtag_stack = false;
- HeapTaggingLevel level =
- __get_tagging_level(memtag_dynamic_entries, phdr_start, phdr_ct, load_bias, &memtag_stack);
- // initial_memtag_stack is used by the linker (in linker.cpp) to communicate than any library
- // linked by this executable enables memtag-stack.
- // memtag_stack is also set for static executables if they request memtag stack via the note,
- // in which case it will differ from initial_memtag_stack.
- if (__libc_shared_globals()->initial_memtag_stack || memtag_stack) {
- memtag_stack = true;
- __libc_shared_globals()->initial_memtag_stack_abi = true;
- __get_bionic_tcb()->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, nullptr);
- }
- if (int64_t timed_upgrade = __get_memtag_upgrade_secs()) {
- if (level == M_HEAP_TAGGING_LEVEL_ASYNC) {
- async_safe_format_log(ANDROID_LOG_INFO, "libc",
- "Attempting timed MTE upgrade from async to sync.");
- __libc_shared_globals()->heap_tagging_upgrade_timer_sec = timed_upgrade;
- level = M_HEAP_TAGGING_LEVEL_SYNC;
- } else if (level != M_HEAP_TAGGING_LEVEL_SYNC) {
- async_safe_format_log(
- ANDROID_LOG_ERROR, "libc",
- "Requested timed MTE upgrade from invalid %s to sync. Ignoring.",
- DescribeTaggingLevel(level));
- }
- }
- if (level == M_HEAP_TAGGING_LEVEL_SYNC || level == M_HEAP_TAGGING_LEVEL_ASYNC) {
- unsigned long prctl_arg = PR_TAGGED_ADDR_ENABLE | PR_MTE_TAG_SET_NONZERO;
- prctl_arg |= (level == M_HEAP_TAGGING_LEVEL_SYNC) ? PR_MTE_TCF_SYNC : PR_MTE_TCF_ASYNC;
-
- // When entering ASYNC mode, specify that we want to allow upgrading to SYNC by OR'ing in the
- // SYNC flag. But if the kernel doesn't support specifying multiple TCF modes, fall back to
- // specifying a single mode.
- if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg | PR_MTE_TCF_SYNC, 0, 0, 0) == 0 ||
- prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg, 0, 0, 0) == 0) {
- __libc_shared_globals()->initial_heap_tagging_level = level;
- __libc_shared_globals()->initial_memtag_stack = memtag_stack;
-
- if (memtag_stack) {
- void* pg_start =
- reinterpret_cast<void*>(page_start(reinterpret_cast<uintptr_t>(stack_top)));
- if (mprotect(pg_start, page_size(), PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN)) {
- async_safe_fatal("error: failed to set PROT_MTE on main thread stack: %m");
- }
- }
- struct sigaction action = {};
- action.sa_flags = SA_SIGINFO | SA_RESTART;
- action.sa_sigaction = __enable_mte_signal_handler;
- sigaction(BIONIC_ENABLE_MTE, &action, nullptr);
- return;
- }
- }
-
- // MTE was either not enabled, or wasn't supported on this device. Try and use
- // TBI.
- if (prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0) == 0) {
- __libc_shared_globals()->initial_heap_tagging_level = M_HEAP_TAGGING_LEVEL_TBI;
- }
- // We did not enable MTE, so we do not need to arm the upgrade timer.
- __libc_shared_globals()->heap_tagging_upgrade_timer_sec = 0;
- // We also didn't enable memtag_stack.
- __libc_shared_globals()->initial_memtag_stack = false;
-}
-#else // __aarch64__
-void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t, uintptr_t, void*) {}
-#endif // __aarch64__
-
void __libc_init_profiling_handlers() {
// The dynamic variant of this function is more interesting, but this
// at least ensures that static binaries aren't killed by the kernel's
@@ -436,7 +182,8 @@
__libc_init_common();
__libc_init_mte(/*memtag_dynamic_entries=*/nullptr,
reinterpret_cast<ElfW(Phdr)*>(getauxval(AT_PHDR)), getauxval(AT_PHNUM),
- /*load_bias = */ 0, /*stack_top = */ raw_args);
+ /*load_bias = */ 0);
+ __libc_init_mte_stack(/*stack_top = */ raw_args);
__libc_init_scudo();
__libc_init_profiling_handlers();
__libc_init_fork_handler();
diff --git a/libc/bionic/malloc_common.cpp b/libc/bionic/malloc_common.cpp
index 596a1fc..441d884 100644
--- a/libc/bionic/malloc_common.cpp
+++ b/libc/bionic/malloc_common.cpp
@@ -332,44 +332,6 @@
#endif
// =============================================================================
-// =============================================================================
-// Platform-internal mallopt variant.
-// =============================================================================
-#if defined(LIBC_STATIC)
-extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
- if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
- return LimitEnable(arg, arg_size);
- }
- if (opcode == M_INITIALIZE_GWP_ASAN) {
- if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
- errno = EINVAL;
- return false;
- }
-
- return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
- }
- if (opcode == M_MEMTAG_STACK_IS_ON) {
- if (arg == nullptr || arg_size != sizeof(bool)) {
- errno = EINVAL;
- return false;
- }
- *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
- return true;
- }
- if (opcode == M_GET_DECAY_TIME_ENABLED) {
- if (arg == nullptr || arg_size != sizeof(bool)) {
- errno = EINVAL;
- return false;
- }
- *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
- return true;
- }
- errno = ENOTSUP;
- return false;
-}
-#endif
-// =============================================================================
-
static constexpr MallocDispatch __libc_malloc_default_dispatch __attribute__((unused)) = {
Malloc(calloc),
Malloc(free),
diff --git a/libc/bionic/malloc_common_dynamic.cpp b/libc/bionic/malloc_common_dynamic.cpp
index 6db6251..7b6d7d4 100644
--- a/libc/bionic/malloc_common_dynamic.cpp
+++ b/libc/bionic/malloc_common_dynamic.cpp
@@ -80,7 +80,7 @@
_Atomic bool gGlobalsMutating = false;
-static bool gZygoteChild = false;
+bool gZygoteChild = false;
// In a Zygote child process, this is set to true if profiling of this process
// is allowed. Note that this is set at a later time than gZygoteChild. The
@@ -89,7 +89,7 @@
// domains if applicable). These two flags are read by the
// BIONIC_SIGNAL_PROFILER handler, which does nothing if the process is not
// profileable.
-static _Atomic bool gZygoteChildProfileable = false;
+_Atomic bool gZygoteChildProfileable = false;
// =============================================================================
@@ -471,93 +471,6 @@
}
// =============================================================================
-// =============================================================================
-// Platform-internal mallopt variant.
-// =============================================================================
-__BIONIC_WEAK_FOR_NATIVE_BRIDGE
-extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
- if (opcode == M_SET_ZYGOTE_CHILD) {
- if (arg != nullptr || arg_size != 0) {
- errno = EINVAL;
- return false;
- }
- gZygoteChild = true;
- return true;
- }
- if (opcode == M_INIT_ZYGOTE_CHILD_PROFILING) {
- if (arg != nullptr || arg_size != 0) {
- errno = EINVAL;
- return false;
- }
- atomic_store_explicit(&gZygoteChildProfileable, true, memory_order_release);
- // Also check if heapprofd should start profiling from app startup.
- HeapprofdInitZygoteChildProfiling();
- return true;
- }
- if (opcode == M_GET_PROCESS_PROFILEABLE) {
- if (arg == nullptr || arg_size != sizeof(bool)) {
- errno = EINVAL;
- return false;
- }
- // Native processes are considered profileable. Zygote children are considered
- // profileable only when appropriately tagged.
- *reinterpret_cast<bool*>(arg) =
- !gZygoteChild || atomic_load_explicit(&gZygoteChildProfileable, memory_order_acquire);
- return true;
- }
- if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
- return LimitEnable(arg, arg_size);
- }
- if (opcode == M_WRITE_MALLOC_LEAK_INFO_TO_FILE) {
- if (arg == nullptr || arg_size != sizeof(FILE*)) {
- errno = EINVAL;
- return false;
- }
- return WriteMallocLeakInfo(reinterpret_cast<FILE*>(arg));
- }
- if (opcode == M_GET_MALLOC_LEAK_INFO) {
- if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
- errno = EINVAL;
- return false;
- }
- return GetMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
- }
- if (opcode == M_FREE_MALLOC_LEAK_INFO) {
- if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
- errno = EINVAL;
- return false;
- }
- return FreeMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
- }
- if (opcode == M_INITIALIZE_GWP_ASAN) {
- if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
- errno = EINVAL;
- return false;
- }
-
- return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
- }
- if (opcode == M_MEMTAG_STACK_IS_ON) {
- if (arg == nullptr || arg_size != sizeof(bool)) {
- errno = EINVAL;
- return false;
- }
- *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
- return true;
- }
- if (opcode == M_GET_DECAY_TIME_ENABLED) {
- if (arg == nullptr || arg_size != sizeof(bool)) {
- errno = EINVAL;
- return false;
- }
- *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
- return true;
- }
- // Try heapprofd's mallopt, as it handles options not covered here.
- return HeapprofdMallopt(opcode, arg, arg_size);
-}
-// =============================================================================
-
#if !defined(__LP64__) && defined(__arm__)
// =============================================================================
// Old platform only functions that some old 32 bit apps are still using.
diff --git a/libc/include/ctype.h b/libc/include/ctype.h
index cb926a4..dc3f673 100644
--- a/libc/include/ctype.h
+++ b/libc/include/ctype.h
@@ -95,7 +95,7 @@
/** Internal implementation detail. Do not use. */
__attribute__((__no_sanitize__("unsigned-integer-overflow")))
-static inline int __bionic_ctype_in_range(unsigned __lo, int __ch, unsigned __hi) {
+__BIONIC_CTYPE_INLINE int __bionic_ctype_in_range(unsigned __lo, int __ch, unsigned __hi) {
return (__BIONIC_CAST(static_cast, unsigned, __ch) - __lo) < (__hi - __lo + 1);
}
diff --git a/libc/kernel/tools/defaults.py b/libc/kernel/tools/defaults.py
index 06afb25..a71318e 100644
--- a/libc/kernel/tools/defaults.py
+++ b/libc/kernel/tools/defaults.py
@@ -133,6 +133,9 @@
# These are required to support the above functions.
"__fswahw32",
"__fswahb32",
+ # As are these, for ILP32.
+ "__arch_swab32",
+ "__arch_swab64",
# This is used by various macros in <linux/ioprio.h>.
"ioprio_value",
diff --git a/libc/kernel/uapi/asm-arm/asm/swab.h b/libc/kernel/uapi/asm-arm/asm/swab.h
index 7684c22..3fff953 100644
--- a/libc/kernel/uapi/asm-arm/asm/swab.h
+++ b/libc/kernel/uapi/asm-arm/asm/swab.h
@@ -11,7 +11,18 @@
#ifndef __STRICT_ANSI__
#define __SWAB_64_THRU_32__
#endif
+static inline __attribute__((__const__)) __u32 __arch_swab32(__u32 x) {
+ __u32 t;
#ifndef __thumb__
+ if(! __builtin_constant_p(x)) {
+ asm("eor\t%0, %1, %1, ror #16" : "=r" (t) : "r" (x));
+ } else
#endif
+ t = x ^ ((x << 16) | (x >> 16));
+ x = (x << 24) | (x >> 8);
+ t &= ~0x00FF0000;
+ x ^= (t >> 8);
+ return x;
+}
#define __arch_swab32 __arch_swab32
#endif
diff --git a/libc/kernel/uapi/asm-x86/asm/swab.h b/libc/kernel/uapi/asm-x86/asm/swab.h
index 31c850d..ce43658 100644
--- a/libc/kernel/uapi/asm-x86/asm/swab.h
+++ b/libc/kernel/uapi/asm-x86/asm/swab.h
@@ -8,9 +8,27 @@
#define _ASM_X86_SWAB_H
#include <linux/types.h>
#include <linux/compiler.h>
+static inline __attribute__((__const__)) __u32 __arch_swab32(__u32 val) {
+ asm("bswapl %0" : "=r" (val) : "0" (val));
+ return val;
+}
#define __arch_swab32 __arch_swab32
+static inline __attribute__((__const__)) __u64 __arch_swab64(__u64 val) {
#ifdef __i386__
+ union {
+ struct {
+ __u32 a;
+ __u32 b;
+ } s;
+ __u64 u;
+ } v;
+ v.u = val;
+ asm("bswapl %0; bswapl %1; xchgl %0,%1" : "=r" (v.s.a), "=r" (v.s.b) : "0" (v.s.a), "1" (v.s.b));
+ return v.u;
#else
+ asm("bswapq %0" : "=r" (val) : "0" (val));
+ return val;
#endif
+}
#define __arch_swab64 __arch_swab64
#endif
diff --git a/libc/private/bionic_globals.h b/libc/private/bionic_globals.h
index a1bebda..cd6dca9 100644
--- a/libc/private/bionic_globals.h
+++ b/libc/private/bionic_globals.h
@@ -157,6 +157,10 @@
};
__LIBC_HIDDEN__ libc_shared_globals* __libc_shared_globals();
+__LIBC_HIDDEN__ bool __libc_mte_enabled();
+__LIBC_HIDDEN__ void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t,
+ uintptr_t);
+__LIBC_HIDDEN__ void __libc_init_mte_stack(void*);
__LIBC_HIDDEN__ void __libc_init_fdsan();
__LIBC_HIDDEN__ void __libc_init_fdtrack();
__LIBC_HIDDEN__ void __libc_init_profiling_handlers();
diff --git a/linker/linker.cpp b/linker/linker.cpp
index fe7d348..227b599 100644
--- a/linker/linker.cpp
+++ b/linker/linker.cpp
@@ -1720,7 +1720,7 @@
} else {
// find_library is used by the initial linking step, so we communicate that we
// want memtag_stack enabled to __libc_init_mte.
- __libc_shared_globals()->initial_memtag_stack = true;
+ __libc_shared_globals()->initial_memtag_stack_abi = true;
}
}
diff --git a/linker/linker_main.cpp b/linker/linker_main.cpp
index 48ed723..69355d3 100644
--- a/linker/linker_main.cpp
+++ b/linker/linker_main.cpp
@@ -71,7 +71,9 @@
static void set_bss_vma_name(soinfo* si);
void __libc_init_mte(const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start,
- size_t phdr_count, uintptr_t load_bias, void* stack_top);
+ size_t phdr_count, uintptr_t load_bias);
+
+void __libc_init_mte_stack(void* stack_top);
static void __linker_cannot_link(const char* argv0) {
__linker_error("CANNOT LINK EXECUTABLE \"%s\": %s", argv0, linker_get_error_buffer());
@@ -465,8 +467,8 @@
#if defined(__aarch64__)
// This has to happen after the find_libraries, which will have collected any possible
// libraries that request memtag_stack in the dynamic section.
- __libc_init_mte(somain->memtag_dynamic_entries(), somain->phdr, somain->phnum, somain->load_bias,
- args.argv);
+ __libc_init_mte(somain->memtag_dynamic_entries(), somain->phdr, somain->phnum, somain->load_bias);
+ __libc_init_mte_stack(args.argv);
#endif
linker_finalize_static_tls();