Merge tag 'android-15.0.0_r32' of https://android.googlesource.com/platform/bionic into HEAD

Android 15.0.0 Release 32 (BP1A.250505.005)

Change-Id: I64da21df6c4e1a24fbe32a45b5daeb0a4638ff02

# -----BEGIN PGP SIGNATURE-----
#
# iF0EABECAB0WIQRDQNE1cO+UXoOBCWTorT+BmrEOeAUCaBqG7QAKCRDorT+BmrEO
# eMavAJ9JAC86mbm72SLYQqOPfTM5rGXzQgCfWf1lq9ODiMbYaQb0S/lMpOfdQa0=
# =ejHB
# -----END PGP SIGNATURE-----
# gpg: Signature faite le mar 06 mai 2025 18:02:21 EDT
# gpg:                avec la clef DSA 4340D13570EF945E83810964E8AD3F819AB10E78
# gpg: Impossible de vérifier la signature : Pas de clef publique
diff --git a/METADATA b/METADATA
deleted file mode 100644
index d97975c..0000000
--- a/METADATA
+++ /dev/null
@@ -1,3 +0,0 @@
-third_party {
-  license_type: NOTICE
-}
diff --git a/OWNERS b/OWNERS
index 3818b1d..1859b9e 100644
--- a/OWNERS
+++ b/OWNERS
@@ -5,3 +5,5 @@
 danalbert@google.com
 rprichard@google.com
 yabinc@google.com
+
+per-file docs/mte.md=eugenis@google.com,fmayer@google.com,pcc@google.com
diff --git a/docs/mte.md b/docs/mte.md
new file mode 100644
index 0000000..3034cc7
--- /dev/null
+++ b/docs/mte.md
@@ -0,0 +1,244 @@
+# Arm Memory Tagging Extension (MTE) implementation
+
+AOSP supports Arm MTE to detect invalid memory accesses. The implementation is
+spread across multiple components, both within and out of the AOSP tree. This
+document gives an overview and pointers about how the various MTE features are
+implemented.
+
+For documentation of the behavior rather than the implementation, see the
+[SAC page on MTE] instead. For MTE for apps, see the [NDK page on MTE].
+
+The relevant components are:
+
+* [LLVM Project] (out of AOSP tree)
+    * Stack tagging instrumentation pass
+    * Scudo memory allocator
+* bionic
+    * libc
+    * dynamic loader
+* Zygote
+* debuggerd
+* [NDK]
+
+## MTE enablement
+
+The way MTE is requested and enabled differs between native binaries and Java
+apps. This is necessarily so, because Java apps get forked from the Zygote,
+while native executables get inintialized by the linker.
+
+### Native binaries
+
+Both AOSP and the NDK allow you to compile C/C++ code that use MTE to detect
+memory safety issues. The [NDK legacy cmake toolchain] and the
+[NDK new cmake toolchain] both support "memtag" as an argument for
+`ANDROID_SANITIZE`. NDK make has no specific support for MTE, but the
+relevant flags can be passed directly as `CFLAGS` and `LDFLAGS`.
+
+For the OS itself, [Soong] supports "memtag_[heap|stack|globals]" as
+`SANITIZE_TARGET  and as `sanitize:` attribute in Android.bp files;
+[Android make] supports the same environment variables as Soong. This passes
+the appropriate flags to the clang driver for both compile and link steps.
+
+#### Linker
+
+* For **dynamic executables** LLD has support to
+  [add appropriate dynamic sections] as defined in the [ELF standard]
+* For **static executables** and as a fallback for older devices, LLD
+  also supports [adding the Android-specific ELF note]
+
+Both of the above are controlled by the linker flag `--android-memtag-mode`
+which is [passed in by the clang driver] if
+`-fsanitize=memtag-[stack|heap|globals]` is [passed in].
+`-fsanitize=memtag` [enables all three] (even for API levels that don't
+implement the runtime for globals, which means builds from old versions
+of clang may no work with newer platform versions that support globals).
+`-fsanitize-memtag-mode` allows to choose between ASYNC and SYNC.
+
+This information can be queried using `llvm-readelf --memtag`.
+
+This information is [picked up by libc init] to decide whether to enable MTE.
+`-fsanitize-heap` controls both whether scudo tags allocations, and whether
+tag checking is enabled.
+
+#### Runtime environment (dynamic loader, libc)
+
+There are two different initialization sequences for libc, both of which end up
+calling `__libc_init_mte`.
+
+N.B. the linker has its own copy of libc, which is used when executing these
+functions. That is why we have to use `__libc_shared_globals` to communicate
+with the libc of the process we are starting.
+
+* **static executables** `__libc_init` is called from `crtbegin.c`, which calls
+                         `__libc_init_mte`
+* **dynamic executables** the linker calls `__libc_init_mte`
+
+`__libc_init_mte` figures out the appropriate MTE level that is requested by
+the process, calls `prctl` to request this from the kernel, and stores data in
+`__libc_shared_globals` which gets picked up later to enable MTE in scudo.
+
+It also does work related to stack tagging and permissive mode, which will be
+detailed later.
+
+### Apps
+
+Apps can request MTE be enabled for their process via the manifest attribute
+`android:memtagMode`. This gets interpreted by Zygote, which always runs with
+`ASYNC` MTE enabled, because MTE for a process can only be disabled after
+it has been initialized (see [Native binaries](#native-binaries)), not enabled.
+
+[decideTaggingLevel] in the Zygote figures out whether to enable MTE for
+an app, and stores it in the `runtimeFlags`, which get picked up by
+[SpecializeCommon] after forking from the Zygote.
+
+## MTE implementation
+
+### Heap Tagging
+
+Heap tagging is implemented in the scudo allocator. On `malloc` and `free`,
+scudo will update the memory's tags to prevent use-after-free and buffer
+overflows.
+
+[scudo's memtag.h] contains helper functions to deal with MTE tag management,
+which are used in [combined.h] and [secondary.h].
+
+
+### Stack Tagging
+
+Stack tagging requires instrumenting function bodies. It is implemented as
+an instrumentation pass in LLVM called [AArch64StackTagging], which sets
+the tags according to the lifetime of stack objects.
+
+The instrumentation pass also supports recording stack history, consisting of:
+
+* PC
+* Frame pointer
+* Base tag
+
+This can be used to reconstruct which stack object was referred to in an
+invalid access. The logic to reconstruct this can be found in the
+[stack script].
+
+
+Stack tagging is enabled in one of two circumstances:
+* at process startup, if the main binary or any of its dependencies are
+  compiled with `memtag-stack`
+* library compiled with `memtag-stack` is `dlopen`ed later, either directly or
+  as a dependency of a `dlopen`ed library. In this case, the
+  [__pthread_internal_remap_stack_with_mte] function is used (called from
+  `memtag_stack_dlopen_callback`). Because `dlopen`
+  is handled by the linker, we have to [store a function pointer] to the
+  process's version of the function in `__libc_shared_globals`.
+
+Enabling stack MTE consists of two operations:
+* Remapping the stacks as `PROT_MTE`
+* Allocating a stack history buffer.
+
+The first operation is only necessary when the process is running with MTE
+enabled. The second operation is also necessary when the process is not running
+with MTE enabled, because the writes to the stack history buffer are
+unconditional.
+
+libc keeps track of this through two globals:
+
+* `__libc_memtag_stack`:  whether stack MTE is enabled on the process, i.e.
+  whether the stack pages are mapped with PROT\_MTE. This is always false if
+  MTE is disabled for the process (i.e. `libc_globals.memtag` is false).
+* `__libc_memtag_stack_abi`: whether the process contains any code that was
+  compiled with memtag-stack. This is true even if the process does not have
+  MTE enabled.
+
+### Globals Tagging
+
+TODO(fmayer): write once submitted
+
+### Crash reporting
+
+For MTE crashes, debuggerd serializes special information into the Tombstone
+proto:
+
+* Tags around fault address
+* Scudo allocation history
+
+This is done in [tombstone\_proto.cpp]. The information is converted to a text
+proto in [tombstone\_proto\_to\_text.cpp].
+
+## Bootloader control
+
+The bootloader API allows userspace to enable MTE on devices that do not ship
+with MTE enabled by default.
+
+See [SAC MTE bootloader support] for the API definition. In AOSP, this API is
+implemented in [system/extras/mtectrl]. mtectrl.rc handles the property
+changes and invokes mtectrl to update the misc partition to communicate
+with the bootloader.
+
+There is also an [API in Device Policy Manager] that allows the device admin
+to enable or disable MTE under certain circumstances.
+
+The device can opt in or out of these APIs by a set of system properties:
+
+* `ro.arm64.memtag.bootctl_supported`: the system property API is supported,
+  and an option is displayed in Developer Options.
+* `ro.arm64.memtag.bootctl_settings_toggle`: an option is displayed in the
+  normal settings. This requires `ro.arm64.memtag.bootctl_supported` to be
+  true. This implies `ro.arm64.memtag.bootctl_device_policy_manager`, if it
+  is not explicitely set.
+* `ro.arm64.memtag.bootctl_device_policy_manager`: the Device Policy Manager
+  API is supported.
+
+## Permissive MTE
+
+Permissive MTE refers to a mode which, instead of crashing the process on an
+MTE fault, records a tombstone but then continues execution of the process.
+An important caveat is that system calls with invalid pointers (where the
+pointer tag does not match the memory tag) still return an error code.
+
+This mode is only available for system services, not apps. It is implemented
+in the [debugger\_signal\_handler] by disabling MTE for the faulting thread.
+Optionally, the user can ask for MTE to be re-enabled after some time.
+This is achieved by arming a timer that calls [enable_mte_signal_handler]
+upon expiry.
+
+## MTE Mode Upgrade
+
+When a system service [crashes in ASYNC mode], we set an impossible signal
+as an exit code (because that signal is always gracefully handled by libc),
+and [in init] we set `BIONIC_MEMTAG_UPGRADE_SECS`, which gets handled by
+[libc startup].
+
+[SpecializeCommon]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/jni/com_android_internal_os_Zygote.cpp?q=f:frameworks%2Fbase%2Fcore%2Fjni%2Fcom_android_internal_os_Zygote.cpp%20%22%20mallopt(M_BIONIC_SET_HEAP_TAGGING_LEVEL,%22&ss=android%2Fplatform%2Fsuperproject%2Fmain
+[LLVM Project]: https://github.com/llvm/llvm-project/
+[NDK]: https://android.googlesource.com/platform/ndk/
+[NDK legacy cmake toolchain]: https://android.googlesource.com/platform/ndk/+/refs/heads/main/build/cmake/android-legacy.toolchain.cmake#490
+[NDK new cmake toolchain]: https://android.googlesource.com/platform/ndk/+/refs/heads/main/build/cmake/flags.cmake#56
+[Soong]: https://cs.android.com/android/platform/superproject/main/+/main:build/soong/cc/sanitize.go?q=sanitize.go&ss=android%2Fplatform%2Fsuperproject%2Fmain
+[decideTaggingLevel]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/com/android/internal/os/Zygote.java?q=symbol:decideTaggingLevel
+[picked up by libc init]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_mte.cpp?q=symbol:__get_tagging_level%20f:bionic
+[enables all three]: https://github.com/llvm/llvm-project/blob/e732d1ce86783b1d7fe30645fcb30434109505b9/clang/include/clang/Basic/Sanitizers.def#L62
+[passed in]: https://github.com/llvm/llvm-project/blob/ff2e619dfcd77328812a42d2ba2b11c3ff96f410/clang/lib/Driver/SanitizerArgs.cpp#L719
+[passed in by the clang driver]: https://github.com/llvm/llvm-project/blob/ff2e619dfcd77328812a42d2ba2b11c3ff96f410/clang/lib/Driver/ToolChains/CommonArgs.cpp#L1595
+[adding the Android-specific ELF note]: https://github.com/llvm/llvm-project/blob/435cb0dc5eca08cdd8d9ed0d887fa1693cc2bf33/lld/ELF/Driver.cpp#L1258
+[ELF standard]: https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#6dynamic-section
+[add appropriate dynamic sections]: https://github.com/llvm/llvm-project/blob/7022498ac2f236e411e8a0f9a48669e754000a4b/lld/ELF/SyntheticSections.cpp#L1473
+[storeTags]: https://cs.android.com/android/platform/superproject/main/+/main:external/scudo/standalone/memtag.h?q=f:scudo%20f:memtag.h%20function:storeTags
+[SAC page on MTE]: https://source.android.com/docs/security/test/memory-safety/arm-mte
+[NDK page on MTE]: https://developer.android.com/ndk/guides/arm-mte
+[AArch64StackTagging]: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/AArch64/AArch64StackTagging.cpp
+[scudo's memtag.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/memtag.h
+[combined.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/combined.h
+[secondary.h]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/scudo/standalone/secondary.h
+[__pthread_internal_remap_stack_with_mte]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/pthread_internal.cpp?q=__pthread_internal_remap_stack_with_mte
+[stack script]: https://cs.android.com/android/platform/superproject/main/+/main:development/scripts/stack?q=stack
+[Android make]: https://cs.android.com/android/platform/superproject/main/+/main:build/make/core/config_sanitizers.mk
+[store a function pointer]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_dynamic.cpp;l=168?q=memtag_stack_dlopen_callback
+[tombstone\_proto.cpp]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/libdebuggerd/tombstone_proto.cpp?q=tombstone_proto.cpp
+[tombstone\_proto\_to\_text.cpp]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/libdebuggerd/tombstone_proto_to_text.cpp
+[SAC MTE bootloader support]: https://source.android.com/docs/security/test/memory-safety/bootloader-support
+[system/extras/mtectrl]: https://cs.android.com/android/platform/superproject/main/+/main:system/extras/mtectrl/
+[API in Device Policy Manager]: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/admin/DevicePolicyManager.java?q=symbol:setMtePolicy%20f:DevicePolicyManager.java
+[debuggerd\_signal_handler]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/handler/debuggerd_handler.cpp?q=f:debuggerd_handler.cpp%20symbol:debuggerd_signal_handler
+[enable_mte_signal_handler]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_mte.cpp?q=symbol:__enable_mte_signal_handler
+[in init]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/init/service.cpp?q=f:system%2Fcore%2Finit%2Fservice.cpp%20should_upgrade_mte
+[crashes in ASYNC mode]: https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/handler/debuggerd_handler.cpp;l=799?q=BIONIC_SIGNAL_ART_PROFILER
+[libc startup]: https://cs.android.com/android/platform/superproject/main/+/main:bionic/libc/bionic/libc_init_mte.cpp?q=BIONIC_MEMTAG_UPGRADE_SECS
diff --git a/docs/status.md b/docs/status.md
index 94784de..e7111d9 100644
--- a/docs/status.md
+++ b/docs/status.md
@@ -62,6 +62,7 @@
 New libc functions in API level 36:
   * `qsort_r`, `sig2str`/`str2sig` (POSIX Issue 8 additions).
   * GNU/BSD extension `lchmod`.
+  * GNU extensions `pthread_getaffinity_np`/`pthread_setaffinity_np`.
   * New system call wrapper: `mseal` (`<sys/mman.h>`).
 
 New libc functions in V (API level 35):
diff --git a/libc/Android.bp b/libc/Android.bp
index eeea728..c34023c 100644
--- a/libc/Android.bp
+++ b/libc/Android.bp
@@ -264,6 +264,7 @@
     name: "libc_init_static",
     defaults: ["libc_defaults"],
     srcs: [
+        "bionic/libc_init_mte.cpp",
         "bionic/libc_init_static.cpp",
         ":elf_note_sources",
     ],
@@ -353,7 +354,6 @@
         "-Wno-unused-parameter",
         "-include netbsd-compat.h",
         "-Wframe-larger-than=66000",
-        "-include private/bsd_sys_param.h",
     ],
 
     local_include_dirs: [
@@ -418,20 +418,6 @@
         "upstream-freebsd/lib/libc/string/wmemmove.c",
         "upstream-freebsd/lib/libc/string/wmemset.c",
     ],
-    arch: {
-        x86: {
-            exclude_srcs: [
-                "upstream-freebsd/lib/libc/string/wcschr.c",
-                "upstream-freebsd/lib/libc/string/wcscmp.c",
-                "upstream-freebsd/lib/libc/string/wcslen.c",
-                "upstream-freebsd/lib/libc/string/wcsrchr.c",
-                "upstream-freebsd/lib/libc/string/wmemcmp.c",
-                "upstream-freebsd/lib/libc/string/wcscat.c",
-                "upstream-freebsd/lib/libc/string/wcscpy.c",
-                "upstream-freebsd/lib/libc/string/wmemcmp.c",
-            ],
-        },
-    },
 
     cflags: [
         "-Wno-sign-compare",
@@ -485,7 +471,6 @@
         "upstream-netbsd/lib/libc/regex/regerror.c",
         "upstream-netbsd/lib/libc/regex/regexec.c",
         "upstream-netbsd/lib/libc/regex/regfree.c",
-        "upstream-netbsd/lib/libc/stdlib/bsearch.c",
         "upstream-netbsd/lib/libc/stdlib/drand48.c",
         "upstream-netbsd/lib/libc/stdlib/erand48.c",
         "upstream-netbsd/lib/libc/stdlib/jrand48.c",
@@ -638,10 +623,7 @@
         arm: {
             srcs: [
                 "upstream-openbsd/lib/libc/string/memchr.c",
-                "upstream-openbsd/lib/libc/string/memrchr.c",
                 "upstream-openbsd/lib/libc/string/stpncpy.c",
-                "upstream-openbsd/lib/libc/string/strlcat.c",
-                "upstream-openbsd/lib/libc/string/strlcpy.c",
                 "upstream-openbsd/lib/libc/string/strncat.c",
                 "upstream-openbsd/lib/libc/string/strncmp.c",
                 "upstream-openbsd/lib/libc/string/strncpy.c",
@@ -651,18 +633,13 @@
             srcs: [
                 "upstream-openbsd/lib/libc/string/strcat.c",
                 "upstream-openbsd/lib/libc/string/stpncpy.c",
-                "upstream-openbsd/lib/libc/string/strlcat.c",
-                "upstream-openbsd/lib/libc/string/strlcpy.c",
                 "upstream-openbsd/lib/libc/string/strncat.c",
                 "upstream-openbsd/lib/libc/string/strncpy.c",
             ],
         },
         riscv64: {
             srcs: [
-                "upstream-openbsd/lib/libc/string/memrchr.c",
                 "upstream-openbsd/lib/libc/string/stpncpy.c",
-                "upstream-openbsd/lib/libc/string/strlcat.c",
-                "upstream-openbsd/lib/libc/string/strlcpy.c",
             ],
         },
         x86: {
@@ -672,10 +649,7 @@
         },
         x86_64: {
             srcs: [
-                "upstream-openbsd/lib/libc/string/memchr.c",
-                "upstream-openbsd/lib/libc/string/memrchr.c",
-                "upstream-openbsd/lib/libc/string/strlcat.c",
-                "upstream-openbsd/lib/libc/string/strlcpy.c",
+                // x86_64 has custom/llvm-libc implementations of all of these.
             ],
         },
     },
@@ -955,6 +929,7 @@
         "bionic/pthread_detach.cpp",
         "bionic/pthread_equal.cpp",
         "bionic/pthread_exit.cpp",
+        "bionic/pthread_getaffinity.cpp",
         "bionic/pthread_getcpuclockid.cpp",
         "bionic/pthread_getschedparam.cpp",
         "bionic/pthread_gettid_np.cpp",
@@ -968,6 +943,7 @@
         "bionic/pthread_sigqueue.cpp",
         "bionic/pthread_self.cpp",
         "bionic/pthread_setname_np.cpp",
+        "bionic/pthread_setaffinity.cpp",
         "bionic/pthread_setschedparam.cpp",
         "bionic/pthread_spinlock.cpp",
         "bionic/ptrace.cpp",
@@ -1183,7 +1159,6 @@
 
                 "arch-x86/string/sse2-memchr-atom.S",
                 "arch-x86/string/sse2-memmove-slm.S",
-                "arch-x86/string/sse2-memrchr-atom.S",
                 "arch-x86/string/sse2-memset-slm.S",
                 "arch-x86/string/sse2-stpcpy-slm.S",
                 "arch-x86/string/sse2-stpncpy-slm.S",
@@ -1193,24 +1168,14 @@
                 "arch-x86/string/sse2-strncpy-slm.S",
                 "arch-x86/string/sse2-strnlen-atom.S",
                 "arch-x86/string/sse2-strrchr-atom.S",
-                "arch-x86/string/sse2-wcschr-atom.S",
-                "arch-x86/string/sse2-wcsrchr-atom.S",
-                "arch-x86/string/sse2-wcslen-atom.S",
-                "arch-x86/string/sse2-wcscmp-atom.S",
 
                 "arch-x86/string/ssse3-memcmp-atom.S",
                 "arch-x86/string/ssse3-strcat-atom.S",
                 "arch-x86/string/ssse3-strcmp-atom.S",
-                "arch-x86/string/ssse3-strlcat-atom.S",
-                "arch-x86/string/ssse3-strlcpy-atom.S",
                 "arch-x86/string/ssse3-strncat-atom.S",
                 "arch-x86/string/ssse3-strncmp-atom.S",
-                "arch-x86/string/ssse3-wcscat-atom.S",
-                "arch-x86/string/ssse3-wcscpy-atom.S",
-                "arch-x86/string/ssse3-wmemcmp-atom.S",
 
                 "arch-x86/string/sse4-memcmp-slm.S",
-                "arch-x86/string/sse4-wmemcmp-slm.S",
 
                 "bionic/strchrnul.cpp",
             ],
@@ -1237,11 +1202,6 @@
                 "arch-x86_64/string/sse4-memcmp-slm.S",
                 "arch-x86_64/string/ssse3-strcmp-slm.S",
                 "arch-x86_64/string/ssse3-strncmp-slm.S",
-
-                "bionic/strchr.cpp",
-                "bionic/strchrnul.cpp",
-                "bionic/strnlen.cpp",
-                "bionic/strrchr.cpp",
             ],
         },
     },
@@ -1260,6 +1220,7 @@
     generated_headers: ["generated_android_ids"],
 
     whole_static_libs: [
+        "//external/llvm-libc:llvmlibc",
         "libsystemproperties",
     ],
 
@@ -1512,6 +1473,7 @@
     srcs: [
         "arch-common/bionic/crtbegin_so.c",
         "arch-common/bionic/crtbrand.S",
+        "bionic/android_mallopt.cpp",
         "bionic/gwp_asan_wrappers.cpp",
         "bionic/heap_tagging.cpp",
         "bionic/icu.cpp",
@@ -1530,6 +1492,7 @@
 filegroup {
     name: "libc_sources_static",
     srcs: [
+        "bionic/android_mallopt.cpp",
         "bionic/gwp_asan_wrappers.cpp",
         "bionic/heap_tagging.cpp",
         "bionic/icu_static.cpp",
@@ -1696,8 +1659,14 @@
     },
     native_bridge_supported: false,
     // It is never correct to depend on this directly. This is only
-    // needed for the runtime apex, and in base_system.mk.
-    visibility: ["//bionic/apex"],
+    // needed for the runtime apex, and in base_system.mk, and system_image_defaults
+    // which is default module for soong-defined system image.
+    visibility: [
+        "//bionic/apex",
+        "//build/make/target/product/generic",
+        //TODO(b/381985636) : Remove visibility to Soong-defined GSI once resolved
+        "//build/make/target/product/gsi",
+    ],
 }
 
 genrule {
@@ -1783,6 +1752,7 @@
     name: "libc_uapi_headers",
     visibility: [
         "//external/musl",
+        "//external/rust/crates/v4l2r/android",
     ],
     llndk: {
         llndk_headers: true,
@@ -2018,15 +1988,6 @@
 cc_defaults {
     name: "crt_so_defaults",
     defaults: ["crt_defaults"],
-
-    arch: {
-        x86: {
-            cflags: ["-fPIC"],
-        },
-        x86_64: {
-            cflags: ["-fPIC"],
-        },
-    },
     stl: "none",
 }
 
@@ -2166,6 +2127,7 @@
 //     async_safe_fatal_va_list
 cc_library_static {
     name: "librust_baremetal",
+    defaults: ["cc_baremetal_defaults"],
     header_libs: ["libc_headers"],
     include_dirs: [
         "bionic/libc/async_safe/include",
@@ -2192,6 +2154,7 @@
         },
     },
     whole_static_libs: [
+        "//external/llvm-libc:llvmlibc",
         "libarm-optimized-routines-mem",
         "libc_netbsd",
     ],
diff --git a/libc/NOTICE b/libc/NOTICE
index 1a84d3c..bca4891 100644
--- a/libc/NOTICE
+++ b/libc/NOTICE
@@ -3585,22 +3585,6 @@
 
 -------------------------------------------------------------------
 
-Copyright (c) 2007 Todd C. Miller <millert@openbsd.org>
-
-Permission to use, copy, modify, and distribute this software for any
-purpose with or without fee is hereby granted, provided that the above
-copyright notice and this permission notice appear in all copies.
-
-THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
-WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
-ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
-WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
-ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
-OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
--------------------------------------------------------------------
-
 Copyright (c) 2007-2008  Michael G Schwern
 
 This software originally derived from Paul Sheer's pivotal_gmtime_r.c.
@@ -3862,36 +3846,6 @@
 
 -------------------------------------------------------------------
 
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
--------------------------------------------------------------------
-
 Copyright (c) 2011 Martin Pieuchot <mpi@openbsd.org>
 
 Permission to use, copy, modify, and distribute this software for any
@@ -3937,36 +3891,6 @@
 
 -------------------------------------------------------------------
 
-Copyright (c) 2011, 2012, 2013 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
--------------------------------------------------------------------
-
 Copyright (c) 2011, Intel Corporation
 All rights reserved.
 
diff --git a/libc/arch-common/bionic/crt_pad_segment.S b/libc/arch-common/bionic/crt_pad_segment.S
index 86c730d..2fbe0b9 100644
--- a/libc/arch-common/bionic/crt_pad_segment.S
+++ b/libc/arch-common/bionic/crt_pad_segment.S
@@ -26,6 +26,12 @@
  * SUCH DAMAGE.
  */
 
+#if defined(__aarch64__)
+#include <private/bionic_asm_arm64.h>
+
+__bionic_asm_custom_note_gnu_section()
+#endif
+
 #include <private/bionic_asm_note.h>
 
   .section ".note.android.pad_segment", "a", %note
diff --git a/libc/arch-x86/dynamic_function_dispatch.cpp b/libc/arch-x86/dynamic_function_dispatch.cpp
index 98d7ec2..240fcdf 100644
--- a/libc/arch-x86/dynamic_function_dispatch.cpp
+++ b/libc/arch-x86/dynamic_function_dispatch.cpp
@@ -38,14 +38,4 @@
 }
 MEMCMP_SHIM()
 
-typedef int wmemcmp_func_t(const wchar_t*, const wchar_t*, size_t);
-DEFINE_IFUNC_FOR(wmemcmp) {
-  __builtin_cpu_init();
-  if (__builtin_cpu_supports("sse4.1")) RETURN_FUNC(wmemcmp_func_t, wmemcmp_sse4);
-  RETURN_FUNC(wmemcmp_func_t, wmemcmp_atom);
-}
-DEFINE_STATIC_SHIM(int wmemcmp(const wchar_t* lhs, const wchar_t* rhs, size_t n) {
-  FORWARD(wmemcmp)(lhs, rhs, n);
-})
-
 }  // extern "C"
diff --git a/libc/arch-x86/string/sse2-memrchr-atom.S b/libc/arch-x86/string/sse2-memrchr-atom.S
deleted file mode 100644
index 1aa1a1a..0000000
--- a/libc/arch-x86/string/sse2-memrchr-atom.S
+++ /dev/null
@@ -1,778 +0,0 @@
-/*
-Copyright (c) 2011, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef L
-# define L(label)	.L##label
-#endif
-
-#ifndef cfi_startproc
-# define cfi_startproc	.cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc	.cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name,  @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name,	.-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG) pushl REG; CFI_PUSH (REG)
-#define POP(REG) popl REG; CFI_POP (REG)
-
-#define PARMS  4
-#define STR1  PARMS
-#define STR2  STR1+4
-#define LEN   STR2+4
-
-	.text
-ENTRY (memrchr)
-	mov	STR1(%esp), %ecx
-	movd	STR2(%esp), %xmm1
-	mov	LEN(%esp), %edx
-
-	test	%edx, %edx
-	jz	L(return_null)
-	sub	$16, %edx
-	jbe	L(length_less16)
-
-	punpcklbw %xmm1, %xmm1
-	add	%edx, %ecx
-	punpcklbw %xmm1, %xmm1
-
-	movdqu	(%ecx), %xmm0
-	pshufd	$0, %xmm1, %xmm1
-	pcmpeqb	%xmm1, %xmm0
-
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(exit_dispatch)
-
-	sub	$64, %ecx
-	mov	%ecx, %eax
-	and	$15, %eax
-	jz	L(loop_prolog)
-
-	add	$16, %ecx
-	add	$16, %edx
-	and	$-16, %ecx
-	sub	%eax, %edx
-
-	.p2align 4
-/* Loop start on aligned string.  */
-L(loop_prolog):
-	sub	$64, %edx
-	jbe	L(exit_loop)
-
-	movdqa	48(%ecx), %xmm0
-	pcmpeqb	%xmm1, %xmm0
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(matches48)
-
-	movdqa	32(%ecx), %xmm2
-	pcmpeqb	%xmm1, %xmm2
-	pmovmskb %xmm2, %eax
-	test	%eax, %eax
-	jnz	L(matches32)
-
-	movdqa	16(%ecx), %xmm3
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %eax
-	test	%eax, %eax
-	jnz	L(matches16)
-
-	movdqa	(%ecx), %xmm4
-	pcmpeqb	%xmm1, %xmm4
-	pmovmskb %xmm4, %eax
-	test	%eax, %eax
-	jnz	L(exit_dispatch)
-
-	sub	$64, %ecx
-	sub	$64, %edx
-	jbe	L(exit_loop)
-
-	movdqa	48(%ecx), %xmm0
-	pcmpeqb	%xmm1, %xmm0
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(matches48)
-
-	movdqa	32(%ecx), %xmm2
-	pcmpeqb	%xmm1, %xmm2
-	pmovmskb %xmm2, %eax
-	test	%eax, %eax
-	jnz	L(matches32)
-
-	movdqa	16(%ecx), %xmm3
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %eax
-	test	%eax, %eax
-	jnz	L(matches16)
-
-	movdqa	(%ecx), %xmm3
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %eax
-	test	%eax, %eax
-	jnz	L(exit_dispatch)
-
-	mov	%ecx, %eax
-	and	$63, %eax
-	test	%eax, %eax
-	jz	L(align64_loop)
-
-	add	$64, %ecx
-	add	$64, %edx
-	and	$-64, %ecx
-	sub	%eax, %edx
-
-	.p2align 4
-L(align64_loop):
-	sub	$64, %ecx
-	sub	$64, %edx
-	jbe	L(exit_loop)
-
-	movdqa	(%ecx), %xmm0
-	movdqa	16(%ecx), %xmm2
-	movdqa	32(%ecx), %xmm3
-	movdqa	48(%ecx), %xmm4
-
-	pcmpeqb	%xmm1, %xmm0
-	pcmpeqb	%xmm1, %xmm2
-	pcmpeqb	%xmm1, %xmm3
-	pcmpeqb	%xmm1, %xmm4
-
-	pmaxub	%xmm3, %xmm0
-	pmaxub	%xmm4, %xmm2
-	pmaxub	%xmm0, %xmm2
-	pmovmskb %xmm2, %eax
-
-	test	%eax, %eax
-	jz	L(align64_loop)
-
-	pmovmskb %xmm4, %eax
-	test	%eax, %eax
-	jnz	L(matches48)
-
-	pmovmskb %xmm3, %eax
-	test	%eax, %eax
-	jnz	L(matches32)
-
-	movdqa	16(%ecx), %xmm2
-
-	pcmpeqb	%xmm1, %xmm2
-	pcmpeqb	(%ecx), %xmm1
-
-	pmovmskb %xmm2, %eax
-	test	%eax, %eax
-	jnz	L(matches16)
-
-	pmovmskb %xmm1, %eax
-	test	%ah, %ah
-	jnz	L(exit_dispatch_high)
-	mov	%al, %dl
-	and	$15 << 4, %dl
-	jnz	L(exit_dispatch_8)
-	test	$0x08, %al
-	jnz	L(exit_4)
-	test	$0x04, %al
-	jnz	L(exit_3)
-	test	$0x02, %al
-	jnz	L(exit_2)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(exit_loop):
-	add	$64, %edx
-	cmp	$32, %edx
-	jbe	L(exit_loop_32)
-
-	movdqa	48(%ecx), %xmm0
-	pcmpeqb	%xmm1, %xmm0
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(matches48)
-
-	movdqa	32(%ecx), %xmm2
-	pcmpeqb	%xmm1, %xmm2
-	pmovmskb %xmm2, %eax
-	test	%eax, %eax
-	jnz	L(matches32)
-
-	movdqa	16(%ecx), %xmm3
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %eax
-	test	%eax, %eax
-	jnz	L(matches16_1)
-	cmp	$48, %edx
-	jbe	L(return_null)
-
-	pcmpeqb	(%ecx), %xmm1
-	pmovmskb %xmm1, %eax
-	test	%eax, %eax
-	jnz	L(matches0_1)
-	xor	%eax, %eax
-	ret
-
-	.p2align 4
-L(exit_loop_32):
-	movdqa	48(%ecx), %xmm0
-	pcmpeqb	%xmm1, %xmm0
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(matches48_1)
-	cmp	$16, %edx
-	jbe	L(return_null)
-
-	pcmpeqb	32(%ecx), %xmm1
-	pmovmskb %xmm1, %eax
-	test	%eax, %eax
-	jnz	L(matches32_1)
-	xor	%eax, %eax
-	ret
-
-	.p2align 4
-L(matches16):
-	lea	16(%ecx), %ecx
-	test	%ah, %ah
-	jnz	L(exit_dispatch_high)
-	mov	%al, %dl
-	and	$15 << 4, %dl
-	jnz	L(exit_dispatch_8)
-	test	$0x08, %al
-	jnz	L(exit_4)
-	test	$0x04, %al
-	jnz	L(exit_3)
-	test	$0x02, %al
-	jnz	L(exit_2)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(matches32):
-	lea	32(%ecx), %ecx
-	test	%ah, %ah
-	jnz	L(exit_dispatch_high)
-	mov	%al, %dl
-	and	$15 << 4, %dl
-	jnz	L(exit_dispatch_8)
-	test	$0x08, %al
-	jnz	L(exit_4)
-	test	$0x04, %al
-	jnz	L(exit_3)
-	test	$0x02, %al
-	jnz	L(exit_2)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(matches48):
-	lea	48(%ecx), %ecx
-
-	.p2align 4
-L(exit_dispatch):
-	test	%ah, %ah
-	jnz	L(exit_dispatch_high)
-	mov	%al, %dl
-	and	$15 << 4, %dl
-	jnz	L(exit_dispatch_8)
-	test	$0x08, %al
-	jnz	L(exit_4)
-	test	$0x04, %al
-	jnz	L(exit_3)
-	test	$0x02, %al
-	jnz	L(exit_2)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_8):
-	test	$0x80, %al
-	jnz	L(exit_8)
-	test	$0x40, %al
-	jnz	L(exit_7)
-	test	$0x20, %al
-	jnz	L(exit_6)
-	lea	4(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_high):
-	mov	%ah, %dh
-	and	$15 << 4, %dh
-	jnz	L(exit_dispatch_high_8)
-	test	$0x08, %ah
-	jnz	L(exit_12)
-	test	$0x04, %ah
-	jnz	L(exit_11)
-	test	$0x02, %ah
-	jnz	L(exit_10)
-	lea	8(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_high_8):
-	test	$0x80, %ah
-	jnz	L(exit_16)
-	test	$0x40, %ah
-	jnz	L(exit_15)
-	test	$0x20, %ah
-	jnz	L(exit_14)
-	lea	12(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_2):
-	lea	1(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_3):
-	lea	2(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_4):
-	lea	3(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_6):
-	lea	5(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_7):
-	lea	6(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_8):
-	lea	7(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_10):
-	lea	9(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_11):
-	lea	10(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_12):
-	lea	11(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_14):
-	lea	13(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_15):
-	lea	14(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_16):
-	lea	15(%ecx), %eax
-	ret
-
-	.p2align 4
-L(matches0_1):
-	lea	-64(%edx), %edx
-
-	test	%ah, %ah
-	jnz	L(exit_dispatch_1_high)
-	mov	%al, %ah
-	and	$15 << 4, %ah
-	jnz	L(exit_dispatch_1_8)
-	test	$0x08, %al
-	jnz	L(exit_1_4)
-	test	$0x04, %al
-	jnz	L(exit_1_3)
-	test	$0x02, %al
-	jnz	L(exit_1_2)
-
-	add	$0, %edx
-	jl	L(return_null)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(matches16_1):
-	lea	-48(%edx), %edx
-	lea	16(%ecx), %ecx
-
-	test	%ah, %ah
-	jnz	L(exit_dispatch_1_high)
-	mov	%al, %ah
-	and	$15 << 4, %ah
-	jnz	L(exit_dispatch_1_8)
-	test	$0x08, %al
-	jnz	L(exit_1_4)
-	test	$0x04, %al
-	jnz	L(exit_1_3)
-	test	$0x02, %al
-	jnz	L(exit_1_2)
-
-	add	$0, %edx
-	jl	L(return_null)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(matches32_1):
-	lea	-32(%edx), %edx
-	lea	32(%ecx), %ecx
-
-	test	%ah, %ah
-	jnz	L(exit_dispatch_1_high)
-	mov	%al, %ah
-	and	$15 << 4, %ah
-	jnz	L(exit_dispatch_1_8)
-	test	$0x08, %al
-	jnz	L(exit_1_4)
-	test	$0x04, %al
-	jnz	L(exit_1_3)
-	test	$0x02, %al
-	jnz	L(exit_1_2)
-
-	add	$0, %edx
-	jl	L(return_null)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(matches48_1):
-	lea	-16(%edx), %edx
-	lea	48(%ecx), %ecx
-
-	.p2align 4
-L(exit_dispatch_1):
-	test	%ah, %ah
-	jnz	L(exit_dispatch_1_high)
-	mov	%al, %ah
-	and	$15 << 4, %ah
-	jnz	L(exit_dispatch_1_8)
-	test	$0x08, %al
-	jnz	L(exit_1_4)
-	test	$0x04, %al
-	jnz	L(exit_1_3)
-	test	$0x02, %al
-	jnz	L(exit_1_2)
-
-	add	$0, %edx
-	jl	L(return_null)
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_1_8):
-	test	$0x80, %al
-	jnz	L(exit_1_8)
-	test	$0x40, %al
-	jnz	L(exit_1_7)
-	test	$0x20, %al
-	jnz	L(exit_1_6)
-
-	add	$4, %edx
-	jl	L(return_null)
-	lea	4(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_1_high):
-	mov	%ah, %al
-	and	$15 << 4, %al
-	jnz	L(exit_dispatch_1_high_8)
-	test	$0x08, %ah
-	jnz	L(exit_1_12)
-	test	$0x04, %ah
-	jnz	L(exit_1_11)
-	test	$0x02, %ah
-	jnz	L(exit_1_10)
-
-	add	$8, %edx
-	jl	L(return_null)
-	lea	8(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_dispatch_1_high_8):
-	test	$0x80, %ah
-	jnz	L(exit_1_16)
-	test	$0x40, %ah
-	jnz	L(exit_1_15)
-	test	$0x20, %ah
-	jnz	L(exit_1_14)
-
-	add	$12, %edx
-	jl	L(return_null)
-	lea	12(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_2):
-	add	$1, %edx
-	jl	L(return_null)
-	lea	1(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_3):
-	add	$2, %edx
-	jl	L(return_null)
-	lea	2(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_4):
-	add	$3, %edx
-	jl	L(return_null)
-	lea	3(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_6):
-	add	$5, %edx
-	jl	L(return_null)
-	lea	5(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_7):
-	add	$6, %edx
-	jl	L(return_null)
-	lea	6(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_8):
-	add	$7, %edx
-	jl	L(return_null)
-	lea	7(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_10):
-	add	$9, %edx
-	jl	L(return_null)
-	lea	9(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_11):
-	add	$10, %edx
-	jl	L(return_null)
-	lea	10(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_12):
-	add	$11, %edx
-	jl	L(return_null)
-	lea	11(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_14):
-	add	$13, %edx
-	jl	L(return_null)
-	lea	13(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_15):
-	add	$14, %edx
-	jl	L(return_null)
-	lea	14(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit_1_16):
-	add	$15, %edx
-	jl	L(return_null)
-	lea	15(%ecx), %eax
-	ret
-
-	.p2align 4
-L(return_null):
-	xor	%eax, %eax
-	ret
-
-	.p2align 4
-L(length_less16_offset0):
-	mov	%dl, %cl
-	pcmpeqb	(%eax), %xmm1
-
-	mov	$1, %edx
-	sal	%cl, %edx
-	sub	$1, %edx
-
-	mov	%eax, %ecx
-	pmovmskb %xmm1, %eax
-
-	and	%edx, %eax
-	test	%eax, %eax
-	jnz	L(exit_dispatch)
-
-	xor	%eax, %eax
-	ret
-
-	.p2align 4
-L(length_less16):
-	punpcklbw %xmm1, %xmm1
-	add	$16, %edx
-	punpcklbw %xmm1, %xmm1
-
-	mov	%ecx, %eax
-	pshufd	$0, %xmm1, %xmm1
-
-	and	$15, %ecx
-	jz	L(length_less16_offset0)
-
-	PUSH	(%edi)
-
-	mov	%cl, %dh
-	add	%dl, %dh
-	and	$-16, %eax
-
-	sub	$16, %dh
-	ja	L(length_less16_part2)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edi
-
-	sar	%cl, %edi
-	add	%ecx, %eax
-	mov	%dl, %cl
-
-	mov	$1, %edx
-	sal	%cl, %edx
-	sub	$1, %edx
-
-	and	%edx, %edi
-	test	%edi, %edi
-	jz	L(ret_null)
-
-	bsr	%edi, %edi
-	add	%edi, %eax
-	POP	(%edi)
-	ret
-
-	CFI_PUSH     (%edi)
-
-	.p2align 4
-L(length_less16_part2):
-	movdqa	16(%eax), %xmm2
-	pcmpeqb	%xmm1, %xmm2
-	pmovmskb %xmm2, %edi
-
-	mov	%cl, %ch
-
-	mov	%dh, %cl
-	mov	$1, %edx
-	sal	%cl, %edx
-	sub	$1, %edx
-
-	and	%edx, %edi
-
-	test	%edi, %edi
-	jnz	L(length_less16_part2_return)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edi
-
-	mov	%ch, %cl
-	sar	%cl, %edi
-	test	%edi, %edi
-	jz	L(ret_null)
-
-	bsr	%edi, %edi
-	add	%edi, %eax
-	xor	%ch, %ch
-	add	%ecx, %eax
-	POP	(%edi)
-	ret
-
-	CFI_PUSH     (%edi)
-
-	.p2align 4
-L(length_less16_part2_return):
-	bsr	%edi, %edi
-	lea	16(%eax, %edi), %eax
-	POP	(%edi)
-	ret
-
-	CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ret_null):
-	xor	%eax, %eax
-	POP	(%edi)
-	ret
-
-END (memrchr)
diff --git a/libc/arch-x86/string/sse2-wcschr-atom.S b/libc/arch-x86/string/sse2-wcschr-atom.S
deleted file mode 100644
index 729302b..0000000
--- a/libc/arch-x86/string/sse2-wcschr-atom.S
+++ /dev/null
@@ -1,267 +0,0 @@
-/*
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef L
-# define L(label)	.L##label
-#endif
-
-#ifndef cfi_startproc
-# define cfi_startproc	.cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc	.cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name,  @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name,	.-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG)	pushl REG; CFI_PUSH (REG)
-#define POP(REG)	popl REG; CFI_POP (REG)
-
-#define PARMS	4
-
-
-#define STR1  PARMS
-#define STR2  STR1+4
-
-	.text
-ENTRY (wcschr)
-
-	mov	STR1(%esp), %ecx
-	movd	STR2(%esp), %xmm1
-
-	mov	%ecx, %eax
-	punpckldq %xmm1, %xmm1
-	pxor	%xmm2, %xmm2
-	punpckldq %xmm1, %xmm1
-
-	and	$63, %eax
-	cmp	$48, %eax
-	ja	L(cross_cache)
-
-	movdqu	(%ecx), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-	or	%eax, %edx
-	jnz	L(matches)
-	and	$-16, %ecx
-	jmp	L(loop)
-
-	.p2align 4
-L(cross_cache):
-	PUSH	(%edi)
-	mov	%ecx, %edi
-	mov	%eax, %ecx
-	and	$-16, %edi
-	and	$15, %ecx
-	movdqa	(%edi), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-
-	sarl	%cl, %edx
-	sarl	%cl, %eax
-	test	%eax, %eax
-	jz	L(unaligned_no_match)
-
-	add	%edi, %ecx
-	POP	(%edi)
-
-	test	%edx, %edx
-	jz	L(match_case1)
-	test	%al, %al
-	jz	L(match_higth_case2)
-	test	$15, %al
-	jnz	L(match_case2_4)
-	test	$15, %dl
-	jnz	L(return_null)
-	lea	4(%ecx), %eax
-	ret
-
-	CFI_PUSH (%edi)
-
-	.p2align 4
-L(unaligned_no_match):
-	mov	%edi, %ecx
-	POP	(%edi)
-
-	test	%edx, %edx
-	jnz	L(return_null)
-
-	pxor	%xmm2, %xmm2
-
-/* Loop start on aligned string.  */
-	.p2align 4
-L(loop):
-	add	$16, %ecx
-	movdqa	(%ecx), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-	or	%eax, %edx
-	jnz	L(matches)
-	add	$16, %ecx
-
-	movdqa	(%ecx), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-	or	%eax, %edx
-	jnz	L(matches)
-	add	$16, %ecx
-
-	movdqa	(%ecx), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-	or	%eax, %edx
-	jnz	L(matches)
-	add	$16, %ecx
-
-	movdqa	(%ecx), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %edx
-	pmovmskb %xmm0, %eax
-	or	%eax, %edx
-	jz	L(loop)
-
-	.p2align 4
-L(matches):
-	pmovmskb %xmm2, %edx
-	test	%eax, %eax
-	jz	L(return_null)
-	test	%edx, %edx
-	jz	L(match_case1)
-
-	.p2align 4
-L(match_case2):
-	test	%al, %al
-	jz	L(match_higth_case2)
-	test	$15, %al
-	jnz	L(match_case2_4)
-	test	$15, %dl
-	jnz	L(return_null)
-	lea	4(%ecx), %eax
-	ret
-
-	.p2align 4
-L(match_case2_4):
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(match_higth_case2):
-	test	%dl, %dl
-	jnz	L(return_null)
-	test	$15, %ah
-	jnz	L(match_case2_12)
-	test	$15, %dh
-	jnz	L(return_null)
-	lea	12(%ecx), %eax
-	ret
-
-	.p2align 4
-L(match_case2_12):
-	lea	8(%ecx), %eax
-	ret
-
-	.p2align 4
-L(match_case1):
-	test	%al, %al
-	jz	L(match_higth_case1)
-
-	test	$0x01, %al
-	jnz	L(exit0)
-	lea	4(%ecx), %eax
-	ret
-
-	.p2align 4
-L(match_higth_case1):
-	test	$0x01, %ah
-	jnz	L(exit3)
-	lea	12(%ecx), %eax
-	ret
-
-	.p2align 4
-L(exit0):
-	mov	%ecx, %eax
-	ret
-
-	.p2align 4
-L(exit3):
-	lea	8(%ecx), %eax
-	ret
-
-	.p2align 4
-L(return_null):
-	xor	%eax, %eax
-	ret
-
-END (wcschr)
diff --git a/libc/arch-x86/string/sse2-wcscmp-atom.S b/libc/arch-x86/string/sse2-wcscmp-atom.S
deleted file mode 100644
index 8867d28..0000000
--- a/libc/arch-x86/string/sse2-wcscmp-atom.S
+++ /dev/null
@@ -1,1062 +0,0 @@
-/*
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef L
-# define L(label)	.L##label
-#endif
-
-#ifndef cfi_startproc
-# define cfi_startproc	.cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc	.cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name, @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG) pushl REG; CFI_PUSH (REG)
-#define POP(REG) popl REG; CFI_POP (REG)
-
-#define ENTRANCE PUSH(%esi); PUSH(%edi)
-#define RETURN  POP(%edi); POP(%esi); ret; CFI_PUSH(%esi); CFI_PUSH(%edi);
-#define PARMS  4
-#define STR1  PARMS
-#define STR2  STR1+4
-
-	.text
-ENTRY (wcscmp)
-/*
-	* This implementation uses SSE to compare up to 16 bytes at a time.
-*/
-	mov	STR1(%esp), %edx
-	mov	STR2(%esp), %eax
-
-	mov	(%eax), %ecx
-	cmp	%ecx, (%edx)
-	jne	L(neq)
-	test	%ecx, %ecx
-	jz	L(eq)
-
-	mov	4(%eax), %ecx
-	cmp	%ecx, 4(%edx)
-	jne	L(neq)
-	test	%ecx, %ecx
-	jz	L(eq)
-
-	mov	8(%eax), %ecx
-	cmp	%ecx, 8(%edx)
-	jne	L(neq)
-	test	%ecx, %ecx
-	jz	L(eq)
-
-	mov	12(%eax), %ecx
-	cmp	%ecx, 12(%edx)
-	jne	L(neq)
-	test	%ecx, %ecx
-	jz	L(eq)
-
-	ENTRANCE
-	add	$16, %eax
-	add	$16, %edx
-
-	mov	%eax, %esi
-	mov	%edx, %edi
-	pxor	%xmm0, %xmm0		/* clear %xmm0 for null char checks */
-	mov	%al, %ch
-	mov	%dl, %cl
-	and	$63, %eax		/* esi alignment in cache line */
-	and	$63, %edx		/* edi alignment in cache line */
-	and	$15, %cl
-	jz	L(continue_00)
-	cmp	$16, %edx
-	jb	L(continue_0)
-	cmp	$32, %edx
-	jb	L(continue_16)
-	cmp	$48, %edx
-	jb	L(continue_32)
-
-L(continue_48):
-	and	$15, %ch
-	jz	L(continue_48_00)
-	cmp	$16, %eax
-	jb	L(continue_0_48)
-	cmp	$32, %eax
-	jb	L(continue_16_48)
-	cmp	$48, %eax
-	jb	L(continue_32_48)
-
-	.p2align 4
-L(continue_48_48):
-	mov	(%esi), %ecx
-	cmp	%ecx, (%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	movdqu	16(%edi), %xmm1
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%edi), %xmm1
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	movdqu	48(%edi), %xmm1
-	movdqu	48(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_48_48)
-
-L(continue_0):
-	and	$15, %ch
-	jz	L(continue_0_00)
-	cmp	$16, %eax
-	jb	L(continue_0_0)
-	cmp	$32, %eax
-	jb	L(continue_0_16)
-	cmp	$48, %eax
-	jb	L(continue_0_32)
-
-	.p2align 4
-L(continue_0_48):
-	mov	(%esi), %ecx
-	cmp	%ecx, (%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	movdqu	16(%edi), %xmm1
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%edi), %xmm1
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	mov	48(%esi), %ecx
-	cmp	%ecx, 48(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	52(%esi), %ecx
-	cmp	%ecx, 52(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	56(%esi), %ecx
-	cmp	%ecx, 56(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	60(%esi), %ecx
-	cmp	%ecx, 60(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_0_48)
-
-	.p2align 4
-L(continue_00):
-	and	$15, %ch
-	jz	L(continue_00_00)
-	cmp	$16, %eax
-	jb	L(continue_00_0)
-	cmp	$32, %eax
-	jb	L(continue_00_16)
-	cmp	$48, %eax
-	jb	L(continue_00_32)
-
-	.p2align 4
-L(continue_00_48):
-	pcmpeqd	(%edi), %xmm0
-	mov	(%edi), %eax
-	pmovmskb %xmm0, %ecx
-	test	%ecx, %ecx
-	jnz	L(less4_double_words1)
-
-	cmp	(%esi), %eax
-	jne	L(nequal)
-
-	mov	4(%edi), %eax
-	cmp	4(%esi), %eax
-	jne	L(nequal)
-
-	mov	8(%edi), %eax
-	cmp	8(%esi), %eax
-	jne	L(nequal)
-
-	mov	12(%edi), %eax
-	cmp	12(%esi), %eax
-	jne	L(nequal)
-
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	32(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	movdqu	48(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	48(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_00_48)
-
-	.p2align 4
-L(continue_32):
-	and	$15, %ch
-	jz	L(continue_32_00)
-	cmp	$16, %eax
-	jb	L(continue_0_32)
-	cmp	$32, %eax
-	jb	L(continue_16_32)
-	cmp	$48, %eax
-	jb	L(continue_32_32)
-
-	.p2align 4
-L(continue_32_48):
-	mov	(%esi), %ecx
-	cmp	%ecx, (%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	16(%esi), %ecx
-	cmp	%ecx, 16(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	20(%esi), %ecx
-	cmp	%ecx, 20(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	24(%esi), %ecx
-	cmp	%ecx, 24(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	28(%esi), %ecx
-	cmp	%ecx, 28(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	movdqu	32(%edi), %xmm1
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	movdqu	48(%edi), %xmm1
-	movdqu	48(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_32_48)
-
-	.p2align 4
-L(continue_16):
-	and	$15, %ch
-	jz	L(continue_16_00)
-	cmp	$16, %eax
-	jb	L(continue_0_16)
-	cmp	$32, %eax
-	jb	L(continue_16_16)
-	cmp	$48, %eax
-	jb	L(continue_16_32)
-
-	.p2align 4
-L(continue_16_48):
-	mov	(%esi), %ecx
-	cmp	%ecx, (%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	movdqu	16(%edi), %xmm1
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	mov	32(%esi), %ecx
-	cmp	%ecx, 32(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	36(%esi), %ecx
-	cmp	%ecx, 36(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	40(%esi), %ecx
-	cmp	%ecx, 40(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	44(%esi), %ecx
-	cmp	%ecx, 44(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	movdqu	48(%edi), %xmm1
-	movdqu	48(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_16_48)
-
-	.p2align 4
-L(continue_00_00):
-	movdqa	(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqa	16(%edi), %xmm3
-	pcmpeqd	%xmm3, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%esi), %xmm3		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm3		/* packed sub of comparison results*/
-	pmovmskb %xmm3, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqa	32(%edi), %xmm5
-	pcmpeqd	%xmm5, %xmm0		/* Any null double_word? */
-	pcmpeqd	32(%esi), %xmm5		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm5		/* packed sub of comparison results*/
-	pmovmskb %xmm5, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	movdqa	48(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	48(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_00_00)
-
-	.p2align 4
-L(continue_00_32):
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	add	$16, %esi
-	add	$16, %edi
-	jmp	L(continue_00_48)
-
-	.p2align 4
-L(continue_00_16):
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	add	$32, %esi
-	add	$32, %edi
-	jmp	L(continue_00_48)
-
-	.p2align 4
-L(continue_00_0):
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm2, %xmm0		/* Any null double_word? */
-	pcmpeqd	32(%edi), %xmm2		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm2		/* packed sub of comparison results*/
-	pmovmskb %xmm2, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	add	$48, %esi
-	add	$48, %edi
-	jmp	L(continue_00_48)
-
-	.p2align 4
-L(continue_48_00):
-	pcmpeqd	(%esi), %xmm0
-	mov	(%edi), %eax
-	pmovmskb %xmm0, %ecx
-	test	%ecx, %ecx
-	jnz	L(less4_double_words1)
-
-	cmp	(%esi), %eax
-	jne	L(nequal)
-
-	mov	4(%edi), %eax
-	cmp	4(%esi), %eax
-	jne	L(nequal)
-
-	mov	8(%edi), %eax
-	cmp	8(%esi), %eax
-	jne	L(nequal)
-
-	mov	12(%edi), %eax
-	cmp	12(%esi), %eax
-	jne	L(nequal)
-
-	movdqu	16(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	32(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	movdqu	48(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	48(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_48)
-
-	add	$64, %esi
-	add	$64, %edi
-	jmp	L(continue_48_00)
-
-	.p2align 4
-L(continue_32_00):
-	movdqu	(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	add	$16, %esi
-	add	$16, %edi
-	jmp	L(continue_48_00)
-
-	.p2align 4
-L(continue_16_00):
-	movdqu	(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	add	$32, %esi
-	add	$32, %edi
-	jmp	L(continue_48_00)
-
-	.p2align 4
-L(continue_0_00):
-	movdqu	(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	16(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%edi), %xmm1
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	32(%esi), %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	add	$48, %esi
-	add	$48, %edi
-	jmp	L(continue_48_00)
-
-	.p2align 4
-L(continue_32_32):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	add	$16, %esi
-	add	$16, %edi
-	jmp	L(continue_48_48)
-
-	.p2align 4
-L(continue_16_16):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%edi), %xmm3
-	movdqu	16(%esi), %xmm4
-	pcmpeqd	%xmm3, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm4, %xmm3		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm3		/* packed sub of comparison results*/
-	pmovmskb %xmm3, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	add	$32, %esi
-	add	$32, %edi
-	jmp	L(continue_48_48)
-
-	.p2align 4
-L(continue_0_0):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%edi), %xmm3
-	movdqu	16(%esi), %xmm4
-	pcmpeqd	%xmm3, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm4, %xmm3		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm3		/* packed sub of comparison results*/
-	pmovmskb %xmm3, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	movdqu	32(%edi), %xmm1
-	movdqu	32(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_32)
-
-	add	$48, %esi
-	add	$48, %edi
-	jmp	L(continue_48_48)
-
-	.p2align 4
-L(continue_0_16):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	movdqu	16(%edi), %xmm1
-	movdqu	16(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words_16)
-
-	add	$32, %esi
-	add	$32, %edi
-	jmp	L(continue_32_48)
-
-	.p2align 4
-L(continue_0_32):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	add	$16, %esi
-	add	$16, %edi
-	jmp	L(continue_16_48)
-
-	.p2align 4
-L(continue_16_32):
-	movdqu	(%edi), %xmm1
-	movdqu	(%esi), %xmm2
-	pcmpeqd	%xmm1, %xmm0		/* Any null double_word? */
-	pcmpeqd	%xmm2, %xmm1		/* compare first 4 double_words for equality */
-	psubb	%xmm0, %xmm1		/* packed sub of comparison results*/
-	pmovmskb %xmm1, %edx
-	sub	$0xffff, %edx		/* if first 4 double_words are same, edx == 0xffff */
-	jnz	L(less4_double_words)
-
-	add	$16, %esi
-	add	$16, %edi
-	jmp	L(continue_32_48)
-
-	.p2align 4
-L(less4_double_words1):
-	cmp	(%esi), %eax
-	jne	L(nequal)
-	test	%eax, %eax
-	jz	L(equal)
-
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	test	%ecx, %ecx
-	jz	L(equal)
-
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	xor	%eax, %eax
-	RETURN
-
-	.p2align 4
-L(less4_double_words):
-	xor	%eax, %eax
-	test	%dl, %dl
-	jz	L(next_two_double_words)
-	and	$15, %dl
-	jz	L(second_double_word)
-	mov	(%esi), %ecx
-	cmp	%ecx, (%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(second_double_word):
-	mov	4(%esi), %ecx
-	cmp	%ecx, 4(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(next_two_double_words):
-	and	$15, %dh
-	jz	L(fourth_double_word)
-	mov	8(%esi), %ecx
-	cmp	%ecx, 8(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(fourth_double_word):
-	mov	12(%esi), %ecx
-	cmp	%ecx, 12(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(less4_double_words_16):
-	xor	%eax, %eax
-	test	%dl, %dl
-	jz	L(next_two_double_words_16)
-	and	$15, %dl
-	jz	L(second_double_word_16)
-	mov	16(%esi), %ecx
-	cmp	%ecx, 16(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(second_double_word_16):
-	mov	20(%esi), %ecx
-	cmp	%ecx, 20(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(next_two_double_words_16):
-	and	$15, %dh
-	jz	L(fourth_double_word_16)
-	mov	24(%esi), %ecx
-	cmp	%ecx, 24(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(fourth_double_word_16):
-	mov	28(%esi), %ecx
-	cmp	%ecx, 28(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(less4_double_words_32):
-	xor	%eax, %eax
-	test	%dl, %dl
-	jz	L(next_two_double_words_32)
-	and	$15, %dl
-	jz	L(second_double_word_32)
-	mov	32(%esi), %ecx
-	cmp	%ecx, 32(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(second_double_word_32):
-	mov	36(%esi), %ecx
-	cmp	%ecx, 36(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(next_two_double_words_32):
-	and	$15, %dh
-	jz	L(fourth_double_word_32)
-	mov	40(%esi), %ecx
-	cmp	%ecx, 40(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(fourth_double_word_32):
-	mov	44(%esi), %ecx
-	cmp	%ecx, 44(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(less4_double_words_48):
-	xor	%eax, %eax
-	test	%dl, %dl
-	jz	L(next_two_double_words_48)
-	and	$15, %dl
-	jz	L(second_double_word_48)
-	mov	48(%esi), %ecx
-	cmp	%ecx, 48(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(second_double_word_48):
-	mov	52(%esi), %ecx
-	cmp	%ecx, 52(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(next_two_double_words_48):
-	and	$15, %dh
-	jz	L(fourth_double_word_48)
-	mov	56(%esi), %ecx
-	cmp	%ecx, 56(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(fourth_double_word_48):
-	mov	60(%esi), %ecx
-	cmp	%ecx, 60(%edi)
-	jne	L(nequal)
-	RETURN
-
-	.p2align 4
-L(nequal):
-	mov	$1, %eax
-	jg	L(return)
-	neg	%eax
-	RETURN
-
-	.p2align 4
-L(return):
-	RETURN
-
-	.p2align 4
-L(equal):
-	xorl	%eax, %eax
-	RETURN
-
-	CFI_POP (%edi)
-	CFI_POP (%esi)
-
-	.p2align 4
-L(neq):
-	mov	$1, %eax
-	jg	L(neq_bigger)
-	neg	%eax
-
-L(neq_bigger):
-	ret
-
-	.p2align 4
-L(eq):
-	xorl	%eax, %eax
-	ret
-
-END (wcscmp)
-
diff --git a/libc/arch-x86/string/sse2-wcslen-atom.S b/libc/arch-x86/string/sse2-wcslen-atom.S
deleted file mode 100644
index 2f10db4..0000000
--- a/libc/arch-x86/string/sse2-wcslen-atom.S
+++ /dev/null
@@ -1,306 +0,0 @@
-/*
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef USE_AS_WCSCAT
-
-# ifndef L
-#  define L(label)	.L##label
-# endif
-
-# ifndef cfi_startproc
-#  define cfi_startproc	.cfi_startproc
-# endif
-
-# ifndef cfi_endproc
-#  define cfi_endproc	.cfi_endproc
-# endif
-
-# ifndef ENTRY
-#  define ENTRY(name)	\
-	.type name,  @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-# endif
-
-# ifndef END
-#  define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-# endif
-
-# define PARMS	4
-# define STR	PARMS
-# define RETURN ret
-
-	.text
-ENTRY (wcslen)
-	mov	STR(%esp), %edx
-#endif
-	cmpl	$0, (%edx)
-	jz	L(exit_tail0)
-	cmpl	$0, 4(%edx)
-	jz	L(exit_tail1)
-	cmpl	$0, 8(%edx)
-	jz	L(exit_tail2)
-	cmpl	$0, 12(%edx)
-	jz	L(exit_tail3)
-	cmpl	$0, 16(%edx)
-	jz	L(exit_tail4)
-	cmpl	$0, 20(%edx)
-	jz	L(exit_tail5)
-	cmpl	$0, 24(%edx)
-	jz	L(exit_tail6)
-	cmpl	$0, 28(%edx)
-	jz	L(exit_tail7)
-
-	pxor	%xmm0, %xmm0
-
-	lea	32(%edx), %eax
-	lea	-16(%eax), %ecx
-	and	$-16, %eax
-
-	pcmpeqd	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	pxor	%xmm1, %xmm1
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	pxor	%xmm2, %xmm2
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	pxor	%xmm3, %xmm3
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	and	$-0x40, %eax
-
-	.p2align 4
-L(aligned_64_loop):
-	movaps	(%eax), %xmm0
-	movaps	16(%eax), %xmm1
-	movaps	32(%eax), %xmm2
-	movaps	48(%eax), %xmm6
-
-	pminub	%xmm1, %xmm0
-	pminub	%xmm6, %xmm2
-	pminub	%xmm0, %xmm2
-	pcmpeqd	%xmm3, %xmm2
-	pmovmskb %xmm2, %edx
-	lea	64(%eax), %eax
-	test	%edx, %edx
-	jz	L(aligned_64_loop)
-
-	pcmpeqd	-64(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	48(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	%xmm1, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	-32(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqd	%xmm6, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	jmp	L(aligned_64_loop)
-
-	.p2align 4
-L(exit):
-	sub	%ecx, %eax
-	shr	$2, %eax
-	test	%dl, %dl
-	jz	L(exit_high)
-
-	mov	%dl, %cl
-	and	$15, %cl
-	jz	L(exit_1)
-	RETURN
-
-	.p2align 4
-L(exit_high):
-	mov	%dh, %ch
-	and	$15, %ch
-	jz	L(exit_3)
-	add	$2, %eax
-	RETURN
-
-	.p2align 4
-L(exit_1):
-	add	$1, %eax
-	RETURN
-
-	.p2align 4
-L(exit_3):
-	add	$3, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail0):
-	xor	%eax, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail1):
-	mov	$1, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail2):
-	mov	$2, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail3):
-	mov	$3, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail4):
-	mov	$4, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail5):
-	mov	$5, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail6):
-	mov	$6, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail7):
-	mov	$7, %eax
-#ifndef USE_AS_WCSCAT
-	RETURN
-
-END (wcslen)
-#endif
diff --git a/libc/arch-x86/string/sse2-wcsrchr-atom.S b/libc/arch-x86/string/sse2-wcsrchr-atom.S
deleted file mode 100644
index 1a55df2..0000000
--- a/libc/arch-x86/string/sse2-wcsrchr-atom.S
+++ /dev/null
@@ -1,402 +0,0 @@
-/*
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef L
-# define L(label)	.L##label
-#endif
-
-#ifndef cfi_startproc
-# define cfi_startproc	.cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc	.cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name, @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG)	pushl REG;	CFI_PUSH (REG)
-#define POP(REG)	popl REG;	CFI_POP (REG)
-
-#define PARMS  8
-#define ENTRANCE PUSH(%edi);
-#define RETURN  POP(%edi);	ret;	CFI_PUSH(%edi);
-
-#define STR1  PARMS
-#define STR2  STR1+4
-
-	.text
-ENTRY (wcsrchr)
-
-	ENTRANCE
-	mov	STR1(%esp), %ecx
-	movd	STR2(%esp), %xmm1
-
-	mov	%ecx, %edi
-	punpckldq %xmm1, %xmm1
-	pxor	%xmm2, %xmm2
-	punpckldq %xmm1, %xmm1
-
-/* ECX has OFFSET. */
-	and	$63, %ecx
-	cmp	$48, %ecx
-	ja	L(crosscache)
-
-/* unaligned string. */
-	movdqu	(%edi), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	pcmpeqd	%xmm1, %xmm0
-/* Find where NULL is.  */
-	pmovmskb %xmm2, %ecx
-/* Check if there is a match.  */
-	pmovmskb %xmm0, %eax
-	add	$16, %edi
-
-	test	%eax, %eax
-	jnz	L(unaligned_match1)
-
-	test	%ecx, %ecx
-	jnz	L(return_null)
-
-	and	$-16, %edi
-
-	PUSH	(%esi)
-
-	xor	%edx, %edx
-	jmp	L(loop)
-
-	CFI_POP	(%esi)
-
-	.p2align 4
-L(unaligned_match1):
-	test	%ecx, %ecx
-	jnz	L(prolog_find_zero_1)
-
-	PUSH	(%esi)
-
-/* Save current match */
-	mov	%eax, %edx
-	mov	%edi, %esi
-	and	$-16, %edi
-	jmp	L(loop)
-
-	CFI_POP	(%esi)
-
-	.p2align 4
-L(crosscache):
-/* Hancle unaligned string.  */
-	and	$15, %ecx
-	and	$-16, %edi
-	pxor	%xmm3, %xmm3
-	movdqa	(%edi), %xmm0
-	pcmpeqd	%xmm0, %xmm3
-	pcmpeqd	%xmm1, %xmm0
-/* Find where NULL is.  */
-	pmovmskb %xmm3, %edx
-/* Check if there is a match.  */
-	pmovmskb %xmm0, %eax
-/* Remove the leading bytes.  */
-	shr	%cl, %edx
-	shr	%cl, %eax
-	add	$16, %edi
-
-	test	%eax, %eax
-	jnz	L(unaligned_match)
-
-	test	%edx, %edx
-	jnz	L(return_null)
-
-	PUSH	(%esi)
-
-	xor	%edx, %edx
-	jmp	L(loop)
-
-	CFI_POP	(%esi)
-
-	.p2align 4
-L(unaligned_match):
-	test	%edx, %edx
-	jnz	L(prolog_find_zero)
-
-	PUSH	(%esi)
-
-	mov	%eax, %edx
-	lea	(%edi, %ecx), %esi
-
-/* Loop start on aligned string.  */
-	.p2align 4
-L(loop):
-	movdqa	(%edi), %xmm0
-	pcmpeqd	%xmm0, %xmm2
-	add	$16, %edi
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm2, %ecx
-	pmovmskb %xmm0, %eax
-	or	%eax, %ecx
-	jnz	L(matches)
-
-	movdqa	(%edi), %xmm3
-	pcmpeqd	%xmm3, %xmm2
-	add	$16, %edi
-	pcmpeqd	%xmm1, %xmm3
-	pmovmskb %xmm2, %ecx
-	pmovmskb %xmm3, %eax
-	or	%eax, %ecx
-	jnz	L(matches)
-
-	movdqa	(%edi), %xmm4
-	pcmpeqd	%xmm4, %xmm2
-	add	$16, %edi
-	pcmpeqd	%xmm1, %xmm4
-	pmovmskb %xmm2, %ecx
-	pmovmskb %xmm4, %eax
-	or	%eax, %ecx
-	jnz	L(matches)
-
-	movdqa	(%edi), %xmm5
-	pcmpeqd	%xmm5, %xmm2
-	add	$16, %edi
-	pcmpeqd	%xmm1, %xmm5
-	pmovmskb %xmm2, %ecx
-	pmovmskb %xmm5, %eax
-	or	%eax, %ecx
-	jz	L(loop)
-
-	.p2align 4
-L(matches):
-	test	%eax, %eax
-	jnz	L(match)
-L(return_value):
-	test	%edx, %edx
-	jz	L(return_null_1)
-	mov	%edx, %eax
-	mov	%esi, %edi
-
-	POP	(%esi)
-
-	test	%ah, %ah
-	jnz	L(match_third_or_fourth_wchar)
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(return_null_1):
-	POP	(%esi)
-
-	xor	%eax, %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(match):
-	pmovmskb %xmm2, %ecx
-	test	%ecx, %ecx
-	jnz	L(find_zero)
-/* save match info */
-	mov	%eax, %edx
-	mov	%edi, %esi
-	jmp	L(loop)
-
-	.p2align 4
-L(find_zero):
-	test	%cl, %cl
-	jz	L(find_zero_in_third_or_fourth_wchar)
-	test	$15, %cl
-	jz	L(find_zero_in_second_wchar)
-	and	$1, %eax
-	jz	L(return_value)
-
-	POP	(%esi)
-
-	lea	-16(%edi), %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(find_zero_in_second_wchar):
-	and	$(1 << 5) - 1, %eax
-	jz	L(return_value)
-
-	POP	(%esi)
-
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(find_zero_in_third_or_fourth_wchar):
-	test	$15, %ch
-	jz	L(find_zero_in_fourth_wchar)
-	and	$(1 << 9) - 1, %eax
-	jz	L(return_value)
-
-	POP	(%esi)
-
-	test	%ah, %ah
-	jnz	L(match_third_wchar)
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(find_zero_in_fourth_wchar):
-
-	POP	(%esi)
-
-	test	%ah, %ah
-	jnz	L(match_third_or_fourth_wchar)
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(match_second_wchar):
-	lea	-12(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(match_third_or_fourth_wchar):
-	test	$15 << 4, %ah
-	jnz	L(match_fourth_wchar)
-	lea	-8(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(match_third_wchar):
-	lea	-8(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(match_fourth_wchar):
-	lea	-4(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(return_null):
-	xor	%eax, %eax
-	RETURN
-
-	.p2align 4
-L(prolog_find_zero):
-	add	%ecx, %edi
-	mov     %edx, %ecx
-L(prolog_find_zero_1):
-	test	%cl, %cl
-	jz	L(prolog_find_zero_in_third_or_fourth_wchar)
-	test	$15, %cl
-	jz	L(prolog_find_zero_in_second_wchar)
-	and	$1, %eax
-	jz	L(return_null)
-
-	lea	-16(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(prolog_find_zero_in_second_wchar):
-	and	$(1 << 5) - 1, %eax
-	jz	L(return_null)
-
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(prolog_find_zero_in_third_or_fourth_wchar):
-	test	$15, %ch
-	jz	L(prolog_find_zero_in_fourth_wchar)
-	and	$(1 << 9) - 1, %eax
-	jz	L(return_null)
-
-	test	%ah, %ah
-	jnz	L(match_third_wchar)
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-	.p2align 4
-L(prolog_find_zero_in_fourth_wchar):
-	test	%ah, %ah
-	jnz	L(match_third_or_fourth_wchar)
-	test	$15 << 4, %al
-	jnz	L(match_second_wchar)
-	lea	-16(%edi), %eax
-	RETURN
-
-END (wcsrchr)
diff --git a/libc/arch-x86/string/sse4-wmemcmp-slm.S b/libc/arch-x86/string/sse4-wmemcmp-slm.S
deleted file mode 100644
index 2bf92f5..0000000
--- a/libc/arch-x86/string/sse4-wmemcmp-slm.S
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
-Copyright (c) 2014, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#define USE_AS_WMEMCMP
-#define MEMCMP wmemcmp_sse4
-#include "sse4-memcmp-slm.S"
diff --git a/libc/arch-x86/string/ssse3-strlcat-atom.S b/libc/arch-x86/string/ssse3-strlcat-atom.S
deleted file mode 100644
index daaf254..0000000
--- a/libc/arch-x86/string/ssse3-strlcat-atom.S
+++ /dev/null
@@ -1,1225 +0,0 @@
-/*
-Copyright (c) 2011, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-/* Optimized strlcat with SSSE3 */
-
-#ifndef cfi_startproc
-# define cfi_startproc	.cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc	.cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name,  @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG)	pushl	REG;	CFI_PUSH (REG)
-#define POP(REG)	popl	REG;	CFI_POP (REG)
-#define L(label)	.L##Prolog_##label
-
-#define DST	4
-#define SRC	DST+8
-#define LEN	SRC+4
-
-	.text
-ENTRY (strlcat)
-	mov	DST(%esp), %edx
-	PUSH	(%ebx)
-	mov	LEN(%esp), %ebx
-	sub	$4, %ebx
-	jbe	L(len_less4_prolog)
-
-#define RETURN	jmp	L(StrcpyStep)
-#define edi	ebx
-
-#define USE_AS_STRNLEN
-#define USE_AS_STRCAT
-#define USE_AS_STRLCAT
-
-#include "sse2-strlen-atom.S"
-
-	.p2align 4
-L(StrcpyStep):
-
-#undef edi
-#undef L
-#define L(label) .L##label
-#undef RETURN
-#define RETURN	POP (%ebx); ret; CFI_PUSH (%ebx);
-#define RETURN1	POP (%edi); POP (%ebx); ret; CFI_PUSH (%ebx); CFI_PUSH (%edi)
-
-        movl	SRC(%esp), %ecx
-	movl	LEN(%esp), %ebx
-
-	cmp	%eax, %ebx
-	je	L(CalculateLengthOfSrcProlog)
-	sub	%eax, %ebx
-
-	test	%ebx, %ebx
-	jz	L(CalculateLengthOfSrcProlog)
-
-	mov	DST + 4(%esp), %edx
-
-	PUSH	(%edi)
-	add	%eax, %edx
-	mov	%ecx, %edi
-	sub	%eax, %edi
-
-	cmp	$8, %ebx
-	jbe	L(StrncpyExit8Bytes)
-
-	cmpb	$0, (%ecx)
-	jz	L(Exit1)
-	cmpb	$0, 1(%ecx)
-	jz	L(Exit2)
-	cmpb	$0, 2(%ecx)
-	jz	L(Exit3)
-	cmpb	$0, 3(%ecx)
-	jz	L(Exit4)
-	cmpb	$0, 4(%ecx)
-	jz	L(Exit5)
-	cmpb	$0, 5(%ecx)
-	jz	L(Exit6)
-	cmpb	$0, 6(%ecx)
-	jz	L(Exit7)
-	cmpb	$0, 7(%ecx)
-	jz	L(Exit8)
-	cmp	$16, %ebx
-	jb	L(StrncpyExit15Bytes)
-	cmpb	$0, 8(%ecx)
-	jz	L(Exit9)
-	cmpb	$0, 9(%ecx)
-	jz	L(Exit10)
-	cmpb	$0, 10(%ecx)
-	jz	L(Exit11)
-	cmpb	$0, 11(%ecx)
-	jz	L(Exit12)
-	cmpb	$0, 12(%ecx)
-	jz	L(Exit13)
-	cmpb	$0, 13(%ecx)
-	jz	L(Exit14)
-	cmpb	$0, 14(%ecx)
-	jz	L(Exit15)
-	cmpb	$0, 15(%ecx)
-	jz	L(Exit16)
-	cmp	$16, %ebx
-	je	L(StrlcpyExit16)
-
-#define USE_AS_STRNCPY
-#include "ssse3-strcpy-atom.S"
-
-	.p2align 4
-L(CopyFrom1To16Bytes):
-	add	%esi, %edx
-	add	%esi, %ecx
-
-	POP	(%esi)
-	test	%al, %al
-	jz	L(ExitHigh8)
-
-L(CopyFrom1To16BytesLess8):
-	mov	%al, %ah
-	and	$15, %ah
-	jz	L(ExitHigh4)
-
-	test	$0x01, %al
-	jnz	L(Exit1)
-	test	$0x02, %al
-	jnz	L(Exit2)
-	test	$0x04, %al
-	jnz	L(Exit3)
-L(Exit4):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-
-	lea	3(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh4):
-	test	$0x10, %al
-	jnz	L(Exit5)
-	test	$0x20, %al
-	jnz	L(Exit6)
-	test	$0x40, %al
-	jnz	L(Exit7)
-L(Exit8):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	lea	7(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh8):
-	mov	%ah, %al
-	and	$15, %al
-	jz	L(ExitHigh12)
-
-	test	$0x01, %ah
-	jnz	L(Exit9)
-	test	$0x02, %ah
-	jnz	L(Exit10)
-	test	$0x04, %ah
-	jnz	L(Exit11)
-L(Exit12):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	lea	11(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh12):
-	test	$0x10, %ah
-	jnz	L(Exit13)
-	test	$0x20, %ah
-	jnz	L(Exit14)
-	test	$0x40, %ah
-	jnz	L(Exit15)
-L(Exit16):
-	movlpd	(%ecx), %xmm0
-	movlpd	8(%ecx), %xmm1
-	movlpd	%xmm0, (%edx)
-	movlpd	%xmm1, 8(%edx)
-
-	lea	15(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	CFI_PUSH(%esi)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase2):
-	add	$16, %ebx
-	add	%esi, %ecx
-	add	%esi, %edx
-
-	POP	(%esi)
-
-	test	%al, %al
-	jz	L(ExitHighCase2)
-
-	cmp	$8, %ebx
-	ja	L(CopyFrom1To16BytesLess8)
-
-	test	$0x01, %al
-	jnz	L(Exit1)
-	cmp	$1, %ebx
-	je	L(StrlcpyExit1)
-	test	$0x02, %al
-	jnz	L(Exit2)
-	cmp	$2, %ebx
-	je	L(StrlcpyExit2)
-	test	$0x04, %al
-	jnz	L(Exit3)
-	cmp	$3, %ebx
-	je	L(StrlcpyExit3)
-	test	$0x08, %al
-	jnz	L(Exit4)
-	cmp	$4, %ebx
-	je	L(StrlcpyExit4)
-	test	$0x10, %al
-	jnz	L(Exit5)
-	cmp	$5, %ebx
-	je	L(StrlcpyExit5)
-	test	$0x20, %al
-	jnz	L(Exit6)
-	cmp	$6, %ebx
-	je	L(StrlcpyExit6)
-	test	$0x40, %al
-	jnz	L(Exit7)
-	cmp	$7, %ebx
-	je	L(StrlcpyExit7)
-	test	$0x80, %al
-	jnz	L(Exit8)
-	jmp	L(StrlcpyExit8)
-
-	.p2align 4
-L(ExitHighCase2):
-	cmp	$8, %ebx
-	jbe	L(CopyFrom1To16BytesLess8Case3)
-
-	test	$0x01, %ah
-	jnz	L(Exit9)
-	cmp	$9, %ebx
-	je	L(StrlcpyExit9)
-	test	$0x02, %ah
-	jnz	L(Exit10)
-	cmp	$10, %ebx
-	je	L(StrlcpyExit10)
-	test	$0x04, %ah
-	jnz	L(Exit11)
-	cmp	$11, %ebx
-	je	L(StrlcpyExit11)
-	test	$0x8, %ah
-	jnz	L(Exit12)
-	cmp	$12, %ebx
-	je	L(StrlcpyExit12)
-	test	$0x10, %ah
-	jnz	L(Exit13)
-	cmp	$13, %ebx
-	je	L(StrlcpyExit13)
-	test	$0x20, %ah
-	jnz	L(Exit14)
-	cmp	$14, %ebx
-	je	L(StrlcpyExit14)
-	test	$0x40, %ah
-	jnz	L(Exit15)
-	cmp	$15, %ebx
-	je	L(StrlcpyExit15)
-	test	$0x80, %ah
-	jnz	L(Exit16)
-	jmp	L(StrlcpyExit16)
-
-	CFI_PUSH(%esi)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase2OrCase3):
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16BytesCase2)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase3):
-	add	$16, %ebx
-	add	%esi, %edx
-	add	%esi, %ecx
-
-	POP	(%esi)
-
-	cmp	$8, %ebx
-	ja	L(ExitHigh8Case3)
-
-L(CopyFrom1To16BytesLess8Case3):
-	cmp	$4, %ebx
-	ja	L(ExitHigh4Case3)
-
-	cmp	$1, %ebx
-	je	L(StrlcpyExit1)
-	cmp	$2, %ebx
-	je	L(StrlcpyExit2)
-	cmp	$3, %ebx
-	je	L(StrlcpyExit3)
-L(StrlcpyExit4):
-	movb	%bh, 3(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	lea	4(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh4Case3):
-	cmp	$5, %ebx
-	je	L(StrlcpyExit5)
-	cmp	$6, %ebx
-	je	L(StrlcpyExit6)
-	cmp	$7, %ebx
-	je	L(StrlcpyExit7)
-L(StrlcpyExit8):
-	movb	%bh, 7(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	lea	8(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh8Case3):
-	cmp	$12, %ebx
-	ja	L(ExitHigh12Case3)
-
-	cmp	$9, %ebx
-	je	L(StrlcpyExit9)
-	cmp	$10, %ebx
-	je	L(StrlcpyExit10)
-	cmp	$11, %ebx
-	je	L(StrlcpyExit11)
-L(StrlcpyExit12):
-	movb	%bh, 11(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	lea	12(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh12Case3):
-	cmp	$13, %ebx
-	je	L(StrlcpyExit13)
-	cmp	$14, %ebx
-	je	L(StrlcpyExit14)
-	cmp	$15, %ebx
-	je	L(StrlcpyExit15)
-L(StrlcpyExit16):
-	movb	%bh, 15(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	lea	16(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(StrlcpyExit1):
-	movb	%bh, (%edx)
-
-	lea	1(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit1):
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	mov	%ecx, %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit2):
-	movb	%bh, 1(%edx)
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	lea	2(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit2):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movl	%edi, %eax
-
-	lea	1(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit3):
-	movb	%bh, 2(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-
-	lea	3(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit3):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	lea	2(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit5):
-	movb	%bh, 4(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	%edi, %eax
-
-	lea	5(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit5):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	lea	4(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit6):
-	movb	%bh, 5(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	lea	6(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit6):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	lea	5(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit7):
-	movb	%bh, 6(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	lea	7(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit7):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	lea	6(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit9):
-	movb	%bh, 8(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	lea	9(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit9):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	lea	8(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit10):
-	movb	%bh, 9(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	lea	10(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit10):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	lea	9(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit11):
-	movb	%bh, 10(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	lea	11(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit11):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	lea	10(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit13):
-	movb	%bh, 12(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	lea	13(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit13):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	lea	12(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit14):
-	movb	%bh, 13(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	lea	14(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit14):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	lea	13(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit15):
-	movb	%bh, 14(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	lea	15(%ecx), %edx
-	mov	%edi, %ecx
-	POP	(%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit15):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	lea	14(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrncpyExit15Bytes):
-	cmp	$12, %ebx
-	ja	L(StrncpyExit15Bytes1)
-
-	cmpb	$0, 8(%ecx)
-	jz	L(Exit9)
-	cmp	$9, %ebx
-	je	L(StrlcpyExit9)
-
-	cmpb	$0, 9(%ecx)
-	jz	L(Exit10)
-	cmp	$10, %ebx
-	je	L(StrlcpyExit10)
-
-	cmpb	$0, 10(%ecx)
-	jz	L(Exit11)
-	cmp	$11, %ebx
-	je	L(StrlcpyExit11)
-
-	cmpb	$0, 11(%ecx)
-	jz	L(Exit12)
-	jmp	L(StrlcpyExit12)
-
-	.p2align 4
-L(StrncpyExit15Bytes1):
-	cmpb	$0, 8(%ecx)
-	jz	L(Exit9)
-	cmpb	$0, 9(%ecx)
-	jz	L(Exit10)
-	cmpb	$0, 10(%ecx)
-	jz	L(Exit11)
-	cmpb	$0, 11(%ecx)
-	jz	L(Exit12)
-
-	cmpb	$0, 12(%ecx)
-	jz	L(Exit13)
-	cmp	$13, %ebx
-	je	L(StrlcpyExit13)
-
-	cmpb	$0, 13(%ecx)
-	jz	L(Exit14)
-	cmp	$14, %ebx
-	je	L(StrlcpyExit14)
-
-	cmpb	$0, 14(%ecx)
-	jz	L(Exit15)
-	jmp	L(StrlcpyExit15)
-
-	.p2align 4
-L(StrncpyExit8Bytes):
-	cmp	$4, %ebx
-	ja	L(StrncpyExit8Bytes1)
-
-	cmpb	$0, (%ecx)
-	jz	L(Exit1)
-	cmp	$1, %ebx
-	je	L(StrlcpyExit1)
-
-	cmpb	$0, 1(%ecx)
-	jz	L(Exit2)
-	cmp	$2, %ebx
-	je	L(StrlcpyExit2)
-
-	cmpb	$0, 2(%ecx)
-	jz	L(Exit3)
-	cmp	$3, %ebx
-	je	L(StrlcpyExit3)
-
-	cmpb	$0, 3(%ecx)
-	jz	L(Exit4)
-	jmp	L(StrlcpyExit4)
-
-	.p2align 4
-L(StrncpyExit8Bytes1):
-	cmpb	$0, (%ecx)
-	jz	L(Exit1)
-	cmpb	$0, 1(%ecx)
-	jz	L(Exit2)
-	cmpb	$0, 2(%ecx)
-	jz	L(Exit3)
-	cmpb	$0, 3(%ecx)
-	jz	L(Exit4)
-
-	cmpb	$0, 4(%ecx)
-	jz	L(Exit5)
-	cmp	$5, %ebx
-	je	L(StrlcpyExit5)
-
-	cmpb	$0, 5(%ecx)
-	jz	L(Exit6)
-	cmp	$6, %ebx
-	je	L(StrlcpyExit6)
-
-	cmpb	$0, 6(%ecx)
-	jz	L(Exit7)
-	cmp	$7, %ebx
-	je	L(StrlcpyExit7)
-
-	cmpb	$0, 7(%ecx)
-	jz	L(Exit8)
-	jmp	L(StrlcpyExit8)
-
-	CFI_POP	(%edi)
-
-
-	.p2align 4
-L(Prolog_return_start_len):
-	movl	LEN(%esp), %ebx
-        movl	SRC(%esp), %ecx
-L(CalculateLengthOfSrcProlog):
-	mov	%ecx, %edx
-	sub	%ebx, %ecx
-
-	.p2align 4
-L(CalculateLengthOfSrc):
-	cmpb	$0, (%edx)
-	jz	L(exit_tail0)
-	cmpb	$0, 1(%edx)
-	jz	L(exit_tail1)
-	cmpb	$0, 2(%edx)
-	jz	L(exit_tail2)
-	cmpb	$0, 3(%edx)
-	jz	L(exit_tail3)
-
-	cmpb	$0, 4(%edx)
-	jz	L(exit_tail4)
-	cmpb	$0, 5(%edx)
-	jz	L(exit_tail5)
-	cmpb	$0, 6(%edx)
-	jz	L(exit_tail6)
-	cmpb	$0, 7(%edx)
-	jz	L(exit_tail7)
-
-	cmpb	$0, 8(%edx)
-	jz	L(exit_tail8)
-	cmpb	$0, 9(%edx)
-	jz	L(exit_tail9)
-	cmpb	$0, 10(%edx)
-	jz	L(exit_tail10)
-	cmpb	$0, 11(%edx)
-	jz	L(exit_tail11)
-
-	cmpb	$0, 12(%edx)
-	jz	L(exit_tail12)
-	cmpb	$0, 13(%edx)
-	jz	L(exit_tail13)
-	cmpb	$0, 14(%edx)
-	jz	L(exit_tail14)
-	cmpb	$0, 15(%edx)
-	jz	L(exit_tail15)
-
-	pxor	%xmm0, %xmm0
-	lea	16(%edx), %eax
-	add	$16, %ecx
-	and	$-16, %eax
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	pxor	%xmm1, %xmm1
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	pxor	%xmm2, %xmm2
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	pxor	%xmm3, %xmm3
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	and	$-0x40, %eax
-
-	.p2align 4
-L(aligned_64_loop):
-	movaps	(%eax), %xmm0
-	movaps	16(%eax), %xmm1
-	movaps	32(%eax), %xmm2
-	movaps	48(%eax), %xmm6
-	pminub	%xmm1, %xmm0
-	pminub	%xmm6, %xmm2
-	pminub	%xmm0, %xmm2
-	pcmpeqb	%xmm3, %xmm2
-	pmovmskb %xmm2, %edx
-	lea	64(%eax), %eax
-	test	%edx, %edx
-	jz	L(aligned_64_loop)
-
-	pcmpeqb	-64(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	48(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	-32(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	%xmm6, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-
-	.p2align 4
-L(exit):
-	sub	%ecx, %eax
-	test	%dl, %dl
-	jz	L(exit_more_8)
-
-	mov	%dl, %cl
-	and	$15, %cl
-	jz	L(exit_more_4)
-	test	$0x01, %dl
-	jnz	L(exit_0)
-	test	$0x02, %dl
-	jnz	L(exit_1)
-	test	$0x04, %dl
-	jnz	L(exit_2)
-	add	$3, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_4):
-	test	$0x10, %dl
-	jnz	L(exit_4)
-	test	$0x20, %dl
-	jnz	L(exit_5)
-	test	$0x40, %dl
-	jnz	L(exit_6)
-	add	$7, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_8):
-	mov	%dh, %ch
-	and	$15, %ch
-	jz	L(exit_more_12)
-	test	$0x01, %dh
-	jnz	L(exit_8)
-	test	$0x02, %dh
-	jnz	L(exit_9)
-	test	$0x04, %dh
-	jnz	L(exit_10)
-	add	$11, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_12):
-	test	$0x10, %dh
-	jnz	L(exit_12)
-	test	$0x20, %dh
-	jnz	L(exit_13)
-	test	$0x40, %dh
-	jnz	L(exit_14)
-	add	$15, %eax
-L(exit_0):
-	RETURN
-
-	.p2align 4
-L(exit_1):
-	add	$1, %eax
-	RETURN
-
-L(exit_2):
-	add	$2, %eax
-	RETURN
-
-L(exit_3):
-	add	$3, %eax
-	RETURN
-
-L(exit_4):
-	add	$4, %eax
-	RETURN
-
-L(exit_5):
-	add	$5, %eax
-	RETURN
-
-L(exit_6):
-	add	$6, %eax
-	RETURN
-
-L(exit_7):
-	add	$7, %eax
-	RETURN
-
-L(exit_8):
-	add	$8, %eax
-	RETURN
-
-L(exit_9):
-	add	$9, %eax
-	RETURN
-
-L(exit_10):
-	add	$10, %eax
-	RETURN
-
-L(exit_11):
-	add	$11, %eax
-	RETURN
-
-L(exit_12):
-	add	$12, %eax
-	RETURN
-
-L(exit_13):
-	add	$13, %eax
-	RETURN
-
-L(exit_14):
-	add	$14, %eax
-	RETURN
-
-L(exit_15):
-	add	$15, %eax
-	RETURN
-
-L(exit_tail0):
-	mov	%edx, %eax
-	sub	%ecx, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail1):
-	lea	1(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail2):
-	lea	2(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail3):
-	lea	3(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail4):
-	lea	4(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail5):
-	lea	5(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail6):
-	lea	6(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail7):
-	lea	7(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail8):
-	lea	8(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail9):
-	lea	9(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail10):
-	lea	10(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail11):
-	lea	11(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail12):
-	lea	12(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail13):
-	lea	13(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail14):
-	lea	14(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail15):
-	lea	15(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-END (strlcat)
diff --git a/libc/arch-x86/string/ssse3-strlcpy-atom.S b/libc/arch-x86/string/ssse3-strlcpy-atom.S
deleted file mode 100644
index cdb17cc..0000000
--- a/libc/arch-x86/string/ssse3-strlcpy-atom.S
+++ /dev/null
@@ -1,1403 +0,0 @@
-/*
-Copyright (c) 2011, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#define USE_AS_STRNCPY
-#define STRCPY strlcpy
-#define STRLEN strlcpy
-#define USE_AS_STRLCPY
-#include "ssse3-strcpy-atom.S"
-
-	.p2align 4
-L(CopyFrom1To16Bytes):
-	add	%esi, %edx
-	add	%esi, %ecx
-
-	POP	(%esi)
-	test	%al, %al
-	jz	L(ExitHigh8)
-
-L(CopyFrom1To16BytesLess8):
-	mov	%al, %ah
-	and	$15, %ah
-	jz	L(ExitHigh4)
-
-	test	$0x01, %al
-	jnz	L(Exit1)
-	test	$0x02, %al
-	jnz	L(Exit2)
-	test	$0x04, %al
-	jnz	L(Exit3)
-L(Exit4):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-
-	lea	3(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh4):
-	test	$0x10, %al
-	jnz	L(Exit5)
-	test	$0x20, %al
-	jnz	L(Exit6)
-	test	$0x40, %al
-	jnz	L(Exit7)
-L(Exit8):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	lea	7(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh8):
-	mov	%ah, %al
-	and	$15, %al
-	jz	L(ExitHigh12)
-
-	test	$0x01, %ah
-	jnz	L(Exit9)
-	test	$0x02, %ah
-	jnz	L(Exit10)
-	test	$0x04, %ah
-	jnz	L(Exit11)
-L(Exit12):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	lea	11(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(ExitHigh12):
-	test	$0x10, %ah
-	jnz	L(Exit13)
-	test	$0x20, %ah
-	jnz	L(Exit14)
-	test	$0x40, %ah
-	jnz	L(Exit15)
-L(Exit16):
-	movlpd	(%ecx), %xmm0
-	movlpd	8(%ecx), %xmm1
-	movlpd	%xmm0, (%edx)
-	movlpd	%xmm1, 8(%edx)
-
-	lea	15(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	CFI_PUSH(%esi)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase2):
-	add	$16, %ebx
-	add	%esi, %ecx
-        add     %esi, %edx
-
-	POP	(%esi)
-
-        test    %al, %al
-        jz      L(ExitHighCase2)
-
-        cmp     $8, %ebx
-        ja      L(CopyFrom1To16BytesLess8)
-
-	test	$0x01, %al
-	jnz	L(Exit1)
-	cmp	$1, %ebx
-	je	L(StrlcpyExit1)
-	test	$0x02, %al
-	jnz	L(Exit2)
-	cmp	$2, %ebx
-	je	L(StrlcpyExit2)
-	test	$0x04, %al
-	jnz	L(Exit3)
-	cmp	$3, %ebx
-	je	L(StrlcpyExit3)
-	test	$0x08, %al
-	jnz	L(Exit4)
-	cmp	$4, %ebx
-	je	L(StrlcpyExit4)
-	test	$0x10, %al
-	jnz	L(Exit5)
-	cmp	$5, %ebx
-	je	L(StrlcpyExit5)
-	test	$0x20, %al
-	jnz	L(Exit6)
-	cmp	$6, %ebx
-	je	L(StrlcpyExit6)
-	test	$0x40, %al
-	jnz	L(Exit7)
-	cmp	$7, %ebx
-	je	L(StrlcpyExit7)
-	test	$0x80, %al
-	jnz	L(Exit8)
-	jmp	L(StrlcpyExit8)
-
-	.p2align 4
-L(ExitHighCase2):
-        cmp     $8, %ebx
-        jbe      L(CopyFrom1To16BytesLess8Case3)
-
-	test	$0x01, %ah
-	jnz	L(Exit9)
-	cmp	$9, %ebx
-	je	L(StrlcpyExit9)
-	test	$0x02, %ah
-	jnz	L(Exit10)
-	cmp	$10, %ebx
-	je	L(StrlcpyExit10)
-	test	$0x04, %ah
-	jnz	L(Exit11)
-	cmp	$11, %ebx
-	je	L(StrlcpyExit11)
-	test	$0x8, %ah
-	jnz	L(Exit12)
-	cmp	$12, %ebx
-	je	L(StrlcpyExit12)
-	test	$0x10, %ah
-	jnz	L(Exit13)
-	cmp	$13, %ebx
-	je	L(StrlcpyExit13)
-	test	$0x20, %ah
-	jnz	L(Exit14)
-	cmp	$14, %ebx
-	je	L(StrlcpyExit14)
-	test	$0x40, %ah
-	jnz	L(Exit15)
-	cmp	$15, %ebx
-	je	L(StrlcpyExit15)
-	test	$0x80, %ah
-	jnz	L(Exit16)
-	jmp	L(StrlcpyExit16)
-
-	CFI_PUSH(%esi)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase2OrCase3):
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16BytesCase2)
-
-	.p2align 4
-L(CopyFrom1To16BytesCase3):
-	add	$16, %ebx
-	add	%esi, %edx
-	add	%esi, %ecx
-
-	POP	(%esi)
-
-	cmp	$8, %ebx
-	ja	L(ExitHigh8Case3)
-
-L(CopyFrom1To16BytesLess8Case3):
-	cmp	$4, %ebx
-	ja	L(ExitHigh4Case3)
-
-	cmp	$1, %ebx
-	je	L(StrlcpyExit1)
-	cmp	$2, %ebx
-	je	L(StrlcpyExit2)
-	cmp	$3, %ebx
-	je	L(StrlcpyExit3)
-L(StrlcpyExit4):
-	movb	%bh, 3(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	lea	4(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh4Case3):
-	cmp	$5, %ebx
-	je	L(StrlcpyExit5)
-	cmp	$6, %ebx
-	je	L(StrlcpyExit6)
-	cmp	$7, %ebx
-	je	L(StrlcpyExit7)
-L(StrlcpyExit8):
-	movb	%bh, 7(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	lea	8(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh8Case3):
-	cmp	$12, %ebx
-	ja	L(ExitHigh12Case3)
-
-	cmp	$9, %ebx
-	je	L(StrlcpyExit9)
-	cmp	$10, %ebx
-	je	L(StrlcpyExit10)
-	cmp	$11, %ebx
-	je	L(StrlcpyExit11)
-L(StrlcpyExit12):
-	movb	%bh, 11(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	lea	12(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(ExitHigh12Case3):
-	cmp	$13, %ebx
-	je	L(StrlcpyExit13)
-	cmp	$14, %ebx
-	je	L(StrlcpyExit14)
-	cmp	$15, %ebx
-	je	L(StrlcpyExit15)
-L(StrlcpyExit16):
-	movb	%bh, 15(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	lea	16(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(StrlcpyExit1):
-	movb	%bh, (%edx)
-
-	lea	1(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit1):
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	mov	%ecx, %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit2):
-	movb	%bh, 1(%edx)
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	lea	2(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit2):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movl	%edi, %eax
-
-	lea	1(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit3):
-	movb	%bh, 2(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-
-	lea	3(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit3):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	lea	2(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit5):
-	movb	%bh, 4(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	%edi, %eax
-
-	lea	5(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit5):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	lea	4(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit6):
-	movb	%bh, 5(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	lea	6(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit6):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	lea	5(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit7):
-	movb	%bh, 6(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	lea	7(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit7):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	lea	6(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit9):
-	movb	%bh, 8(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	lea	9(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit9):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	lea	8(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit10):
-	movb	%bh, 9(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	lea	10(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit10):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	lea	9(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit11):
-	movb	%bh, 10(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	lea	11(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit11):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	lea	10(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit13):
-	movb	%bh, 12(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	lea	13(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit13):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	lea	12(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit14):
-	movb	%bh, 13(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	lea	14(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit14):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	lea	13(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-	.p2align 4
-L(StrlcpyExit15):
-	movb	%bh, 14(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	lea	15(%ecx), %edx
-	mov	%edi, %ecx
-        POP     (%edi)
-	jmp	L(CalculateLengthOfSrc)
-        CFI_PUSH     (%edi)
-
-	.p2align 4
-L(Exit15):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	lea	14(%ecx), %eax
-	sub	%edi, %eax
-	RETURN1
-
-        CFI_POP (%edi)
-
-	.p2align 4
-L(StrlcpyExit0):
-	movl	$0, %eax
-	RETURN
-
-	.p2align 4
-L(StrncpyExit15Bytes):
-	cmp	$12, %ebx
-	ja	L(StrncpyExit15Bytes1)
-
-	cmpb	$0, 8(%ecx)
-	jz	L(ExitTail9)
-	cmp	$9, %ebx
-	je	L(StrlcpyExitTail9)
-
-	cmpb	$0, 9(%ecx)
-	jz	L(ExitTail10)
-	cmp	$10, %ebx
-	je	L(StrlcpyExitTail10)
-
-	cmpb	$0, 10(%ecx)
-	jz	L(ExitTail11)
-	cmp	$11, %ebx
-	je	L(StrlcpyExitTail11)
-
-	cmpb	$0, 11(%ecx)
-	jz	L(ExitTail12)
-
-	movb	%bh, 11(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	lea	12(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(StrncpyExit15Bytes1):
-	cmpb	$0, 8(%ecx)
-	jz	L(ExitTail9)
-	cmpb	$0, 9(%ecx)
-	jz	L(ExitTail10)
-	cmpb	$0, 10(%ecx)
-	jz	L(ExitTail11)
-	cmpb	$0, 11(%ecx)
-	jz	L(ExitTail12)
-
-	cmpb	$0, 12(%ecx)
-	jz	L(ExitTail13)
-	cmp	$13, %ebx
-	je	L(StrlcpyExitTail13)
-
-	cmpb	$0, 13(%ecx)
-	jz	L(ExitTail14)
-	cmp	$14, %ebx
-	je	L(StrlcpyExitTail14)
-
-	cmpb	$0, 14(%ecx)
-	jz	L(ExitTail15)
-
-	movb	%bh, 14(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	lea	15(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(StrncpyExit8Bytes):
-	cmp	$4, %ebx
-	ja	L(StrncpyExit8Bytes1)
-
-	test	%ebx, %ebx
-	jz	L(StrlcpyExitTail0)
-
-	cmpb	$0, (%ecx)
-	jz	L(ExitTail1)
-	cmp	$1, %ebx
-	je	L(StrlcpyExitTail1)
-
-	cmpb	$0, 1(%ecx)
-	jz	L(ExitTail2)
-	cmp	$2, %ebx
-	je	L(StrlcpyExitTail2)
-
-	cmpb	$0, 2(%ecx)
-	jz	L(ExitTail3)
-	cmp	$3, %ebx
-	je	L(StrlcpyExitTail3)
-
-	cmpb	$0, 3(%ecx)
-	jz	L(ExitTail4)
-
-	movb	%bh, 3(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	lea	4(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(StrncpyExit8Bytes1):
-	cmpb	$0, (%ecx)
-	jz	L(ExitTail1)
-	cmpb	$0, 1(%ecx)
-	jz	L(ExitTail2)
-	cmpb	$0, 2(%ecx)
-	jz	L(ExitTail3)
-	cmpb	$0, 3(%ecx)
-	jz	L(ExitTail4)
-
-	cmpb	$0, 4(%ecx)
-	jz	L(ExitTail5)
-	cmp	$5, %ebx
-	je	L(StrlcpyExitTail5)
-
-	cmpb	$0, 5(%ecx)
-	jz	L(ExitTail6)
-	cmp	$6, %ebx
-	je	L(StrlcpyExitTail6)
-
-	cmpb	$0, 6(%ecx)
-	jz	L(ExitTail7)
-	cmp	$7, %ebx
-	je	L(StrlcpyExitTail7)
-
-	cmpb	$0, 7(%ecx)
-	jz	L(ExitTail8)
-
-	movb	%bh, 7(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	lea	8(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(StrlcpyExitTail0):
-	mov	%ecx, %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(StrlcpyExitTail1):
-	movb	%bh, (%edx)
-
-	lea	1(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail1):
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	mov	$0, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail2):
-	movb	%bh, 1(%edx)
-	movb	(%ecx), %al
-	movb	%al, (%edx)
-
-	lea	2(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail2):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movl	%edx, %eax
-
-	mov	$1, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail3):
-	movb	%bh, 2(%edx)
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-
-	lea	3(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail3):
-	movw	(%ecx), %ax
-	movw	%ax, (%edx)
-	movb	2(%ecx), %al
-	movb	%al, 2(%edx)
-
-	mov	$2, %eax
-	RETURN
-
-	.p2align 4
-L(ExitTail4):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-
-	mov	$3, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail5):
-	movb	%bh, 4(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	%edx, %eax
-
-	lea	5(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail5):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	mov	$4, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail6):
-	movb	%bh, 5(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movb	4(%ecx), %al
-	movb	%al, 4(%edx)
-
-	lea	6(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail6):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	mov	$5, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail7):
-	movb	%bh, 6(%edx)
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movw	4(%ecx), %ax
-	movw	%ax, 4(%edx)
-
-	lea	7(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail7):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	3(%ecx), %eax
-	movl	%eax, 3(%edx)
-
-	mov	$6, %eax
-	RETURN
-
-	.p2align 4
-L(ExitTail8):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	mov	$7, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail9):
-	movb	%bh, 8(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-
-	lea	9(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail9):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	mov	$8, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail10):
-	movb	%bh, 9(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movb	8(%ecx), %al
-	movb	%al, 8(%edx)
-
-	lea	10(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail10):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	mov	$9, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail11):
-	movb	%bh, 10(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movw	8(%ecx), %ax
-	movw	%ax, 8(%edx)
-
-	lea	11(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail11):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	7(%ecx), %eax
-	movl	%eax, 7(%edx)
-
-	mov	$10, %eax
-	RETURN
-
-	.p2align 4
-L(ExitTail12):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	mov	$11, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail13):
-	movb	%bh, 12(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-
-	lea	13(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail13):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	mov	$12, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail14):
-	movb	%bh, 13(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	5(%ecx), %xmm0
-	movlpd	%xmm0, 5(%edx)
-
-	lea	14(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail14):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	6(%ecx), %xmm0
-	movlpd	%xmm0, 6(%edx)
-
-	mov	$13, %eax
-	RETURN
-
-	.p2align 4
-L(ExitTail15):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	mov	$14, %eax
-	RETURN
-
-	.p2align 4
-L(StrlcpyExitTail16):
-	movb	%bh, 15(%edx)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movlpd	7(%ecx), %xmm0
-	movlpd	%xmm0, 7(%edx)
-
-	lea	16(%ecx), %edx
-	jmp	L(CalculateLengthOfSrc)
-
-	.p2align 4
-L(ExitTail16):
-	movlpd	(%ecx), %xmm0
-	movlpd	8(%ecx), %xmm1
-	movlpd	%xmm0, (%edx)
-	movlpd	%xmm1, 8(%edx)
-
-	mov	$15, %eax
-	RETURN
-
-	.p2align 4
-L(CalculateLengthOfSrc):
-	xor	%eax, %eax
-	cmpb	$0, (%edx)
-	jz	L(exit_tail0)
-	cmpb	$0, 1(%edx)
-	jz	L(exit_tail1)
-	cmpb	$0, 2(%edx)
-	jz	L(exit_tail2)
-	cmpb	$0, 3(%edx)
-	jz	L(exit_tail3)
-
-	cmpb	$0, 4(%edx)
-	jz	L(exit_tail4)
-	cmpb	$0, 5(%edx)
-	jz	L(exit_tail5)
-	cmpb	$0, 6(%edx)
-	jz	L(exit_tail6)
-	cmpb	$0, 7(%edx)
-	jz	L(exit_tail7)
-
-	cmpb	$0, 8(%edx)
-	jz	L(exit_tail8)
-	cmpb	$0, 9(%edx)
-	jz	L(exit_tail9)
-	cmpb	$0, 10(%edx)
-	jz	L(exit_tail10)
-	cmpb	$0, 11(%edx)
-	jz	L(exit_tail11)
-
-	cmpb	$0, 12(%edx)
-	jz	L(exit_tail12)
-	cmpb	$0, 13(%edx)
-	jz	L(exit_tail13)
-	cmpb	$0, 14(%edx)
-	jz	L(exit_tail14)
-	cmpb	$0, 15(%edx)
-	jz	L(exit_tail15)
-
-	pxor	%xmm0, %xmm0
-	lea	16(%edx), %eax
-	add	$16, %ecx
-	and	$-16, %eax
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	pxor	%xmm1, %xmm1
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	pxor	%xmm2, %xmm2
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	pxor	%xmm3, %xmm3
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm0
-	pmovmskb %xmm0, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm1
-	pmovmskb %xmm1, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm2
-	pmovmskb %xmm2, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	16(%eax), %eax
-	test	%edx, %edx
-	jnz	L(exit)
-
-	and	$-0x40, %eax
-
-	.p2align 4
-L(aligned_64_loop):
-	movaps	(%eax), %xmm0
-	movaps	16(%eax), %xmm1
-	movaps	32(%eax), %xmm2
-	movaps	48(%eax), %xmm6
-	pminub	%xmm1, %xmm0
-	pminub	%xmm6, %xmm2
-	pminub	%xmm0, %xmm2
-	pcmpeqb	%xmm3, %xmm2
-	pmovmskb %xmm2, %edx
-	lea	64(%eax), %eax
-	test	%edx, %edx
-	jz	L(aligned_64_loop)
-
-	pcmpeqb	-64(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	48(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	%xmm1, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	-32(%eax), %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-	test	%edx, %edx
-	jnz	L(exit)
-
-	pcmpeqb	%xmm6, %xmm3
-	pmovmskb %xmm3, %edx
-	lea	-16(%ecx), %ecx
-
-	.p2align 4
-L(exit):
-	sub	%ecx, %eax
-	test	%dl, %dl
-	jz	L(exit_more_8)
-
-	mov	%dl, %cl
-	and	$15, %cl
-	jz	L(exit_more_4)
-	test	$0x01, %dl
-	jnz	L(exit_0)
-	test	$0x02, %dl
-	jnz	L(exit_1)
-	test	$0x04, %dl
-	jnz	L(exit_2)
-	add	$3, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_4):
-	test	$0x10, %dl
-	jnz	L(exit_4)
-	test	$0x20, %dl
-	jnz	L(exit_5)
-	test	$0x40, %dl
-	jnz	L(exit_6)
-	add	$7, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_8):
-	mov	%dh, %ch
-	and	$15, %ch
-	jz	L(exit_more_12)
-	test	$0x01, %dh
-	jnz	L(exit_8)
-	test	$0x02, %dh
-	jnz	L(exit_9)
-	test	$0x04, %dh
-	jnz	L(exit_10)
-	add	$11, %eax
-	RETURN
-
-	.p2align 4
-L(exit_more_12):
-	test	$0x10, %dh
-	jnz	L(exit_12)
-	test	$0x20, %dh
-	jnz	L(exit_13)
-	test	$0x40, %dh
-	jnz	L(exit_14)
-	add	$15, %eax
-L(exit_0):
-	RETURN
-
-	.p2align 4
-L(exit_1):
-	add	$1, %eax
-	RETURN
-
-L(exit_2):
-	add	$2, %eax
-	RETURN
-
-L(exit_3):
-	add	$3, %eax
-	RETURN
-
-L(exit_4):
-	add	$4, %eax
-	RETURN
-
-L(exit_5):
-	add	$5, %eax
-	RETURN
-
-L(exit_6):
-	add	$6, %eax
-	RETURN
-
-L(exit_7):
-	add	$7, %eax
-	RETURN
-
-L(exit_8):
-	add	$8, %eax
-	RETURN
-
-L(exit_9):
-	add	$9, %eax
-	RETURN
-
-L(exit_10):
-	add	$10, %eax
-	RETURN
-
-L(exit_11):
-	add	$11, %eax
-	RETURN
-
-L(exit_12):
-	add	$12, %eax
-	RETURN
-
-L(exit_13):
-	add	$13, %eax
-	RETURN
-
-L(exit_14):
-	add	$14, %eax
-	RETURN
-
-L(exit_15):
-	add	$15, %eax
-	RETURN
-
-L(exit_tail0):
-	mov	%edx, %eax
-	sub	%ecx, %eax
-	RETURN
-
-	.p2align 4
-L(exit_tail1):
-	lea	1(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail2):
-	lea	2(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail3):
-	lea	3(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail4):
-	lea	4(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail5):
-	lea	5(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail6):
-	lea	6(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail7):
-	lea	7(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail8):
-	lea	8(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail9):
-	lea	9(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail10):
-	lea	10(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail11):
-	lea	11(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail12):
-	lea	12(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail13):
-	lea	13(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail14):
-	lea	14(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-L(exit_tail15):
-	lea	15(%edx), %eax
-	sub	%ecx, %eax
-	RETURN
-
-END (STRCPY)
-
diff --git a/libc/arch-x86/string/ssse3-wcscat-atom.S b/libc/arch-x86/string/ssse3-wcscat-atom.S
deleted file mode 100644
index 8a389a3..0000000
--- a/libc/arch-x86/string/ssse3-wcscat-atom.S
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
-Copyright (c) 2011 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef L
-# define L(label)	.L##label
-#endif
-
-#ifndef cfi_startproc
-# define cfi_startproc                  .cfi_startproc
-#endif
-
-#ifndef cfi_endproc
-# define cfi_endproc                    .cfi_endproc
-#endif
-
-#ifndef cfi_rel_offset
-# define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-#endif
-
-#ifndef cfi_restore
-# define cfi_restore(reg)	.cfi_restore reg
-#endif
-
-#ifndef cfi_adjust_cfa_offset
-# define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-#endif
-
-#ifndef ENTRY
-# define ENTRY(name)	\
-	.type name,  @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-#endif
-
-#ifndef END
-# define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-#endif
-
-#define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-#define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-#define PUSH(REG)	pushl REG;	CFI_PUSH (REG)
-#define POP(REG)	popl REG;	CFI_POP (REG)
-
-#define PARMS  4
-#define STR1  PARMS+4
-#define STR2  STR1+4
-
-#define USE_AS_WCSCAT
-
-.text
-ENTRY (wcscat)
-	PUSH    (%edi)
-	mov	STR1(%esp), %edi
-	mov	%edi, %edx
-
-#define RETURN  jmp L(WcscpyAtom)
-#include "sse2-wcslen-atom.S"
-
-L(WcscpyAtom):
-	shl	$2, %eax
-	mov	STR2(%esp), %ecx
-	lea	(%edi, %eax), %edx
-
-	cmpl	$0, (%ecx)
-	jz	L(Exit4)
-	cmpl	$0, 4(%ecx)
-	jz	L(Exit8)
-	cmpl	$0, 8(%ecx)
-	jz	L(Exit12)
-	cmpl	$0, 12(%ecx)
-	jz	L(Exit16)
-
-#undef RETURN
-#define RETURN  POP(%edi);	ret;	CFI_PUSH(%edi)
-#include "ssse3-wcscpy-atom.S"
-
-END (wcscat)
diff --git a/libc/arch-x86/string/ssse3-wcscpy-atom.S b/libc/arch-x86/string/ssse3-wcscpy-atom.S
deleted file mode 100644
index 27cb61e..0000000
--- a/libc/arch-x86/string/ssse3-wcscpy-atom.S
+++ /dev/null
@@ -1,652 +0,0 @@
-/*
-Copyright (c) 2011, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#ifndef USE_AS_WCSCAT
-
-# ifndef L
-#  define L(label)	.L##label
-# endif
-
-# ifndef cfi_startproc
-#  define cfi_startproc	.cfi_startproc
-# endif
-
-# ifndef cfi_endproc
-#  define cfi_endproc	.cfi_endproc
-# endif
-
-# ifndef cfi_rel_offset
-#  define cfi_rel_offset(reg, off)	.cfi_rel_offset reg, off
-# endif
-
-# ifndef cfi_restore
-#  define cfi_restore(reg)	.cfi_restore reg
-# endif
-
-# ifndef cfi_adjust_cfa_offset
-#  define cfi_adjust_cfa_offset(off)	.cfi_adjust_cfa_offset off
-# endif
-
-# ifndef ENTRY
-#  define ENTRY(name)	\
-	.type name, @function;	\
-	.globl name;	\
-	.p2align 4;	\
-name:	\
-	cfi_startproc
-# endif
-
-# ifndef END
-#  define END(name)	\
-	cfi_endproc;	\
-	.size name, .-name
-# endif
-
-# define CFI_PUSH(REG)	\
-	cfi_adjust_cfa_offset (4);	\
-	cfi_rel_offset (REG, 0)
-
-# define CFI_POP(REG)	\
-	cfi_adjust_cfa_offset (-4);	\
-	cfi_restore (REG)
-
-# define PUSH(REG)	pushl REG; CFI_PUSH (REG)
-# define POP(REG)	popl REG; CFI_POP (REG)
-
-# define PARMS	4
-# define RETURN	POP (%edi); ret; CFI_PUSH (%edi)
-
-# define STR1	PARMS
-# define STR2	STR1+4
-# define LEN	STR2+4
-
-.text
-ENTRY (wcscpy)
-	mov	STR1(%esp), %edx
-	mov	STR2(%esp), %ecx
-
-	cmpl	$0, (%ecx)
-	jz	L(ExitTail4)
-	cmpl	$0, 4(%ecx)
-	jz	L(ExitTail8)
-	cmpl	$0, 8(%ecx)
-	jz	L(ExitTail12)
-	cmpl	$0, 12(%ecx)
-	jz	L(ExitTail16)
-
-	PUSH	(%edi)
-	mov	%edx, %edi
-#endif
-	PUSH	(%esi)
-	lea	16(%ecx), %esi
-
-	and	$-16, %esi
-
-	pxor	%xmm0, %xmm0
-	pcmpeqd	(%esi), %xmm0
-	movdqu	(%ecx), %xmm1
-	movdqu	%xmm1, (%edx)
-
-	pmovmskb %xmm0, %eax
-	sub	%ecx, %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	mov	%edx, %eax
-	lea	16(%edx), %edx
-	and	$-16, %edx
-	sub	%edx, %eax
-
-	sub	%eax, %ecx
-	mov	%ecx, %eax
-	and	$0xf, %eax
-	mov	$0, %esi
-
-	jz	L(Align16Both)
-	cmp	$4, %eax
-	je	L(Shl4)
-	cmp	$8, %eax
-	je	L(Shl8)
-	jmp	L(Shl12)
-
-L(Align16Both):
-	movaps	(%ecx), %xmm1
-	movaps	16(%ecx), %xmm2
-	movaps	%xmm1, (%edx)
-	pcmpeqd	%xmm2, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	16(%ecx, %esi), %xmm3
-	movaps	%xmm2, (%edx, %esi)
-	pcmpeqd	%xmm3, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	16(%ecx, %esi), %xmm4
-	movaps	%xmm3, (%edx, %esi)
-	pcmpeqd	%xmm4, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	16(%ecx, %esi), %xmm1
-	movaps	%xmm4, (%edx, %esi)
-	pcmpeqd	%xmm1, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	16(%ecx, %esi), %xmm2
-	movaps	%xmm1, (%edx, %esi)
-	pcmpeqd	%xmm2, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	16(%ecx, %esi), %xmm3
-	movaps	%xmm2, (%edx, %esi)
-	pcmpeqd	%xmm3, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	%xmm3, (%edx, %esi)
-	mov	%ecx, %eax
-	lea	16(%ecx, %esi), %ecx
-	and	$-0x40, %ecx
-	sub	%ecx, %eax
-	sub	%eax, %edx
-
-	mov	$-0x40, %esi
-
-L(Aligned64Loop):
-	movaps	(%ecx), %xmm2
-	movaps	32(%ecx), %xmm3
-	movaps	%xmm2, %xmm4
-	movaps	16(%ecx), %xmm5
-	movaps	%xmm3, %xmm6
-	movaps	48(%ecx), %xmm7
-	pminub	%xmm5, %xmm2
-	pminub	%xmm7, %xmm3
-	pminub	%xmm2, %xmm3
-	lea	64(%edx), %edx
-	pcmpeqd	%xmm0, %xmm3
-	lea	64(%ecx), %ecx
-	pmovmskb %xmm3, %eax
-
-	test	%eax, %eax
-	jnz	L(Aligned64Leave)
-	movaps	%xmm4, -64(%edx)
-	movaps	%xmm5, -48(%edx)
-	movaps	%xmm6, -32(%edx)
-	movaps	%xmm7, -16(%edx)
-	jmp	L(Aligned64Loop)
-
-L(Aligned64Leave):
-	pcmpeqd	%xmm4, %xmm0
-	pmovmskb %xmm0, %eax
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	pcmpeqd	%xmm5, %xmm0
-	pmovmskb %xmm0, %eax
-	movaps	%xmm4, -64(%edx)
-	lea	16(%esi), %esi
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	pcmpeqd	%xmm6, %xmm0
-	pmovmskb %xmm0, %eax
-	movaps	%xmm5, -48(%edx)
-	lea	16(%esi), %esi
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	movaps	%xmm6, -32(%edx)
-	pcmpeqd	%xmm7, %xmm0
-	pmovmskb %xmm0, %eax
-	lea	16(%esi), %esi
-	test	%eax, %eax
-	jnz	L(CopyFrom1To16Bytes)
-
-	mov	$-0x40, %esi
-	movaps	%xmm7, -16(%edx)
-	jmp	L(Aligned64Loop)
-
-	.p2align 4
-L(Shl4):
-	movaps	-4(%ecx), %xmm1
-	movaps	12(%ecx), %xmm2
-L(Shl4Start):
-	pcmpeqd	%xmm2, %xmm0
-	pmovmskb %xmm0, %eax
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl4LoopExit)
-
-	palignr	$4, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	28(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm1
-
-	test	%eax, %eax
-	jnz	L(Shl4LoopExit)
-
-	palignr	$4, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	28(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl4LoopExit)
-
-	palignr	$4, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	28(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-
-	test	%eax, %eax
-	jnz	L(Shl4LoopExit)
-
-	palignr	$4, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	lea	28(%ecx), %ecx
-	lea	16(%edx), %edx
-
-	mov	%ecx, %eax
-	and	$-0x40, %ecx
-	sub	%ecx, %eax
-	lea	-12(%ecx), %ecx
-	sub	%eax, %edx
-
-	movaps	-4(%ecx), %xmm1
-
-L(Shl4LoopStart):
-	movaps	12(%ecx), %xmm2
-	movaps	28(%ecx), %xmm3
-	movaps	%xmm3, %xmm6
-	movaps	44(%ecx), %xmm4
-	movaps	%xmm4, %xmm7
-	movaps	60(%ecx), %xmm5
-	pminub	%xmm2, %xmm6
-	pminub	%xmm5, %xmm7
-	pminub	%xmm6, %xmm7
-	pcmpeqd	%xmm0, %xmm7
-	pmovmskb %xmm7, %eax
-	movaps	%xmm5, %xmm7
-	palignr	$4, %xmm4, %xmm5
-	palignr	$4, %xmm3, %xmm4
-	test	%eax, %eax
-	jnz	L(Shl4Start)
-
-	palignr	$4, %xmm2, %xmm3
-	lea	64(%ecx), %ecx
-	palignr	$4, %xmm1, %xmm2
-	movaps	%xmm7, %xmm1
-	movaps	%xmm5, 48(%edx)
-	movaps	%xmm4, 32(%edx)
-	movaps	%xmm3, 16(%edx)
-	movaps	%xmm2, (%edx)
-	lea	64(%edx), %edx
-	jmp	L(Shl4LoopStart)
-
-L(Shl4LoopExit):
-	movlpd	(%ecx), %xmm0
-	movl	8(%ecx), %esi
-	movlpd	%xmm0, (%edx)
-	movl	%esi, 8(%edx)
-	POP	(%esi)
-	add	$12, %edx
-	add	$12, %ecx
-	test	%al, %al
-	jz	L(ExitHigh)
-	test	$0x01, %al
-	jnz	L(Exit4)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	%edi, %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(Shl8):
-	movaps	-8(%ecx), %xmm1
-	movaps	8(%ecx), %xmm2
-L(Shl8Start):
-	pcmpeqd	%xmm2, %xmm0
-	pmovmskb %xmm0, %eax
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl8LoopExit)
-
-	palignr	$8, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	24(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm1
-
-	test	%eax, %eax
-	jnz	L(Shl8LoopExit)
-
-	palignr	$8, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	24(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl8LoopExit)
-
-	palignr	$8, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	24(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-
-	test	%eax, %eax
-	jnz	L(Shl8LoopExit)
-
-	palignr	$8, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	lea	24(%ecx), %ecx
-	lea	16(%edx), %edx
-
-	mov	%ecx, %eax
-	and	$-0x40, %ecx
-	sub	%ecx, %eax
-	lea	-8(%ecx), %ecx
-	sub	%eax, %edx
-
-	movaps	-8(%ecx), %xmm1
-
-L(Shl8LoopStart):
-	movaps	8(%ecx), %xmm2
-	movaps	24(%ecx), %xmm3
-	movaps	%xmm3, %xmm6
-	movaps	40(%ecx), %xmm4
-	movaps	%xmm4, %xmm7
-	movaps	56(%ecx), %xmm5
-	pminub	%xmm2, %xmm6
-	pminub	%xmm5, %xmm7
-	pminub	%xmm6, %xmm7
-	pcmpeqd	%xmm0, %xmm7
-	pmovmskb %xmm7, %eax
-	movaps	%xmm5, %xmm7
-	palignr	$8, %xmm4, %xmm5
-	palignr	$8, %xmm3, %xmm4
-	test	%eax, %eax
-	jnz	L(Shl8Start)
-
-	palignr	$8, %xmm2, %xmm3
-	lea	64(%ecx), %ecx
-	palignr	$8, %xmm1, %xmm2
-	movaps	%xmm7, %xmm1
-	movaps	%xmm5, 48(%edx)
-	movaps	%xmm4, 32(%edx)
-	movaps	%xmm3, 16(%edx)
-	movaps	%xmm2, (%edx)
-	lea	64(%edx), %edx
-	jmp	L(Shl8LoopStart)
-
-L(Shl8LoopExit):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	POP	(%esi)
-	add	$8, %edx
-	add	$8, %ecx
-	test	%al, %al
-	jz	L(ExitHigh)
-	test	$0x01, %al
-	jnz	L(Exit4)
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	%edi, %eax
-	RETURN
-
-	CFI_PUSH	(%esi)
-
-	.p2align 4
-L(Shl12):
-	movaps	-12(%ecx), %xmm1
-	movaps	4(%ecx), %xmm2
-L(Shl12Start):
-	pcmpeqd	%xmm2, %xmm0
-	pmovmskb %xmm0, %eax
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl12LoopExit)
-
-	palignr	$12, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	20(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm1
-
-	test	%eax, %eax
-	jnz	L(Shl12LoopExit)
-
-	palignr	$12, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	20(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-	movaps	%xmm2, %xmm3
-
-	test	%eax, %eax
-	jnz	L(Shl12LoopExit)
-
-	palignr	$12, %xmm1, %xmm2
-	movaps	%xmm2, (%edx)
-	movaps	20(%ecx), %xmm2
-
-	pcmpeqd	%xmm2, %xmm0
-	lea	16(%edx), %edx
-	pmovmskb %xmm0, %eax
-	lea	16(%ecx), %ecx
-
-	test	%eax, %eax
-	jnz	L(Shl12LoopExit)
-
-	palignr	$12, %xmm3, %xmm2
-	movaps	%xmm2, (%edx)
-	lea	20(%ecx), %ecx
-	lea	16(%edx), %edx
-
-	mov	%ecx, %eax
-	and	$-0x40, %ecx
-	sub	%ecx, %eax
-	lea	-4(%ecx), %ecx
-	sub	%eax, %edx
-
-	movaps	-12(%ecx), %xmm1
-
-L(Shl12LoopStart):
-	movaps	4(%ecx), %xmm2
-	movaps	20(%ecx), %xmm3
-	movaps	%xmm3, %xmm6
-	movaps	36(%ecx), %xmm4
-	movaps	%xmm4, %xmm7
-	movaps	52(%ecx), %xmm5
-	pminub	%xmm2, %xmm6
-	pminub	%xmm5, %xmm7
-	pminub	%xmm6, %xmm7
-	pcmpeqd	%xmm0, %xmm7
-	pmovmskb %xmm7, %eax
-	movaps	%xmm5, %xmm7
-	palignr	$12, %xmm4, %xmm5
-	palignr	$12, %xmm3, %xmm4
-	test	%eax, %eax
-	jnz	L(Shl12Start)
-
-	palignr	$12, %xmm2, %xmm3
-	lea	64(%ecx), %ecx
-	palignr	$12, %xmm1, %xmm2
-	movaps	%xmm7, %xmm1
-	movaps	%xmm5, 48(%edx)
-	movaps	%xmm4, 32(%edx)
-	movaps	%xmm3, 16(%edx)
-	movaps	%xmm2, (%edx)
-	lea	64(%edx), %edx
-	jmp	L(Shl12LoopStart)
-
-L(Shl12LoopExit):
-	movl	(%ecx), %esi
-	movl	%esi, (%edx)
-	mov	$4, %esi
-
-	.p2align 4
-L(CopyFrom1To16Bytes):
-	add	%esi, %edx
-	add	%esi, %ecx
-
-	POP	(%esi)
-	test	%al, %al
-	jz	L(ExitHigh)
-	test	$0x01, %al
-	jnz	L(Exit4)
-L(Exit8):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	%edi, %eax
-	RETURN
-
-	.p2align 4
-L(ExitHigh):
-	test	$0x01, %ah
-	jnz	L(Exit12)
-L(Exit16):
-	movdqu	(%ecx), %xmm0
-	movdqu	%xmm0, (%edx)
-	movl	%edi, %eax
-	RETURN
-
-	.p2align 4
-L(Exit4):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	%edi, %eax
-	RETURN
-
-	.p2align 4
-L(Exit12):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-	movl	%edi, %eax
-	RETURN
-
-CFI_POP	(%edi)
-
-	.p2align 4
-L(ExitTail4):
-	movl	(%ecx), %eax
-	movl	%eax, (%edx)
-	movl	%edx, %eax
-	ret
-
-	.p2align 4
-L(ExitTail8):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	%edx, %eax
-	ret
-
-	.p2align 4
-L(ExitTail12):
-	movlpd	(%ecx), %xmm0
-	movlpd	%xmm0, (%edx)
-	movl	8(%ecx), %eax
-	movl	%eax, 8(%edx)
-	movl	%edx, %eax
-	ret
-
-	.p2align 4
-L(ExitTail16):
-	movdqu	(%ecx), %xmm0
-	movdqu	%xmm0, (%edx)
-	movl	%edx, %eax
-	ret
-
-#ifndef USE_AS_WCSCAT
-END (wcscpy)
-#endif
diff --git a/libc/arch-x86/string/ssse3-wmemcmp-atom.S b/libc/arch-x86/string/ssse3-wmemcmp-atom.S
deleted file mode 100644
index a81b78b..0000000
--- a/libc/arch-x86/string/ssse3-wmemcmp-atom.S
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
-Copyright (c) 2011, 2012, 2013 Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-    * this list of conditions and the following disclaimer.
-
-    * Redistributions in binary form must reproduce the above copyright notice,
-    * this list of conditions and the following disclaimer in the documentation
-    * and/or other materials provided with the distribution.
-
-    * Neither the name of Intel Corporation nor the names of its contributors
-    * may be used to endorse or promote products derived from this software
-    * without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*/
-
-#define MEMCMP  wmemcmp_atom
-
-#define USE_WCHAR
-#define USE_AS_WMEMCMP 1
-#include "ssse3-memcmp-atom.S"
diff --git a/libc/bionic/__libc_init_main_thread.cpp b/libc/bionic/__libc_init_main_thread.cpp
index 1b539f2..0d557f1 100644
--- a/libc/bionic/__libc_init_main_thread.cpp
+++ b/libc/bionic/__libc_init_main_thread.cpp
@@ -44,7 +44,7 @@
 // Declared in "private/bionic_ssp.h".
 uintptr_t __stack_chk_guard = 0;
 
-static pthread_internal_t main_thread;
+BIONIC_USED_BEFORE_LINKER_RELOCATES static pthread_internal_t main_thread;
 
 // Setup for the main thread. For dynamic executables, this is called by the
 // linker _before_ libc is mapped in memory. This means that all writes to
diff --git a/libc/bionic/android_mallopt.cpp b/libc/bionic/android_mallopt.cpp
new file mode 100644
index 0000000..79e4072
--- /dev/null
+++ b/libc/bionic/android_mallopt.cpp
@@ -0,0 +1,146 @@
+/*
+ * Copyright (C) 2009 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <errno.h>
+#include <stdatomic.h>
+
+#include <platform/bionic/malloc.h>
+#include <private/bionic_globals.h>
+
+#include "gwp_asan_wrappers.h"
+#include "malloc_limit.h"
+
+#if !defined(LIBC_STATIC)
+#include <stdio.h>
+
+#include <private/bionic_defs.h>
+
+#include "malloc_heapprofd.h"
+
+extern bool gZygoteChild;
+extern _Atomic bool gZygoteChildProfileable;
+
+bool WriteMallocLeakInfo(FILE* fp);
+bool GetMallocLeakInfo(android_mallopt_leak_info_t* leak_info);
+bool FreeMallocLeakInfo(android_mallopt_leak_info_t* leak_info);
+#endif
+
+// =============================================================================
+// Platform-internal mallopt variant.
+// =============================================================================
+#if !defined(LIBC_STATIC)
+__BIONIC_WEAK_FOR_NATIVE_BRIDGE
+#endif
+extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
+  // Functionality available in both static and dynamic libc.
+  if (opcode == M_GET_DECAY_TIME_ENABLED) {
+    if (arg == nullptr || arg_size != sizeof(bool)) {
+      errno = EINVAL;
+      return false;
+    }
+    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
+    return true;
+  }
+  if (opcode == M_INITIALIZE_GWP_ASAN) {
+    if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
+      errno = EINVAL;
+      return false;
+    }
+
+    return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
+  }
+  if (opcode == M_MEMTAG_STACK_IS_ON) {
+    if (arg == nullptr || arg_size != sizeof(bool)) {
+      errno = EINVAL;
+      return false;
+    }
+    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
+    return true;
+  }
+  if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
+    return LimitEnable(arg, arg_size);
+  }
+
+#if defined(LIBC_STATIC)
+  errno = ENOTSUP;
+  return false;
+#else
+  if (opcode == M_SET_ZYGOTE_CHILD) {
+    if (arg != nullptr || arg_size != 0) {
+      errno = EINVAL;
+      return false;
+    }
+    gZygoteChild = true;
+    return true;
+  }
+  if (opcode == M_INIT_ZYGOTE_CHILD_PROFILING) {
+    if (arg != nullptr || arg_size != 0) {
+      errno = EINVAL;
+      return false;
+    }
+    atomic_store_explicit(&gZygoteChildProfileable, true, memory_order_release);
+    // Also check if heapprofd should start profiling from app startup.
+    HeapprofdInitZygoteChildProfiling();
+    return true;
+  }
+  if (opcode == M_GET_PROCESS_PROFILEABLE) {
+    if (arg == nullptr || arg_size != sizeof(bool)) {
+      errno = EINVAL;
+      return false;
+    }
+    // Native processes are considered profileable. Zygote children are considered
+    // profileable only when appropriately tagged.
+    *reinterpret_cast<bool*>(arg) =
+        !gZygoteChild || atomic_load_explicit(&gZygoteChildProfileable, memory_order_acquire);
+    return true;
+  }
+  if (opcode == M_WRITE_MALLOC_LEAK_INFO_TO_FILE) {
+    if (arg == nullptr || arg_size != sizeof(FILE*)) {
+      errno = EINVAL;
+      return false;
+    }
+    return WriteMallocLeakInfo(reinterpret_cast<FILE*>(arg));
+  }
+  if (opcode == M_GET_MALLOC_LEAK_INFO) {
+    if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
+      errno = EINVAL;
+      return false;
+    }
+    return GetMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
+  }
+  if (opcode == M_FREE_MALLOC_LEAK_INFO) {
+    if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
+      errno = EINVAL;
+      return false;
+    }
+    return FreeMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
+  }
+  // Try heapprofd's mallopt, as it handles options not covered here.
+  return HeapprofdMallopt(opcode, arg, arg_size);
+#endif
+}
diff --git a/libc/bionic/bionic_call_ifunc_resolver.cpp b/libc/bionic/bionic_call_ifunc_resolver.cpp
index e44d998..d5a812c 100644
--- a/libc/bionic/bionic_call_ifunc_resolver.cpp
+++ b/libc/bionic/bionic_call_ifunc_resolver.cpp
@@ -31,6 +31,7 @@
 #include <sys/hwprobe.h>
 #include <sys/ifunc.h>
 
+#include "bionic/macros.h"
 #include "private/bionic_auxv.h"
 
 // This code is called in the linker before it has been relocated, so minimize calls into other
@@ -40,8 +41,8 @@
 ElfW(Addr) __bionic_call_ifunc_resolver(ElfW(Addr) resolver_addr) {
 #if defined(__aarch64__)
   typedef ElfW(Addr) (*ifunc_resolver_t)(uint64_t, __ifunc_arg_t*);
-  static __ifunc_arg_t arg;
-  static bool initialized = false;
+  BIONIC_USED_BEFORE_LINKER_RELOCATES static __ifunc_arg_t arg;
+  BIONIC_USED_BEFORE_LINKER_RELOCATES static bool initialized = false;
   if (!initialized) {
     initialized = true;
     arg._size = sizeof(__ifunc_arg_t);
diff --git a/libc/bionic/elf_note.cpp b/libc/bionic/elf_note.cpp
index d5cd5de..9cc6b21 100644
--- a/libc/bionic/elf_note.cpp
+++ b/libc/bionic/elf_note.cpp
@@ -38,31 +38,34 @@
     return false;
   }
 
+  size_t note_name_len = strlen(note_name) + 1;
+
   ElfW(Addr) p = note_addr;
   ElfW(Addr) note_end = p + phdr_note->p_memsz;
-
   while (p + sizeof(ElfW(Nhdr)) <= note_end) {
+    // Parse the note and check it's structurally valid.
     const ElfW(Nhdr)* note = reinterpret_cast<const ElfW(Nhdr)*>(p);
     p += sizeof(ElfW(Nhdr));
     const char* name = reinterpret_cast<const char*>(p);
-    p += align_up(note->n_namesz, 4);
+    if (__builtin_add_overflow(p, align_up(note->n_namesz, 4), &p)) {
+      return false;
+    }
     const char* desc = reinterpret_cast<const char*>(p);
-    p += align_up(note->n_descsz, 4);
+    if (__builtin_add_overflow(p, align_up(note->n_descsz, 4), &p)) {
+      return false;
+    }
     if (p > note_end) {
-      break;
-    }
-    if (note->n_type != note_type) {
-      continue;
-    }
-    size_t note_name_len = strlen(note_name) + 1;
-    if (note->n_namesz != note_name_len || strncmp(note_name, name, note_name_len) != 0) {
-      break;
+      return false;
     }
 
-    *note_hdr = note;
-    *note_desc = desc;
-
-    return true;
+    // Is this the note we're looking for?
+    if (note->n_type == note_type &&
+        note->n_namesz == note_name_len &&
+        strncmp(note_name, name, note_name_len) == 0) {
+      *note_hdr = note;
+      *note_desc = desc;
+      return true;
+    }
   }
   return false;
 }
diff --git a/libc/bionic/fts.c b/libc/bionic/fts.c
index 1287267..c36835e 100644
--- a/libc/bionic/fts.c
+++ b/libc/bionic/fts.c
@@ -29,7 +29,6 @@
  * SUCH DAMAGE.
  */
 
-#include <sys/param.h>	/* ALIGN */
 #include <sys/stat.h>
 
 #include <dirent.h>
@@ -37,6 +36,7 @@
 #include <fcntl.h>
 #include <fts.h>
 #include <limits.h>
+#include <stdalign.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
@@ -912,10 +912,14 @@
 	 * be careful that the stat structure is reasonably aligned.  Since the
 	 * fts_name field is declared to be of size 1, the fts_name pointer is
 	 * namelen + 2 before the first possible address of the stat structure.
+	 *
+	 * We can't use the same trick FreeBSD uses here because our fts_name
+	 * is a char[1] rather than a char*. This is also the reason we don't
+	 * need to say `namelen + 1`. We just assume the worst alignment.
 	 */
 	len = sizeof(FTSENT) + namelen;
 	if (!ISSET(FTS_NOSTAT))
-		len += sizeof(struct stat) + ALIGNBYTES;
+		len += alignof(struct stat) + sizeof(struct stat);
 	if ((p = calloc(1, len)) == NULL)
 		return (NULL);
 
@@ -923,7 +927,7 @@
 	p->fts_namelen = namelen;
 	p->fts_instr = FTS_NOINSTR;
 	if (!ISSET(FTS_NOSTAT))
-		p->fts_statp = (struct stat *)ALIGN(p->fts_name + namelen + 2);
+		p->fts_statp = (struct stat *)__builtin_align_up(p->fts_name + namelen + 2, alignof(struct stat));
 	memcpy(p->fts_name, name, namelen);
 
 	return (p);
diff --git a/libc/bionic/libc_init_mte.cpp b/libc/bionic/libc_init_mte.cpp
new file mode 100644
index 0000000..3c8ef7d
--- /dev/null
+++ b/libc/bionic/libc_init_mte.cpp
@@ -0,0 +1,325 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <android/api-level.h>
+#include <elf.h>
+#include <errno.h>
+#include <malloc.h>
+#include <signal.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/auxv.h>
+#include <sys/mman.h>
+
+#include "async_safe/log.h"
+#include "heap_tagging.h"
+#include "libc_init_common.h"
+#include "platform/bionic/macros.h"
+#include "platform/bionic/mte.h"
+#include "platform/bionic/page.h"
+#include "platform/bionic/reserved_signals.h"
+#include "private/KernelArgumentBlock.h"
+#include "private/bionic_asm.h"
+#include "private/bionic_asm_note.h"
+#include "private/bionic_call_ifunc_resolver.h"
+#include "private/bionic_elf_tls.h"
+#include "private/bionic_globals.h"
+#include "private/bionic_tls.h"
+#include "private/elf_note.h"
+#include "pthread_internal.h"
+#include "sys/system_properties.h"
+#include "sysprop_helpers.h"
+
+#ifdef __aarch64__
+extern "C" const char* __gnu_basename(const char* path);
+
+static HeapTaggingLevel __get_memtag_level_from_note(const ElfW(Phdr) * phdr_start, size_t phdr_ct,
+                                                     const ElfW(Addr) load_bias, bool* stack) {
+  const ElfW(Nhdr) * note;
+  const char* desc;
+  if (!__find_elf_note(NT_ANDROID_TYPE_MEMTAG, "Android", phdr_start, phdr_ct, &note, &desc,
+                       load_bias)) {
+    return M_HEAP_TAGGING_LEVEL_TBI;
+  }
+
+  // Previously (in Android 12), if the note was != 4 bytes, we check-failed
+  // here. Let's be more permissive to allow future expansion.
+  if (note->n_descsz < 4) {
+    async_safe_fatal("unrecognized android.memtag note: n_descsz = %d, expected >= 4",
+                     note->n_descsz);
+  }
+
+  // `desc` is always aligned due to ELF requirements, enforced in __find_elf_note().
+  ElfW(Word) note_val = *reinterpret_cast<const ElfW(Word)*>(desc);
+  *stack = (note_val & NT_MEMTAG_STACK) != 0;
+
+  // Warning: In Android 12, any value outside of bits [0..3] resulted in a check-fail.
+  if (!(note_val & (NT_MEMTAG_HEAP | NT_MEMTAG_STACK))) {
+    async_safe_format_log(ANDROID_LOG_INFO, "libc",
+                          "unrecognised memtag note_val did not specificy heap or stack: %u",
+                          note_val);
+    return M_HEAP_TAGGING_LEVEL_TBI;
+  }
+
+  unsigned mode = note_val & NT_MEMTAG_LEVEL_MASK;
+  switch (mode) {
+    case NT_MEMTAG_LEVEL_NONE:
+      // Note, previously (in Android 12), NT_MEMTAG_LEVEL_NONE was
+      // NT_MEMTAG_LEVEL_DEFAULT, which implied SYNC mode. This was never used
+      // by anyone, but we note it (heh) here for posterity, in case the zero
+      // level becomes meaningful, and binaries with this note can be executed
+      // on Android 12 devices.
+      return M_HEAP_TAGGING_LEVEL_TBI;
+    case NT_MEMTAG_LEVEL_ASYNC:
+      return M_HEAP_TAGGING_LEVEL_ASYNC;
+    case NT_MEMTAG_LEVEL_SYNC:
+    default:
+      // We allow future extensions to specify mode 3 (currently unused), with
+      // the idea that it might be used for ASYMM mode or something else. On
+      // this version of Android, it falls back to SYNC mode.
+      return M_HEAP_TAGGING_LEVEL_SYNC;
+  }
+}
+
+// Returns true if there's an environment setting (either sysprop or env var)
+// that should overwrite the ELF note, and places the equivalent heap tagging
+// level into *level.
+static bool get_environment_memtag_setting(HeapTaggingLevel* level) {
+  static const char kMemtagPrognameSyspropPrefix[] = "arm64.memtag.process.";
+  static const char kMemtagGlobalSysprop[] = "persist.arm64.memtag.default";
+  static const char kMemtagOverrideSyspropPrefix[] =
+      "persist.device_config.memory_safety_native.mode_override.process.";
+
+  const char* progname = __libc_shared_globals()->init_progname;
+  if (progname == nullptr) return false;
+
+  const char* basename = __gnu_basename(progname);
+
+  char options_str[PROP_VALUE_MAX];
+  char sysprop_name[512];
+  async_safe_format_buffer(sysprop_name, sizeof(sysprop_name), "%s%s", kMemtagPrognameSyspropPrefix,
+                           basename);
+  char remote_sysprop_name[512];
+  async_safe_format_buffer(remote_sysprop_name, sizeof(remote_sysprop_name), "%s%s",
+                           kMemtagOverrideSyspropPrefix, basename);
+  const char* sys_prop_names[] = {sysprop_name, remote_sysprop_name, kMemtagGlobalSysprop};
+
+  if (!get_config_from_env_or_sysprops("MEMTAG_OPTIONS", sys_prop_names, arraysize(sys_prop_names),
+                                       options_str, sizeof(options_str))) {
+    return false;
+  }
+
+  if (strcmp("sync", options_str) == 0) {
+    *level = M_HEAP_TAGGING_LEVEL_SYNC;
+  } else if (strcmp("async", options_str) == 0) {
+    *level = M_HEAP_TAGGING_LEVEL_ASYNC;
+  } else if (strcmp("off", options_str) == 0) {
+    *level = M_HEAP_TAGGING_LEVEL_TBI;
+  } else {
+    async_safe_format_log(
+        ANDROID_LOG_ERROR, "libc",
+        "unrecognized memtag level: \"%s\" (options are \"sync\", \"async\", or \"off\").",
+        options_str);
+    return false;
+  }
+
+  return true;
+}
+
+// Returns the initial heap tagging level. Note: This function will never return
+// M_HEAP_TAGGING_LEVEL_NONE, if MTE isn't enabled for this process we enable
+// M_HEAP_TAGGING_LEVEL_TBI.
+static HeapTaggingLevel __get_tagging_level(const memtag_dynamic_entries_t* memtag_dynamic_entries,
+                                            const void* phdr_start, size_t phdr_ct,
+                                            uintptr_t load_bias, bool* stack) {
+  HeapTaggingLevel level = M_HEAP_TAGGING_LEVEL_TBI;
+
+  // If the dynamic entries exist, use those. Otherwise, fall back to the old
+  // Android note, which is still used for fully static executables. When
+  // -fsanitize=memtag* is used in newer toolchains, currently both the dynamic
+  // entries and the old note are created, but we'd expect to move to just the
+  // dynamic entries for dynamically linked executables in the future. In
+  // addition, there's still some cleanup of the build system (that uses a
+  // manually-constructed note) needed. For more information about the dynamic
+  // entries, see:
+  // https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#dynamic-section
+  if (memtag_dynamic_entries && memtag_dynamic_entries->has_memtag_mode) {
+    switch (memtag_dynamic_entries->memtag_mode) {
+      case 0:
+        level = M_HEAP_TAGGING_LEVEL_SYNC;
+        break;
+      case 1:
+        level = M_HEAP_TAGGING_LEVEL_ASYNC;
+        break;
+      default:
+        async_safe_format_log(ANDROID_LOG_INFO, "libc",
+                              "unrecognised DT_AARCH64_MEMTAG_MODE value: %u",
+                              memtag_dynamic_entries->memtag_mode);
+    }
+    *stack = memtag_dynamic_entries->memtag_stack;
+  } else {
+    level = __get_memtag_level_from_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct,
+                                         load_bias, stack);
+  }
+
+  // We can't short-circuit the environment override, as `stack` is still inherited from the
+  // binary's settings.
+  get_environment_memtag_setting(&level);
+  return level;
+}
+
+static void __enable_mte_signal_handler(int, siginfo_t* info, void*) {
+  if (info->si_code != SI_TIMER) {
+    async_safe_format_log(ANDROID_LOG_ERROR, "libc", "Got BIONIC_ENABLE_MTE not from SI_TIMER");
+    return;
+  }
+  int tagged_addr_ctrl = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);
+  if (tagged_addr_ctrl < 0) {
+    async_safe_fatal("failed to PR_GET_TAGGED_ADDR_CTRL: %m");
+  }
+  if ((tagged_addr_ctrl & PR_MTE_TCF_MASK) != PR_MTE_TCF_NONE) {
+    return;
+  }
+  async_safe_format_log(ANDROID_LOG_INFO, "libc",
+                        "Re-enabling MTE, value: %x (tagged_addr_ctrl %lu)",
+                        info->si_value.sival_int, info->si_value.sival_int & PR_MTE_TCF_MASK);
+  tagged_addr_ctrl =
+      (tagged_addr_ctrl & ~PR_MTE_TCF_MASK) | (info->si_value.sival_int & PR_MTE_TCF_MASK);
+  if (prctl(PR_SET_TAGGED_ADDR_CTRL, tagged_addr_ctrl, 0, 0, 0) < 0) {
+    async_safe_fatal("failed to PR_SET_TAGGED_ADDR_CTRL %d: %m", tagged_addr_ctrl);
+  }
+}
+
+static int64_t __get_memtag_upgrade_secs() {
+  char* env = getenv("BIONIC_MEMTAG_UPGRADE_SECS");
+  if (!env) return 0;
+  int64_t timed_upgrade = 0;
+  static const char kAppProcessName[] = "app_process64";
+  const char* progname = __libc_shared_globals()->init_progname;
+  progname = progname ? __gnu_basename(progname) : nullptr;
+  // disable timed upgrade for zygote, as the thread spawned will violate the requirement
+  // that it be single-threaded.
+  if (!progname || strncmp(progname, kAppProcessName, sizeof(kAppProcessName)) != 0) {
+    char* endptr;
+    timed_upgrade = strtoll(env, &endptr, 10);
+    if (*endptr != '\0' || timed_upgrade < 0) {
+      async_safe_format_log(ANDROID_LOG_ERROR, "libc",
+                            "Invalid value for BIONIC_MEMTAG_UPGRADE_SECS: %s", env);
+      timed_upgrade = 0;
+    }
+  }
+  // Make sure that this does not get passed to potential processes inheriting
+  // this environment.
+  unsetenv("BIONIC_MEMTAG_UPGRADE_SECS");
+  return timed_upgrade;
+}
+
+// Figure out the desired memory tagging mode (sync/async, heap/globals/stack) for this executable.
+// This function is called from the linker before the main executable is relocated.
+__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte(
+    const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start, size_t phdr_ct,
+    uintptr_t load_bias) {
+  bool memtag_stack = false;
+  HeapTaggingLevel level =
+      __get_tagging_level(memtag_dynamic_entries, phdr_start, phdr_ct, load_bias, &memtag_stack);
+  if (memtag_stack) __libc_shared_globals()->initial_memtag_stack_abi = true;
+
+  if (int64_t timed_upgrade = __get_memtag_upgrade_secs()) {
+    if (level == M_HEAP_TAGGING_LEVEL_ASYNC) {
+      async_safe_format_log(ANDROID_LOG_INFO, "libc",
+                            "Attempting timed MTE upgrade from async to sync.");
+      __libc_shared_globals()->heap_tagging_upgrade_timer_sec = timed_upgrade;
+      level = M_HEAP_TAGGING_LEVEL_SYNC;
+    } else if (level != M_HEAP_TAGGING_LEVEL_SYNC) {
+      async_safe_format_log(ANDROID_LOG_ERROR, "libc",
+                            "Requested timed MTE upgrade from invalid %s to sync. Ignoring.",
+                            DescribeTaggingLevel(level));
+    }
+  }
+  if (level == M_HEAP_TAGGING_LEVEL_SYNC || level == M_HEAP_TAGGING_LEVEL_ASYNC) {
+    unsigned long prctl_arg = PR_TAGGED_ADDR_ENABLE | PR_MTE_TAG_SET_NONZERO;
+    prctl_arg |= (level == M_HEAP_TAGGING_LEVEL_SYNC) ? PR_MTE_TCF_SYNC : PR_MTE_TCF_ASYNC;
+
+    // When entering ASYNC mode, specify that we want to allow upgrading to SYNC by OR'ing in the
+    // SYNC flag. But if the kernel doesn't support specifying multiple TCF modes, fall back to
+    // specifying a single mode.
+    if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg | PR_MTE_TCF_SYNC, 0, 0, 0) == 0 ||
+        prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg, 0, 0, 0) == 0) {
+      __libc_shared_globals()->initial_heap_tagging_level = level;
+
+      struct sigaction action = {};
+      action.sa_flags = SA_SIGINFO | SA_RESTART;
+      action.sa_sigaction = __enable_mte_signal_handler;
+      sigaction(BIONIC_ENABLE_MTE, &action, nullptr);
+      return;
+    }
+  }
+
+  // MTE was either not enabled, or wasn't supported on this device. Try and use
+  // TBI.
+  if (prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0) == 0) {
+    __libc_shared_globals()->initial_heap_tagging_level = M_HEAP_TAGGING_LEVEL_TBI;
+  }
+  // We did not enable MTE, so we do not need to arm the upgrade timer.
+  __libc_shared_globals()->heap_tagging_upgrade_timer_sec = 0;
+}
+
+// Figure out whether we need to map the stack as PROT_MTE.
+// For dynamic executables, this has to be called after loading all
+// DT_NEEDED libraries, in case one of them needs stack MTE.
+__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte_stack(void* stack_top) {
+  if (!__libc_shared_globals()->initial_memtag_stack_abi) {
+    return;
+  }
+
+  // Even if the device doesn't support MTE, we have to allocate stack
+  // history buffers for code compiled for stack MTE. That is because the
+  // codegen expects a buffer to be present in TLS_SLOT_STACK_MTE either
+  // way.
+  __get_bionic_tcb()->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, nullptr);
+
+  if (__libc_mte_enabled()) {
+    __libc_shared_globals()->initial_memtag_stack = true;
+    void* pg_start = reinterpret_cast<void*>(page_start(reinterpret_cast<uintptr_t>(stack_top)));
+    if (mprotect(pg_start, page_size(), PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN)) {
+      async_safe_fatal("error: failed to set PROT_MTE on main thread stack: %m");
+    }
+  }
+}
+
+#else   // __aarch64__
+void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t, uintptr_t) {}
+void __libc_init_mte_stack(void*) {}
+#endif  // __aarch64__
+
+bool __libc_mte_enabled() {
+  HeapTaggingLevel lvl = __libc_shared_globals()->initial_heap_tagging_level;
+  return lvl == M_HEAP_TAGGING_LEVEL_SYNC || lvl == M_HEAP_TAGGING_LEVEL_ASYNC;
+}
diff --git a/libc/bionic/libc_init_static.cpp b/libc/bionic/libc_init_static.cpp
index 7c46113..cd96375 100644
--- a/libc/bionic/libc_init_static.cpp
+++ b/libc/bionic/libc_init_static.cpp
@@ -157,260 +157,6 @@
 
   layout.finish_layout();
 }
-
-#ifdef __aarch64__
-static HeapTaggingLevel __get_memtag_level_from_note(const ElfW(Phdr) * phdr_start, size_t phdr_ct,
-                                                     const ElfW(Addr) load_bias, bool* stack) {
-  const ElfW(Nhdr) * note;
-  const char* desc;
-  if (!__find_elf_note(NT_ANDROID_TYPE_MEMTAG, "Android", phdr_start, phdr_ct, &note, &desc,
-                       load_bias)) {
-    return M_HEAP_TAGGING_LEVEL_TBI;
-  }
-
-  // Previously (in Android 12), if the note was != 4 bytes, we check-failed
-  // here. Let's be more permissive to allow future expansion.
-  if (note->n_descsz < 4) {
-    async_safe_fatal("unrecognized android.memtag note: n_descsz = %d, expected >= 4",
-                     note->n_descsz);
-  }
-
-  // `desc` is always aligned due to ELF requirements, enforced in __find_elf_note().
-  ElfW(Word) note_val = *reinterpret_cast<const ElfW(Word)*>(desc);
-  *stack = (note_val & NT_MEMTAG_STACK) != 0;
-
-  // Warning: In Android 12, any value outside of bits [0..3] resulted in a check-fail.
-  if (!(note_val & (NT_MEMTAG_HEAP | NT_MEMTAG_STACK))) {
-    async_safe_format_log(ANDROID_LOG_INFO, "libc",
-                          "unrecognised memtag note_val did not specificy heap or stack: %u",
-                          note_val);
-    return M_HEAP_TAGGING_LEVEL_TBI;
-  }
-
-  unsigned mode = note_val & NT_MEMTAG_LEVEL_MASK;
-  switch (mode) {
-    case NT_MEMTAG_LEVEL_NONE:
-      // Note, previously (in Android 12), NT_MEMTAG_LEVEL_NONE was
-      // NT_MEMTAG_LEVEL_DEFAULT, which implied SYNC mode. This was never used
-      // by anyone, but we note it (heh) here for posterity, in case the zero
-      // level becomes meaningful, and binaries with this note can be executed
-      // on Android 12 devices.
-      return M_HEAP_TAGGING_LEVEL_TBI;
-    case NT_MEMTAG_LEVEL_ASYNC:
-      return M_HEAP_TAGGING_LEVEL_ASYNC;
-    case NT_MEMTAG_LEVEL_SYNC:
-    default:
-      // We allow future extensions to specify mode 3 (currently unused), with
-      // the idea that it might be used for ASYMM mode or something else. On
-      // this version of Android, it falls back to SYNC mode.
-      return M_HEAP_TAGGING_LEVEL_SYNC;
-  }
-}
-
-// Returns true if there's an environment setting (either sysprop or env var)
-// that should overwrite the ELF note, and places the equivalent heap tagging
-// level into *level.
-static bool get_environment_memtag_setting(HeapTaggingLevel* level) {
-  static const char kMemtagPrognameSyspropPrefix[] = "arm64.memtag.process.";
-  static const char kMemtagGlobalSysprop[] = "persist.arm64.memtag.default";
-  static const char kMemtagOverrideSyspropPrefix[] =
-      "persist.device_config.memory_safety_native.mode_override.process.";
-
-  const char* progname = __libc_shared_globals()->init_progname;
-  if (progname == nullptr) return false;
-
-  const char* basename = __gnu_basename(progname);
-
-  char options_str[PROP_VALUE_MAX];
-  char sysprop_name[512];
-  async_safe_format_buffer(sysprop_name, sizeof(sysprop_name), "%s%s", kMemtagPrognameSyspropPrefix,
-                           basename);
-  char remote_sysprop_name[512];
-  async_safe_format_buffer(remote_sysprop_name, sizeof(remote_sysprop_name), "%s%s",
-                           kMemtagOverrideSyspropPrefix, basename);
-  const char* sys_prop_names[] = {sysprop_name, remote_sysprop_name, kMemtagGlobalSysprop};
-
-  if (!get_config_from_env_or_sysprops("MEMTAG_OPTIONS", sys_prop_names, arraysize(sys_prop_names),
-                                       options_str, sizeof(options_str))) {
-    return false;
-  }
-
-  if (strcmp("sync", options_str) == 0) {
-    *level = M_HEAP_TAGGING_LEVEL_SYNC;
-  } else if (strcmp("async", options_str) == 0) {
-    *level = M_HEAP_TAGGING_LEVEL_ASYNC;
-  } else if (strcmp("off", options_str) == 0) {
-    *level = M_HEAP_TAGGING_LEVEL_TBI;
-  } else {
-    async_safe_format_log(
-        ANDROID_LOG_ERROR, "libc",
-        "unrecognized memtag level: \"%s\" (options are \"sync\", \"async\", or \"off\").",
-        options_str);
-    return false;
-  }
-
-  return true;
-}
-
-// Returns the initial heap tagging level. Note: This function will never return
-// M_HEAP_TAGGING_LEVEL_NONE, if MTE isn't enabled for this process we enable
-// M_HEAP_TAGGING_LEVEL_TBI.
-static HeapTaggingLevel __get_tagging_level(const memtag_dynamic_entries_t* memtag_dynamic_entries,
-                                            const void* phdr_start, size_t phdr_ct,
-                                            uintptr_t load_bias, bool* stack) {
-  HeapTaggingLevel level = M_HEAP_TAGGING_LEVEL_TBI;
-
-  // If the dynamic entries exist, use those. Otherwise, fall back to the old
-  // Android note, which is still used for fully static executables. When
-  // -fsanitize=memtag* is used in newer toolchains, currently both the dynamic
-  // entries and the old note are created, but we'd expect to move to just the
-  // dynamic entries for dynamically linked executables in the future. In
-  // addition, there's still some cleanup of the build system (that uses a
-  // manually-constructed note) needed. For more information about the dynamic
-  // entries, see:
-  // https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#dynamic-section
-  if (memtag_dynamic_entries && memtag_dynamic_entries->has_memtag_mode) {
-    switch (memtag_dynamic_entries->memtag_mode) {
-      case 0:
-        level = M_HEAP_TAGGING_LEVEL_SYNC;
-        break;
-      case 1:
-        level = M_HEAP_TAGGING_LEVEL_ASYNC;
-        break;
-      default:
-        async_safe_format_log(ANDROID_LOG_INFO, "libc",
-                              "unrecognised DT_AARCH64_MEMTAG_MODE value: %u",
-                              memtag_dynamic_entries->memtag_mode);
-    }
-    *stack = memtag_dynamic_entries->memtag_stack;
-  } else {
-    level = __get_memtag_level_from_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct,
-                                         load_bias, stack);
-  }
-
-  // We can't short-circuit the environment override, as `stack` is still inherited from the
-  // binary's settings.
-  get_environment_memtag_setting(&level);
-  return level;
-}
-
-static void __enable_mte_signal_handler(int, siginfo_t* info, void*) {
-  if (info->si_code != SI_TIMER) {
-    async_safe_format_log(ANDROID_LOG_ERROR, "libc", "Got BIONIC_ENABLE_MTE not from SI_TIMER");
-    return;
-  }
-  int tagged_addr_ctrl = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);
-  if (tagged_addr_ctrl < 0) {
-    async_safe_fatal("failed to PR_GET_TAGGED_ADDR_CTRL: %m");
-  }
-  if ((tagged_addr_ctrl & PR_MTE_TCF_MASK) != PR_MTE_TCF_NONE) {
-    return;
-  }
-  async_safe_format_log(ANDROID_LOG_INFO, "libc",
-                        "Re-enabling MTE, value: %x (tagged_addr_ctrl %lu)",
-                        info->si_value.sival_int, info->si_value.sival_int & PR_MTE_TCF_MASK);
-  tagged_addr_ctrl =
-      (tagged_addr_ctrl & ~PR_MTE_TCF_MASK) | (info->si_value.sival_int & PR_MTE_TCF_MASK);
-  if (prctl(PR_SET_TAGGED_ADDR_CTRL, tagged_addr_ctrl, 0, 0, 0) < 0) {
-    async_safe_fatal("failed to PR_SET_TAGGED_ADDR_CTRL %d: %m", tagged_addr_ctrl);
-  }
-}
-
-static int64_t __get_memtag_upgrade_secs() {
-  char* env = getenv("BIONIC_MEMTAG_UPGRADE_SECS");
-  if (!env) return 0;
-  int64_t timed_upgrade = 0;
-  static const char kAppProcessName[] = "app_process64";
-  const char* progname = __libc_shared_globals()->init_progname;
-  progname = progname ? __gnu_basename(progname) : nullptr;
-  // disable timed upgrade for zygote, as the thread spawned will violate the requirement
-  // that it be single-threaded.
-  if (!progname || strncmp(progname, kAppProcessName, sizeof(kAppProcessName)) != 0) {
-    char* endptr;
-    timed_upgrade = strtoll(env, &endptr, 10);
-    if (*endptr != '\0' || timed_upgrade < 0) {
-      async_safe_format_log(ANDROID_LOG_ERROR, "libc",
-                            "Invalid value for BIONIC_MEMTAG_UPGRADE_SECS: %s", env);
-      timed_upgrade = 0;
-    }
-  }
-  // Make sure that this does not get passed to potential processes inheriting
-  // this environment.
-  unsetenv("BIONIC_MEMTAG_UPGRADE_SECS");
-  return timed_upgrade;
-}
-
-// Figure out the desired memory tagging mode (sync/async, heap/globals/stack) for this executable.
-// This function is called from the linker before the main executable is relocated.
-__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte(
-    const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start, size_t phdr_ct,
-    uintptr_t load_bias, void* stack_top) {
-  bool memtag_stack = false;
-  HeapTaggingLevel level =
-      __get_tagging_level(memtag_dynamic_entries, phdr_start, phdr_ct, load_bias, &memtag_stack);
-  // initial_memtag_stack is used by the linker (in linker.cpp) to communicate than any library
-  // linked by this executable enables memtag-stack.
-  // memtag_stack is also set for static executables if they request memtag stack via the note,
-  // in which case it will differ from initial_memtag_stack.
-  if (__libc_shared_globals()->initial_memtag_stack || memtag_stack) {
-    memtag_stack = true;
-    __libc_shared_globals()->initial_memtag_stack_abi = true;
-    __get_bionic_tcb()->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, nullptr);
-  }
-  if (int64_t timed_upgrade = __get_memtag_upgrade_secs()) {
-    if (level == M_HEAP_TAGGING_LEVEL_ASYNC) {
-      async_safe_format_log(ANDROID_LOG_INFO, "libc",
-                            "Attempting timed MTE upgrade from async to sync.");
-      __libc_shared_globals()->heap_tagging_upgrade_timer_sec = timed_upgrade;
-      level = M_HEAP_TAGGING_LEVEL_SYNC;
-    } else if (level != M_HEAP_TAGGING_LEVEL_SYNC) {
-      async_safe_format_log(
-          ANDROID_LOG_ERROR, "libc",
-          "Requested timed MTE upgrade from invalid %s to sync. Ignoring.",
-          DescribeTaggingLevel(level));
-    }
-  }
-  if (level == M_HEAP_TAGGING_LEVEL_SYNC || level == M_HEAP_TAGGING_LEVEL_ASYNC) {
-    unsigned long prctl_arg = PR_TAGGED_ADDR_ENABLE | PR_MTE_TAG_SET_NONZERO;
-    prctl_arg |= (level == M_HEAP_TAGGING_LEVEL_SYNC) ? PR_MTE_TCF_SYNC : PR_MTE_TCF_ASYNC;
-
-    // When entering ASYNC mode, specify that we want to allow upgrading to SYNC by OR'ing in the
-    // SYNC flag. But if the kernel doesn't support specifying multiple TCF modes, fall back to
-    // specifying a single mode.
-    if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg | PR_MTE_TCF_SYNC, 0, 0, 0) == 0 ||
-        prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg, 0, 0, 0) == 0) {
-      __libc_shared_globals()->initial_heap_tagging_level = level;
-      __libc_shared_globals()->initial_memtag_stack = memtag_stack;
-
-      if (memtag_stack) {
-        void* pg_start =
-            reinterpret_cast<void*>(page_start(reinterpret_cast<uintptr_t>(stack_top)));
-        if (mprotect(pg_start, page_size(), PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN)) {
-          async_safe_fatal("error: failed to set PROT_MTE on main thread stack: %m");
-        }
-      }
-      struct sigaction action = {};
-      action.sa_flags = SA_SIGINFO | SA_RESTART;
-      action.sa_sigaction = __enable_mte_signal_handler;
-      sigaction(BIONIC_ENABLE_MTE, &action, nullptr);
-      return;
-    }
-  }
-
-  // MTE was either not enabled, or wasn't supported on this device. Try and use
-  // TBI.
-  if (prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0) == 0) {
-    __libc_shared_globals()->initial_heap_tagging_level = M_HEAP_TAGGING_LEVEL_TBI;
-  }
-  // We did not enable MTE, so we do not need to arm the upgrade timer.
-  __libc_shared_globals()->heap_tagging_upgrade_timer_sec = 0;
-  // We also didn't enable memtag_stack.
-  __libc_shared_globals()->initial_memtag_stack = false;
-}
-#else   // __aarch64__
-void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t, uintptr_t, void*) {}
-#endif  // __aarch64__
-
 void __libc_init_profiling_handlers() {
   // The dynamic variant of this function is more interesting, but this
   // at least ensures that static binaries aren't killed by the kernel's
@@ -436,7 +182,8 @@
   __libc_init_common();
   __libc_init_mte(/*memtag_dynamic_entries=*/nullptr,
                   reinterpret_cast<ElfW(Phdr)*>(getauxval(AT_PHDR)), getauxval(AT_PHNUM),
-                  /*load_bias = */ 0, /*stack_top = */ raw_args);
+                  /*load_bias = */ 0);
+  __libc_init_mte_stack(/*stack_top = */ raw_args);
   __libc_init_scudo();
   __libc_init_profiling_handlers();
   __libc_init_fork_handler();
@@ -508,6 +255,6 @@
 // compiled with -ffreestanding to avoid implicit string.h function calls. (It shouldn't strictly
 // be necessary, though.)
 __LIBC_HIDDEN__ libc_shared_globals* __libc_shared_globals() {
-  static libc_shared_globals globals;
+  BIONIC_USED_BEFORE_LINKER_RELOCATES static libc_shared_globals globals;
   return &globals;
 }
diff --git a/libc/bionic/malloc_common.cpp b/libc/bionic/malloc_common.cpp
index 596a1fc..441d884 100644
--- a/libc/bionic/malloc_common.cpp
+++ b/libc/bionic/malloc_common.cpp
@@ -332,44 +332,6 @@
 #endif
 // =============================================================================
 
-// =============================================================================
-// Platform-internal mallopt variant.
-// =============================================================================
-#if defined(LIBC_STATIC)
-extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
-  if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
-    return LimitEnable(arg, arg_size);
-  }
-  if (opcode == M_INITIALIZE_GWP_ASAN) {
-    if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
-      errno = EINVAL;
-      return false;
-    }
-
-    return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
-  }
-  if (opcode == M_MEMTAG_STACK_IS_ON) {
-    if (arg == nullptr || arg_size != sizeof(bool)) {
-      errno = EINVAL;
-      return false;
-    }
-    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
-    return true;
-  }
-  if (opcode == M_GET_DECAY_TIME_ENABLED) {
-    if (arg == nullptr || arg_size != sizeof(bool)) {
-      errno = EINVAL;
-      return false;
-    }
-    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
-    return true;
-  }
-  errno = ENOTSUP;
-  return false;
-}
-#endif
-// =============================================================================
-
 static constexpr MallocDispatch __libc_malloc_default_dispatch __attribute__((unused)) = {
   Malloc(calloc),
   Malloc(free),
diff --git a/libc/bionic/malloc_common_dynamic.cpp b/libc/bionic/malloc_common_dynamic.cpp
index 6db6251..7b6d7d4 100644
--- a/libc/bionic/malloc_common_dynamic.cpp
+++ b/libc/bionic/malloc_common_dynamic.cpp
@@ -80,7 +80,7 @@
 
 _Atomic bool gGlobalsMutating = false;
 
-static bool gZygoteChild = false;
+bool gZygoteChild = false;
 
 // In a Zygote child process, this is set to true if profiling of this process
 // is allowed. Note that this is set at a later time than gZygoteChild. The
@@ -89,7 +89,7 @@
 // domains if applicable). These two flags are read by the
 // BIONIC_SIGNAL_PROFILER handler, which does nothing if the process is not
 // profileable.
-static _Atomic bool gZygoteChildProfileable = false;
+_Atomic bool gZygoteChildProfileable = false;
 
 // =============================================================================
 
@@ -471,93 +471,6 @@
 }
 // =============================================================================
 
-// =============================================================================
-// Platform-internal mallopt variant.
-// =============================================================================
-__BIONIC_WEAK_FOR_NATIVE_BRIDGE
-extern "C" bool android_mallopt(int opcode, void* arg, size_t arg_size) {
-  if (opcode == M_SET_ZYGOTE_CHILD) {
-    if (arg != nullptr || arg_size != 0) {
-      errno = EINVAL;
-      return false;
-    }
-    gZygoteChild = true;
-    return true;
-  }
-  if (opcode == M_INIT_ZYGOTE_CHILD_PROFILING) {
-    if (arg != nullptr || arg_size != 0) {
-      errno = EINVAL;
-      return false;
-    }
-    atomic_store_explicit(&gZygoteChildProfileable, true, memory_order_release);
-    // Also check if heapprofd should start profiling from app startup.
-    HeapprofdInitZygoteChildProfiling();
-    return true;
-  }
-  if (opcode == M_GET_PROCESS_PROFILEABLE) {
-    if (arg == nullptr || arg_size != sizeof(bool)) {
-      errno = EINVAL;
-      return false;
-    }
-    // Native processes are considered profileable. Zygote children are considered
-    // profileable only when appropriately tagged.
-    *reinterpret_cast<bool*>(arg) =
-        !gZygoteChild || atomic_load_explicit(&gZygoteChildProfileable, memory_order_acquire);
-    return true;
-  }
-  if (opcode == M_SET_ALLOCATION_LIMIT_BYTES) {
-    return LimitEnable(arg, arg_size);
-  }
-  if (opcode == M_WRITE_MALLOC_LEAK_INFO_TO_FILE) {
-    if (arg == nullptr || arg_size != sizeof(FILE*)) {
-      errno = EINVAL;
-      return false;
-    }
-    return WriteMallocLeakInfo(reinterpret_cast<FILE*>(arg));
-  }
-  if (opcode == M_GET_MALLOC_LEAK_INFO) {
-    if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
-      errno = EINVAL;
-      return false;
-    }
-    return GetMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
-  }
-  if (opcode == M_FREE_MALLOC_LEAK_INFO) {
-    if (arg == nullptr || arg_size != sizeof(android_mallopt_leak_info_t)) {
-      errno = EINVAL;
-      return false;
-    }
-    return FreeMallocLeakInfo(reinterpret_cast<android_mallopt_leak_info_t*>(arg));
-  }
-  if (opcode == M_INITIALIZE_GWP_ASAN) {
-    if (arg == nullptr || arg_size != sizeof(android_mallopt_gwp_asan_options_t)) {
-      errno = EINVAL;
-      return false;
-    }
-
-    return EnableGwpAsan(*reinterpret_cast<android_mallopt_gwp_asan_options_t*>(arg));
-  }
-  if (opcode == M_MEMTAG_STACK_IS_ON) {
-    if (arg == nullptr || arg_size != sizeof(bool)) {
-      errno = EINVAL;
-      return false;
-    }
-    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_memtag_stack);
-    return true;
-  }
-  if (opcode == M_GET_DECAY_TIME_ENABLED) {
-    if (arg == nullptr || arg_size != sizeof(bool)) {
-      errno = EINVAL;
-      return false;
-    }
-    *reinterpret_cast<bool*>(arg) = atomic_load(&__libc_globals->decay_time_enabled);
-    return true;
-  }
-  // Try heapprofd's mallopt, as it handles options not covered here.
-  return HeapprofdMallopt(opcode, arg, arg_size);
-}
-// =============================================================================
-
 #if !defined(__LP64__) && defined(__arm__)
 // =============================================================================
 // Old platform only functions that some old 32 bit apps are still using.
diff --git a/libc/bionic/posix_timers.cpp b/libc/bionic/posix_timers.cpp
index 65749a4..9516059 100644
--- a/libc/bionic/posix_timers.cpp
+++ b/libc/bionic/posix_timers.cpp
@@ -141,7 +141,7 @@
   // Otherwise, this must be SIGEV_THREAD timer...
   timer->callback = evp->sigev_notify_function;
   timer->callback_argument = evp->sigev_value;
-  atomic_init(&timer->deleted, false);
+  atomic_store_explicit(&timer->deleted, false, memory_order_relaxed);
 
   // Check arguments that the kernel doesn't care about but we do.
   if (timer->callback == nullptr) {
diff --git a/libc/bionic/pthread_barrier.cpp b/libc/bionic/pthread_barrier.cpp
index 1618222..ff048a6 100644
--- a/libc/bionic/pthread_barrier.cpp
+++ b/libc/bionic/pthread_barrier.cpp
@@ -95,8 +95,8 @@
     return EINVAL;
   }
   barrier->init_count = count;
-  atomic_init(&barrier->state, WAIT);
-  atomic_init(&barrier->wait_count, 0);
+  atomic_store_explicit(&barrier->state, WAIT, memory_order_relaxed);
+  atomic_store_explicit(&barrier->wait_count, 0, memory_order_relaxed);
   barrier->pshared = false;
   if (attr != nullptr && (*attr & 1)) {
     barrier->pshared = true;
diff --git a/libc/bionic/pthread_cond.cpp b/libc/bionic/pthread_cond.cpp
index f444676..197fd19 100644
--- a/libc/bionic/pthread_cond.cpp
+++ b/libc/bionic/pthread_cond.cpp
@@ -140,10 +140,10 @@
   if (attr != nullptr) {
     init_state = (*attr & COND_FLAGS_MASK);
   }
-  atomic_init(&cond->state, init_state);
+  atomic_store_explicit(&cond->state, init_state, memory_order_relaxed);
 
 #if defined(__LP64__)
-  atomic_init(&cond->waiters, 0);
+  atomic_store_explicit(&cond->waiters, 0, memory_order_relaxed);
 #endif
 
   return 0;
diff --git a/libc/bionic/pthread_create.cpp b/libc/bionic/pthread_create.cpp
index ba20c51..3fa8ee6 100644
--- a/libc/bionic/pthread_create.cpp
+++ b/libc/bionic/pthread_create.cpp
@@ -159,11 +159,11 @@
 int __init_thread(pthread_internal_t* thread) {
   thread->cleanup_stack = nullptr;
 
-  if (__predict_true((thread->attr.flags & PTHREAD_ATTR_FLAG_DETACHED) == 0)) {
-    atomic_init(&thread->join_state, THREAD_NOT_JOINED);
-  } else {
-    atomic_init(&thread->join_state, THREAD_DETACHED);
+  ThreadJoinState state = THREAD_NOT_JOINED;
+  if (__predict_false((thread->attr.flags & PTHREAD_ATTR_FLAG_DETACHED) != 0)) {
+    state = THREAD_DETACHED;
   }
+  atomic_store_explicit(&thread->join_state, state, memory_order_relaxed);
 
   // Set the scheduling policy/priority of the thread if necessary.
   bool need_set = true;
@@ -351,15 +351,20 @@
 
 extern "C" int __rt_sigprocmask(int, const sigset64_t*, sigset64_t*, size_t);
 
-__attribute__((no_sanitize("hwaddress")))
+__attribute__((no_sanitize("hwaddress", "memtag")))
 #if defined(__aarch64__)
 // This function doesn't return, but it does appear in stack traces. Avoid using return PAC in this
 // function because we may end up resetting IA, which may confuse unwinders due to mismatching keys.
 __attribute__((target("branch-protection=bti")))
 #endif
-static int __pthread_start(void* arg) {
+static int
+__pthread_start(void* arg) {
   pthread_internal_t* thread = reinterpret_cast<pthread_internal_t*>(arg);
-
+#if defined(__aarch64__)
+  if (thread->should_allocate_stack_mte_ringbuffer) {
+    thread->bionic_tcb->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, thread);
+  }
+#endif
   __hwasan_thread_enter();
 
   // Wait for our creating thread to release us. This lets it have time to
@@ -450,9 +455,9 @@
 // This has to be done under g_thread_creation_lock or g_thread_list_lock to avoid racing with
 // __pthread_internal_remap_stack_with_mte.
 #ifdef __aarch64__
-  if (__libc_memtag_stack_abi) {
-    tcb->tls_slot(TLS_SLOT_STACK_MTE) = __allocate_stack_mte_ringbuffer(0, thread);
-  }
+  thread->should_allocate_stack_mte_ringbuffer = __libc_memtag_stack_abi;
+#else
+  thread->should_allocate_stack_mte_ringbuffer = false;
 #endif
 
   sigset64_t block_all_mask;
diff --git a/libc/bionic/pthread_exit.cpp b/libc/bionic/pthread_exit.cpp
index 0181aba..27d05c2 100644
--- a/libc/bionic/pthread_exit.cpp
+++ b/libc/bionic/pthread_exit.cpp
@@ -33,10 +33,11 @@
 #include <string.h>
 #include <sys/mman.h>
 
-#include "private/bionic_constants.h"
-#include "private/bionic_defs.h"
+#include "platform/bionic/mte.h"
 #include "private/ScopedRWLock.h"
 #include "private/ScopedSignalBlocker.h"
+#include "private/bionic_constants.h"
+#include "private/bionic_defs.h"
 #include "pthread_internal.h"
 
 extern "C" __noreturn void _exit_with_stack_teardown(void*, size_t);
@@ -67,7 +68,7 @@
 }
 
 __BIONIC_WEAK_FOR_NATIVE_BRIDGE
-void pthread_exit(void* return_value) {
+__attribute__((no_sanitize("memtag"))) void pthread_exit(void* return_value) {
   // Call dtors for thread_local objects first.
   __cxa_thread_finalize();
 
@@ -138,6 +139,13 @@
   __notify_thread_exit_callbacks();
   __hwasan_thread_exit();
 
+#if defined(__aarch64__)
+  if (void* stack_mte_tls = thread->bionic_tcb->tls_slot(TLS_SLOT_STACK_MTE)) {
+    stack_mte_free_ringbuffer(reinterpret_cast<uintptr_t>(stack_mte_tls));
+  }
+#endif
+  // Everything below this line needs to be no_sanitize("memtag").
+
   if (old_state == THREAD_DETACHED && thread->mmap_size != 0) {
     // We need to free mapped space for detached threads when they exit.
     // That's not something we can do in C.
diff --git a/libc/bionic/pthread_getaffinity.cpp b/libc/bionic/pthread_getaffinity.cpp
new file mode 100644
index 0000000..9ce436c
--- /dev/null
+++ b/libc/bionic/pthread_getaffinity.cpp
@@ -0,0 +1,42 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <errno.h>
+
+#include "private/ErrnoRestorer.h"
+#include "pthread_internal.h"
+
+int pthread_getaffinity_np(pthread_t t, size_t cpu_set_size, cpu_set_t* cpu_set) {
+  ErrnoRestorer errno_restorer;
+
+  pid_t tid = __pthread_internal_gettid(t, "pthread_getaffinity_np");
+  if (tid == -1) return ESRCH;
+
+  if (sched_getaffinity(tid, cpu_set_size, cpu_set) == -1) return errno;
+  return 0;
+}
diff --git a/libc/bionic/pthread_internal.cpp b/libc/bionic/pthread_internal.cpp
index 14cc7da..4f2ad0c 100644
--- a/libc/bionic/pthread_internal.cpp
+++ b/libc/bionic/pthread_internal.cpp
@@ -34,6 +34,7 @@
 #include <string.h>
 #include <sys/mman.h>
 #include <sys/prctl.h>
+#include <sys/types.h>
 
 #include <async_safe/log.h>
 #include <bionic/mte.h>
@@ -76,15 +77,6 @@
 }
 
 static void __pthread_internal_free(pthread_internal_t* thread) {
-#ifdef __aarch64__
-  if (void* stack_mte_tls = thread->bionic_tcb->tls_slot(TLS_SLOT_STACK_MTE)) {
-    size_t size =
-        stack_mte_ringbuffer_size_from_pointer(reinterpret_cast<uintptr_t>(stack_mte_tls));
-    void* ptr = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(stack_mte_tls) &
-                                        ((1ULL << 56ULL) - 1ULL));
-    munmap(ptr, size);
-  }
-#endif
   if (thread->mmap_size != 0) {
     // Free mapped space, including thread stack and pthread_internal_t.
     munmap(thread->mmap_base, thread->mmap_size);
@@ -216,7 +208,10 @@
   __libc_memtag_stack_abi = true;
 
   for (pthread_internal_t* t = g_thread_list; t != nullptr; t = t->next) {
-    if (t->terminating) continue;
+    // should_allocate_stack_mte_ringbuffer indicates the thread is already
+    // aware that this process requires stack MTE, and will allocate the
+    // ring buffer in __pthread_start.
+    if (t->terminating || t->should_allocate_stack_mte_ringbuffer) continue;
     t->bionic_tcb->tls_slot(TLS_SLOT_STACK_MTE) =
         __allocate_stack_mte_ringbuffer(0, t->is_main() ? nullptr : t);
   }
@@ -264,8 +259,7 @@
   g_func = func;
   g_arg = arg;
 
-  static _Atomic(bool) g_retval;
-  atomic_init(&g_retval, true);
+  static _Atomic(bool) g_retval(true);
 
   auto handler = [](int, siginfo_t*, void*) {
     ErrnoRestorer restorer;
diff --git a/libc/bionic/pthread_internal.h b/libc/bionic/pthread_internal.h
index 5db42ab..cbaa9a6 100644
--- a/libc/bionic/pthread_internal.h
+++ b/libc/bionic/pthread_internal.h
@@ -181,6 +181,7 @@
 
   bionic_tcb* bionic_tcb;
   char stack_mte_ringbuffer_vma_name_buffer[32];
+  bool should_allocate_stack_mte_ringbuffer;
 
   bool is_main() { return start_routine == nullptr; }
 };
diff --git a/libc/bionic/pthread_mutex.cpp b/libc/bionic/pthread_mutex.cpp
index 0a452e9..c99717a 100644
--- a/libc/bionic/pthread_mutex.cpp
+++ b/libc/bionic/pthread_mutex.cpp
@@ -509,8 +509,8 @@
     memset(mutex, 0, sizeof(pthread_mutex_internal_t));
 
     if (__predict_true(attr == nullptr)) {
-        atomic_init(&mutex->state, MUTEX_TYPE_BITS_NORMAL);
-        return 0;
+      atomic_store_explicit(&mutex->state, MUTEX_TYPE_BITS_NORMAL, memory_order_relaxed);
+      return 0;
     }
 
     uint16_t state = 0;
@@ -543,13 +543,13 @@
         }
         mutex->pi_mutex_id = id;
 #endif
-        atomic_init(&mutex->state, PI_MUTEX_STATE);
+        atomic_store_explicit(&mutex->state, PI_MUTEX_STATE, memory_order_relaxed);
         PIMutex& pi_mutex = mutex->ToPIMutex();
         pi_mutex.type = *attr & MUTEXATTR_TYPE_MASK;
         pi_mutex.shared = (*attr & MUTEXATTR_SHARED_MASK) != 0;
     } else {
-        atomic_init(&mutex->state, state);
-        atomic_init(&mutex->owner_tid, 0);
+      atomic_store_explicit(&mutex->state, state, memory_order_relaxed);
+      atomic_store_explicit(&mutex->owner_tid, 0, memory_order_relaxed);
     }
     return 0;
 }
diff --git a/libc/bionic/pthread_rwlock.cpp b/libc/bionic/pthread_rwlock.cpp
index 6f3c6fe..92134b4 100644
--- a/libc/bionic/pthread_rwlock.cpp
+++ b/libc/bionic/pthread_rwlock.cpp
@@ -247,7 +247,7 @@
     }
   }
 
-  atomic_init(&rwlock->state, 0);
+  atomic_store_explicit(&rwlock->state, 0, memory_order_relaxed);
   rwlock->pending_lock.init(rwlock->pshared);
   return 0;
 }
diff --git a/libc/bionic/pthread_setaffinity.cpp b/libc/bionic/pthread_setaffinity.cpp
new file mode 100644
index 0000000..6db418e
--- /dev/null
+++ b/libc/bionic/pthread_setaffinity.cpp
@@ -0,0 +1,42 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <errno.h>
+
+#include "private/ErrnoRestorer.h"
+#include "pthread_internal.h"
+
+int pthread_setaffinity_np(pthread_t t, size_t cpu_set_size, const cpu_set_t* cpu_set) {
+  ErrnoRestorer errno_restorer;
+
+  pid_t tid = __pthread_internal_gettid(t, "pthread_setaffinity_np");
+  if (tid == -1) return ESRCH;
+
+  if (sched_setaffinity(tid, cpu_set_size, cpu_set) == -1) return errno;
+  return 0;
+}
diff --git a/libc/bionic/semaphore.cpp b/libc/bionic/semaphore.cpp
index 33552a9..2c9b745 100644
--- a/libc/bionic/semaphore.cpp
+++ b/libc/bionic/semaphore.cpp
@@ -113,7 +113,7 @@
   }
 
   atomic_uint* sem_count_ptr = SEM_TO_ATOMIC_POINTER(sem);
-  atomic_init(sem_count_ptr, count);
+  atomic_store_explicit(sem_count_ptr, count, memory_order_relaxed);
   return 0;
 }
 
diff --git a/libc/bionic/stdlib_l.cpp b/libc/bionic/stdlib_l.cpp
index a636d08..58a9079 100644
--- a/libc/bionic/stdlib_l.cpp
+++ b/libc/bionic/stdlib_l.cpp
@@ -26,17 +26,11 @@
  * SUCH DAMAGE.
  */
 
+#define __BIONIC_STDLIB_INLINE /* Out of line. */
 #include <stdlib.h>
-#include <xlocale.h>
+#include <bits/stdlib_inlines.h>
 
-double strtod_l(const char* s, char** end_ptr, locale_t) {
-  return strtod(s, end_ptr);
-}
-
-float strtof_l(const char* s, char** end_ptr, locale_t) {
-  return strtof(s, end_ptr);
-}
-
+// strtold_l was introduced in API level 21, so it isn't polyfilled any more.
 long double strtold_l(const char* s, char** end_ptr, locale_t) {
   return strtold(s, end_ptr);
 }
diff --git a/libc/dns/net/gethnamaddr.c b/libc/dns/net/gethnamaddr.c
index add124f..1ffabfa 100644
--- a/libc/dns/net/gethnamaddr.c
+++ b/libc/dns/net/gethnamaddr.c
@@ -495,7 +495,7 @@
 	*he = NO_RECOVERY;
 	return NULL;
 success:
-	bp = (char *)ALIGN(bp);
+	bp = __builtin_align_up(bp, sizeof(uintptr_t));
 	n = (int)(ap - aliases);
 	qlen = (n + 1) * sizeof(*hent->h_aliases);
 	if ((size_t)(ep - bp) < qlen)
@@ -616,7 +616,7 @@
 	}
 
 	// Fix alignment after variable-length data.
-	ptr = (char*)ALIGN(ptr);
+	ptr = __builtin_align_up(ptr, sizeof(uintptr_t));
 
 	int aliases_len = ((int)(aliases - aliases_ptrs) + 1) * sizeof(*hp->h_aliases);
 	if (ptr + aliases_len > hbuf_end) {
@@ -653,7 +653,7 @@
 	}
 
 	// Fix alignment after variable-length data.
-	ptr = (char*)ALIGN(ptr);
+	ptr = __builtin_align_up(ptr, sizeof(uintptr_t));
 
 	int addrs_len = ((int)(addr_p - addr_ptrs) + 1) * sizeof(*hp->h_addr_list);
 	if (ptr + addrs_len > hbuf_end) {
diff --git a/libc/dns/net/sethostent.c b/libc/dns/net/sethostent.c
index 5c4bdb5..8ea4315 100644
--- a/libc/dns/net/sethostent.c
+++ b/libc/dns/net/sethostent.c
@@ -198,7 +198,7 @@
 				HENT_SCOPY(aliases[anum], hp->h_aliases[anum],
 				    ptr, len);
 			}
-			ptr = (void *)ALIGN(ptr);
+			ptr = __builtin_align_up(ptr, sizeof(uintptr_t));
 			if ((size_t)(ptr - buf) >= info->buflen)
 				goto nospc;
 		}
diff --git a/libc/execinfo/include/execinfo.h b/libc/execinfo/include/execinfo.h
index e092c00..c8f9e21 100644
--- a/libc/execinfo/include/execinfo.h
+++ b/libc/execinfo/include/execinfo.h
@@ -30,8 +30,10 @@
 
 /*
  * This file is exported as part of libexecinfo for use with musl, which doesn't
- * define __INTRODUCED_IN.  Stub it out.
+ * define __INTRODUCED_IN or __BIONIC_AVAILABILITY_GUARD.  Stub them out.
  */
 #define __INTRODUCED_IN(x)
+#define __BIONIC_AVAILABILITY_GUARD(x) 1
 #include <bionic/execinfo.h>
+#undef __BIONIC_AVAILABILITY_GUARD
 #undef __INTRODUCED_IN
diff --git a/libc/include/android/api-level.h b/libc/include/android/api-level.h
index 1bde3a5..c9536c1 100644
--- a/libc/include/android/api-level.h
+++ b/libc/include/android/api-level.h
@@ -189,7 +189,11 @@
  *
  * Available since API level 24.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int android_get_application_target_sdk_version() __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 #if __ANDROID_API__ < 29
 
diff --git a/libc/include/android/crash_detail.h b/libc/include/android/crash_detail.h
index 946a3ab..fd1312a 100644
--- a/libc/include/android/crash_detail.h
+++ b/libc/include/android/crash_detail.h
@@ -33,9 +33,10 @@
  * @brief Attach extra information to android crashes.
  */
 
-#include <stddef.h>
 #include <sys/cdefs.h>
 
+#include <stddef.h>
+
 __BEGIN_DECLS
 
 typedef struct crash_detail_t crash_detail_t;
@@ -79,6 +80,8 @@
  *
  * \return a handle to the extra crash detail.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 crash_detail_t* _Nullable android_crash_detail_register(
     const void* _Nonnull name, size_t name_size, const void* _Nullable data, size_t data_size) __INTRODUCED_IN(35);
 
@@ -122,5 +125,7 @@
  * \param name_size number of bytes of the buffer pointed to by name
  */
 void android_crash_detail_replace_name(crash_detail_t* _Nonnull crash_detail, const void* _Nonnull name, size_t name_size) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 __END_DECLS
diff --git a/libc/include/android/dlext.h b/libc/include/android/dlext.h
index 842ceea..d8d2752 100644
--- a/libc/include/android/dlext.h
+++ b/libc/include/android/dlext.h
@@ -16,10 +16,11 @@
 
 #pragma once
 
+#include <sys/cdefs.h>
+
 #include <stdbool.h>
 #include <stddef.h>
 #include <stdint.h>
-#include <sys/cdefs.h>
 #include <sys/types.h>  /* for off64_t */
 
 /**
diff --git a/libc/include/android/fdsan.h b/libc/include/android/fdsan.h
index 4540498..a04fc7e 100644
--- a/libc/include/android/fdsan.h
+++ b/libc/include/android/fdsan.h
@@ -28,9 +28,10 @@
 
 #pragma once
 
+#include <sys/cdefs.h>
+
 #include <stdbool.h>
 #include <stdint.h>
-#include <sys/cdefs.h>
 
 __BEGIN_DECLS
 
@@ -134,6 +135,8 @@
 /*
  * Create an owner tag with the specified type and least significant 56 bits of tag.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 uint64_t android_fdsan_create_owner_tag(enum android_fdsan_owner_type type, uint64_t tag) __INTRODUCED_IN(29) __attribute__((__weak__));
 
 /*
@@ -168,6 +171,8 @@
  * Get an owner tag's value, with the type masked off.
  */
 uint64_t android_fdsan_get_tag_value(uint64_t tag) __INTRODUCED_IN(29);
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 
 enum android_fdsan_error_level {
   // No errors.
@@ -186,6 +191,8 @@
 /*
  * Get the error level.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 enum android_fdsan_error_level android_fdsan_get_error_level() __INTRODUCED_IN(29) __attribute__((__weak__));
 
 /*
@@ -203,9 +210,15 @@
  * (e.g. postfork).
  */
 enum android_fdsan_error_level android_fdsan_set_error_level(enum android_fdsan_error_level new_level) __INTRODUCED_IN(29) __attribute__((__weak__));
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 
 /*
  * Set the error level to the global setting if available, or a default value.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 enum android_fdsan_error_level android_fdsan_set_error_level_from_property(enum android_fdsan_error_level default_level) __INTRODUCED_IN(30) __attribute__((__weak__));
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 __END_DECLS
diff --git a/libc/include/android/legacy_stdlib_inlines.h b/libc/include/android/legacy_stdlib_inlines.h
index a5a07ef..d228e67 100644
--- a/libc/include/android/legacy_stdlib_inlines.h
+++ b/libc/include/android/legacy_stdlib_inlines.h
@@ -30,22 +30,9 @@
 
 #include <sys/cdefs.h>
 
-
 #if __ANDROID_API__ < 26
 
-#include <stdlib.h>
-#include <xlocale.h>
-
-__BEGIN_DECLS
-
-static __inline double strtod_l(const char* _Nonnull __s, char* _Nullable * _Nullable __end_ptr, locale_t _Nonnull __l) {
-  return strtod(__s, __end_ptr);
-}
-
-static __inline float strtof_l(const char* _Nonnull __s, char* _Nullable * _Nullable __end_ptr, locale_t _Nonnull __l) {
-  return strtof(__s, __end_ptr);
-}
-
-__END_DECLS
+#define __BIONIC_THREADS_INLINE static __inline
+#include <bits/stdlib_inlines.h>
 
 #endif
diff --git a/libc/include/android/set_abort_message.h b/libc/include/android/set_abort_message.h
index a778057..6ad5678 100644
--- a/libc/include/android/set_abort_message.h
+++ b/libc/include/android/set_abort_message.h
@@ -33,10 +33,11 @@
  * @brief The android_set_abort_message() function.
  */
 
+#include <sys/cdefs.h>
+
 #include <stddef.h>
 #include <stdint.h>
 #include <string.h>
-#include <sys/cdefs.h>
 
 __BEGIN_DECLS
 
diff --git a/libc/include/android/versioning.h b/libc/include/android/versioning.h
index fe9264d..1cf6e51 100644
--- a/libc/include/android/versioning.h
+++ b/libc/include/android/versioning.h
@@ -32,6 +32,11 @@
 // load even on systems too old to contain the API, but calls must be guarded
 // with `__builtin_available(android api_level, *)` to avoid segfaults.
 #define __BIONIC_AVAILABILITY(__what, ...) __attribute__((__availability__(android,__what __VA_OPT__(,) __VA_ARGS__)))
+
+// When the caller is using weak API references, we should expose the decls for
+// APIs which are not available in the caller's minSdkVersion, otherwise there's
+// no way to take advantage of the weak references.
+#define __BIONIC_AVAILABILITY_GUARD(api_level) 1
 #else
 // The 'strict' flag is required for NDK clients where the code was not written
 // to handle the case where the API was available at build-time but not at
@@ -40,9 +45,25 @@
 // compile in this mode (or worse, if the build doesn't use
 // -Werror=unguarded-availability, it would build but crash at runtime).
 #define __BIONIC_AVAILABILITY(__what, ...) __attribute__((__availability__(android,strict,__what __VA_OPT__(,) __VA_ARGS__)))
+
+// When the caller is using strict API references, we hide APIs which are not
+// available in the caller's minSdkVersion. This is a bionic-only deviation in
+// behavior from the rest of the NDK headers, but it's necessary to maintain
+// source compatibility with 3p libraries that either can't correctly detect API
+// availability (either incorrectly detecting as always-available or as
+// never-available, but neither is true), or define their own polyfills which
+// conflict with our declarations.
+//
+// https://github.com/android/ndk/issues/2081
+#define __BIONIC_AVAILABILITY_GUARD(api_level) (__ANDROID_MIN_SDK_VERSION__ >= (api_level))
 #endif
 
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wc23-extensions"
+// Passing no argument for the '...' parameter of a variadic macro is a C23 extension
 #define __INTRODUCED_IN(api_level) __BIONIC_AVAILABILITY(introduced=api_level)
+#pragma clang diagnostic pop
+
 #define __DEPRECATED_IN(api_level, msg) __BIONIC_AVAILABILITY(deprecated=api_level, message=msg)
 #define __REMOVED_IN(api_level, msg) __BIONIC_AVAILABILITY(obsoleted=api_level, message=msg)
 
@@ -59,9 +80,3 @@
 #define __INTRODUCED_IN_32(api_level)
 #define __INTRODUCED_IN_64(api_level) __BIONIC_AVAILABILITY(introduced=api_level)
 #endif
-
-// Vendor modules do not follow SDK versioning. Ignore NDK guards for vendor modules.
-#if defined(__ANDROID_VENDOR__)
-#undef __BIONIC_AVAILABILITY
-#define __BIONIC_AVAILABILITY(api_level, ...)
-#endif // defined(__ANDROID_VENDOR__)
diff --git a/libc/include/arpa/ftp.h b/libc/include/arpa/ftp.h
index 081c037..fecbf7f 100644
--- a/libc/include/arpa/ftp.h
+++ b/libc/include/arpa/ftp.h
@@ -34,6 +34,8 @@
 #ifndef _ARPA_FTP_H_
 #define	_ARPA_FTP_H_
 
+#include <sys/cdefs.h>
+
 /* Definitions for FTP; see RFC-765. */
 
 /*
diff --git a/libc/include/arpa/inet.h b/libc/include/arpa/inet.h
index f00f2c1..ce9dd93 100644
--- a/libc/include/arpa/inet.h
+++ b/libc/include/arpa/inet.h
@@ -29,9 +29,10 @@
 #ifndef _ARPA_INET_H_
 #define _ARPA_INET_H_
 
+#include <sys/cdefs.h>
+
 #include <netinet/in.h>
 #include <stdint.h>
-#include <sys/cdefs.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/arpa/nameser.h b/libc/include/arpa/nameser.h
index 97109ee..3e0025e 100644
--- a/libc/include/arpa/nameser.h
+++ b/libc/include/arpa/nameser.h
@@ -55,9 +55,10 @@
 
 #define BIND_4_COMPAT
 
-#include <sys/types.h>
 #include <sys/cdefs.h>
 
+#include <sys/types.h>
+
 /*
  * Revision information.  This is the release date in YYYYMMDD format.
  * It can change every day so the right thing to do with it is use it
@@ -547,6 +548,8 @@
 #define ns_sprintrrf __ns_sprintrrf
 #endif
 
+
+#if __BIONIC_AVAILABILITY_GUARD(22)
 int ns_msg_getflag(ns_msg __handle, int __flag) __INTRODUCED_IN(22);
 uint16_t ns_get16(const u_char* _Nonnull __src) __INTRODUCED_IN(22);
 uint32_t ns_get32(const u_char* _Nonnull __src) __INTRODUCED_IN(22);
@@ -570,6 +573,8 @@
 
 int ns_makecanon(const char* _Nonnull __src, char* _Nonnull __dst, size_t __dst_size) __INTRODUCED_IN(22);
 int ns_samename(const char* _Nonnull __lhs, const char* _Nonnull __rhs) __INTRODUCED_IN(22);
+#endif /* __BIONIC_AVAILABILITY_GUARD(22) */
+
 
 __END_DECLS
 
diff --git a/libc/include/arpa/nameser_compat.h b/libc/include/arpa/nameser_compat.h
index e4e9335..027e5ca 100644
--- a/libc/include/arpa/nameser_compat.h
+++ b/libc/include/arpa/nameser_compat.h
@@ -40,9 +40,10 @@
 #ifndef _ARPA_NAMESER_COMPAT_
 #define	_ARPA_NAMESER_COMPAT_
 
-#include <endian.h>
 #include <sys/cdefs.h>
 
+#include <endian.h>
+
 #define	__BIND		19950621	/* (DEAD) interface version stamp. */
 
 /*
diff --git a/libc/include/arpa/telnet.h b/libc/include/arpa/telnet.h
index 758e9b8..30d8f21 100644
--- a/libc/include/arpa/telnet.h
+++ b/libc/include/arpa/telnet.h
@@ -33,6 +33,8 @@
 #ifndef _ARPA_TELNET_H_
 #define	_ARPA_TELNET_H_
 
+#include <sys/cdefs.h>
+
 /*
  * Definitions for the TELNET protocol.
  */
diff --git a/libc/include/bits/bionic_multibyte_result.h b/libc/include/bits/bionic_multibyte_result.h
index 0d5cf21..930e67c 100644
--- a/libc/include/bits/bionic_multibyte_result.h
+++ b/libc/include/bits/bionic_multibyte_result.h
@@ -34,9 +34,10 @@
  * conversion APIs defined by C.
  */
 
-#include <stddef.h>
 #include <sys/cdefs.h>
 
+#include <stddef.h>
+
 __BEGIN_DECLS
 
 /**
diff --git a/libc/include/bits/fortify/poll.h b/libc/include/bits/fortify/poll.h
index f2e27d7..1b4a5bf 100644
--- a/libc/include/bits/fortify/poll.h
+++ b/libc/include/bits/fortify/poll.h
@@ -30,9 +30,17 @@
 #error "Never include this file directly; instead, include <poll.h>"
 #endif
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int __poll_chk(struct pollfd* _Nullable, nfds_t, int, size_t) __INTRODUCED_IN(23);
 int __ppoll_chk(struct pollfd* _Nullable, nfds_t, const struct timespec* _Nullable, const sigset_t* _Nullable, size_t) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int __ppoll64_chk(struct pollfd* _Nullable, nfds_t, const struct timespec* _Nullable, const sigset64_t* _Nullable, size_t) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 #if defined(__BIONIC_FORTIFY)
 #define __bos_fd_count_trivially_safe(bos_val, fds, fd_count)              \
diff --git a/libc/include/bits/fortify/socket.h b/libc/include/bits/fortify/socket.h
index 1c3605b..bd626f9 100644
--- a/libc/include/bits/fortify/socket.h
+++ b/libc/include/bits/fortify/socket.h
@@ -30,7 +30,11 @@
 #error "Never include this file directly; instead, include <sys/socket.h>"
 #endif
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 ssize_t __sendto_chk(int, const void* _Nonnull, size_t, size_t, int, const struct sockaddr* _Nullable, socklen_t) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 ssize_t __recvfrom_chk(int, void* _Nullable, size_t, size_t, int, struct sockaddr* _Nullable, socklen_t* _Nullable);
 
 #if defined(__BIONIC_FORTIFY)
diff --git a/libc/include/bits/fortify/stdio.h b/libc/include/bits/fortify/stdio.h
index e4607e0..f9faeba 100644
--- a/libc/include/bits/fortify/stdio.h
+++ b/libc/include/bits/fortify/stdio.h
@@ -31,8 +31,12 @@
 #endif
 
 char* _Nullable __fgets_chk(char* _Nonnull, int, FILE* _Nonnull, size_t);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 size_t __fread_chk(void* _Nonnull, size_t, size_t, FILE* _Nonnull, size_t) __INTRODUCED_IN(24);
 size_t __fwrite_chk(const void* _Nonnull, size_t, size_t, FILE* _Nonnull, size_t) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 #if defined(__BIONIC_FORTIFY) && !defined(__BIONIC_NO_STDIO_FORTIFY)
 
diff --git a/libc/include/bits/fortify/string.h b/libc/include/bits/fortify/string.h
index 4d32b04..6f0ee4a 100644
--- a/libc/include/bits/fortify/string.h
+++ b/libc/include/bits/fortify/string.h
@@ -30,8 +30,12 @@
 #error "Never include this file directly; instead, include <string.h>"
 #endif
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void* _Nullable __memchr_chk(const void* _Nonnull, int, size_t, size_t) __INTRODUCED_IN(23);
 void* _Nullable __memrchr_chk(const void* _Nonnull, int, size_t, size_t) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 char* _Nonnull __stpncpy_chk2(char* _Nonnull, const char* _Nonnull, size_t, size_t, size_t);
 char* _Nonnull __strncpy_chk2(char* _Nonnull, const char* _Nonnull, size_t, size_t, size_t);
 size_t __strlcpy_chk(char* _Nonnull, const char* _Nonnull, size_t, size_t);
@@ -220,8 +224,13 @@
 }
 
 #if __BIONIC_FORTIFY_RUNTIME_CHECKS_ENABLED
-__BIONIC_FORTIFY_INLINE
-size_t strlen(const char* _Nonnull const s __pass_object_size0) __overloadable {
+/*
+ * Clang, when parsing C, can fold strlen to a constant without LLVM's help.
+ * This doesn't apply to overloads of strlen, so write this differently. We
+ * can't use `__pass_object_size0` here, but that's fine: it doesn't help much
+ * on __always_inline functions.
+ */
+extern __always_inline __inline__ __attribute__((gnu_inline)) size_t strlen(const char* _Nonnull s) {
     return __strlen_chk(s, __bos0(s));
 }
 #endif
diff --git a/libc/include/bits/fortify/unistd.h b/libc/include/bits/fortify/unistd.h
index 7eda1a6..9acb942 100644
--- a/libc/include/bits/fortify/unistd.h
+++ b/libc/include/bits/fortify/unistd.h
@@ -29,24 +29,52 @@
 #error "Never include this file directly; instead, include <unistd.h>"
 #endif
 
-char* _Nullable __getcwd_chk(char* _Nullable, size_t, size_t) __INTRODUCED_IN(24);
 
+#if __BIONIC_AVAILABILITY_GUARD(24)
+char* _Nullable __getcwd_chk(char* _Nullable, size_t, size_t) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
+
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 ssize_t __pread_chk(int, void* _Nonnull, size_t, off_t, size_t) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 ssize_t __pread_real(int, void* _Nonnull, size_t, off_t) __RENAME(pread);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 ssize_t __pread64_chk(int, void* _Nonnull, size_t, off64_t, size_t) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 ssize_t __pread64_real(int, void* _Nonnull, size_t, off64_t) __RENAME(pread64);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 ssize_t __pwrite_chk(int, const void* _Nonnull, size_t, off_t, size_t) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 ssize_t __pwrite_real(int, const void* _Nonnull, size_t, off_t) __RENAME(pwrite);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 ssize_t __pwrite64_chk(int, const void* _Nonnull, size_t, off64_t, size_t) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 ssize_t __pwrite64_real(int, const void* _Nonnull, size_t, off64_t) __RENAME(pwrite64);
 
 ssize_t __read_chk(int, void* __BIONIC_COMPLICATED_NULLNESS, size_t, size_t);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 ssize_t __write_chk(int, const void* __BIONIC_COMPLICATED_NULLNESS, size_t, size_t) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 ssize_t __readlink_chk(const char* _Nonnull, char* _Nonnull, size_t, size_t) __INTRODUCED_IN(23);
 ssize_t __readlinkat_chk(int dirfd, const char* _Nonnull, char* _Nonnull, size_t, size_t) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 #if defined(__BIONIC_FORTIFY)
 
diff --git a/libc/include/bits/getentropy.h b/libc/include/bits/getentropy.h
index 98d8879..c878470 100644
--- a/libc/include/bits/getentropy.h
+++ b/libc/include/bits/getentropy.h
@@ -48,6 +48,10 @@
  *
  * See also arc4random_buf() which is available in all API levels.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 __nodiscard int getentropy(void* _Nonnull __buffer, size_t __buffer_size) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/bits/glibc-syscalls.h b/libc/include/bits/glibc-syscalls.h
index 7b171e8..8c5a91d 100644
--- a/libc/include/bits/glibc-syscalls.h
+++ b/libc/include/bits/glibc-syscalls.h
@@ -36,9 +36,6 @@
 #if defined(__NR_arch_prctl)
   #define SYS_arch_prctl __NR_arch_prctl
 #endif
-#if defined(__NR_arch_specific_syscall)
-  #define SYS_arch_specific_syscall __NR_arch_specific_syscall
-#endif
 #if defined(__NR_arm_fadvise64_64)
   #define SYS_arm_fadvise64_64 __NR_arm_fadvise64_64
 #endif
@@ -1272,9 +1269,6 @@
 #if defined(__NR_syscall)
   #define SYS_syscall __NR_syscall
 #endif
-#if defined(__NR_syscalls)
-  #define SYS_syscalls __NR_syscalls
-#endif
 #if defined(__NR_sysfs)
   #define SYS_sysfs __NR_sysfs
 #endif
@@ -1371,6 +1365,9 @@
 #if defined(__NR_unshare)
   #define SYS_unshare __NR_unshare
 #endif
+#if defined(__NR_uretprobe)
+  #define SYS_uretprobe __NR_uretprobe
+#endif
 #if defined(__NR_uselib)
   #define SYS_uselib __NR_uselib
 #endif
diff --git a/libc/include/bits/lockf.h b/libc/include/bits/lockf.h
index 195b34a..8f922b9 100644
--- a/libc/include/bits/lockf.h
+++ b/libc/include/bits/lockf.h
@@ -56,6 +56,8 @@
  *
  * See also flock().
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int lockf(int __fd, int __op, off_t __length) __RENAME_IF_FILE_OFFSET64(lockf64) __INTRODUCED_IN(24);
 
 /**
@@ -63,5 +65,7 @@
  * even from a 32-bit process without `_FILE_OFFSET_BITS=64`.
  */
 int lockf64(int __fd, int __op, off64_t __length) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 __END_DECLS
diff --git a/libc/include/bits/seek_constants.h b/libc/include/bits/seek_constants.h
index bfc02a8..a4fffb2 100644
--- a/libc/include/bits/seek_constants.h
+++ b/libc/include/bits/seek_constants.h
@@ -33,6 +33,8 @@
  * @brief The `SEEK_` constants.
  */
 
+#include <sys/cdefs.h>
+
 /** Seek to an absolute offset. */
 #define SEEK_SET 0
 /** Seek relative to the current offset. */
diff --git a/libc/include/bits/stdatomic.h b/libc/include/bits/stdatomic.h
index c74eafd..ebdc9e5 100644
--- a/libc/include/bits/stdatomic.h
+++ b/libc/include/bits/stdatomic.h
@@ -134,6 +134,8 @@
 	memory_order_seq_cst = __ATOMIC_SEQ_CST
 } memory_order;
 
+#define kill_dependency(y) (y)
+
 /*
  * 7.17.4 Fences.
  */
diff --git a/libc/include/bits/stdlib_inlines.h b/libc/include/bits/stdlib_inlines.h
new file mode 100644
index 0000000..fffca19
--- /dev/null
+++ b/libc/include/bits/stdlib_inlines.h
@@ -0,0 +1,48 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <xlocale.h>
+#include <sys/cdefs.h>
+
+#if !defined(__BIONIC_STDLIB_INLINE)
+#define __BIONIC_STDLIB_INLINE static __inline
+#endif
+
+__BEGIN_DECLS
+
+__BIONIC_STDLIB_INLINE double strtod_l(const char* _Nonnull __s, char* _Nullable * _Nullable __end_ptr, locale_t _Nonnull __l) {
+  return strtod(__s, __end_ptr);
+}
+
+__BIONIC_STDLIB_INLINE float strtof_l(const char* _Nonnull __s, char* _Nullable * _Nullable __end_ptr, locale_t _Nonnull __l) {
+  return strtof(__s, __end_ptr);
+}
+
+__END_DECLS
diff --git a/libc/include/bits/strcasecmp.h b/libc/include/bits/strcasecmp.h
index be910ad..d76cec9 100644
--- a/libc/include/bits/strcasecmp.h
+++ b/libc/include/bits/strcasecmp.h
@@ -51,7 +51,11 @@
 /**
  * Like strcasecmp() but taking a `locale_t`.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int strcasecmp_l(const char* _Nonnull __s1, const char* _Nonnull __s2, locale_t _Nonnull __l) __attribute_pure__ __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [strncasecmp(3)](https://man7.org/linux/man-pages/man3/strncasecmp.3.html) compares the first
@@ -66,6 +70,10 @@
 /**
  * Like strncasecmp() but taking a `locale_t`.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int strncasecmp_l(const char* _Nonnull __s1, const char* _Nonnull __s2, size_t __n, locale_t _Nonnull __l) __attribute_pure__ __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/bits/swab.h b/libc/include/bits/swab.h
index 9591c2e..da2865a 100644
--- a/libc/include/bits/swab.h
+++ b/libc/include/bits/swab.h
@@ -28,8 +28,9 @@
 
 #pragma once
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+
+#include <stdint.h>
 #include <sys/types.h>
 
 #if !defined(__BIONIC_SWAB_INLINE)
diff --git a/libc/include/bits/termios_inlines.h b/libc/include/bits/termios_inlines.h
index a884b59..bb04e4d 100644
--- a/libc/include/bits/termios_inlines.h
+++ b/libc/include/bits/termios_inlines.h
@@ -29,8 +29,9 @@
 #ifndef _BITS_TERMIOS_INLINES_H_
 #define _BITS_TERMIOS_INLINES_H_
 
-#include <errno.h>
 #include <sys/cdefs.h>
+
+#include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/types.h>
 
diff --git a/libc/include/bits/termios_winsize_inlines.h b/libc/include/bits/termios_winsize_inlines.h
index ae246e4..86777b0 100644
--- a/libc/include/bits/termios_winsize_inlines.h
+++ b/libc/include/bits/termios_winsize_inlines.h
@@ -28,8 +28,9 @@
 
 #pragma once
 
-#include <errno.h>
 #include <sys/cdefs.h>
+
+#include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/types.h>
 
diff --git a/libc/include/bits/threads_inlines.h b/libc/include/bits/threads_inlines.h
index 05b785a..ab294c1 100644
--- a/libc/include/bits/threads_inlines.h
+++ b/libc/include/bits/threads_inlines.h
@@ -28,6 +28,8 @@
 
 #pragma once
 
+#include <sys/cdefs.h>
+
 #include <threads.h>
 
 #include <errno.h>
diff --git a/libc/include/bits/wctype.h b/libc/include/bits/wctype.h
index 13a4254..d0cffec 100644
--- a/libc/include/bits/wctype.h
+++ b/libc/include/bits/wctype.h
@@ -58,8 +58,12 @@
 int iswctype(wint_t __wc, wctype_t __type);
 
 typedef const void* wctrans_t;
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 wint_t towctrans(wint_t __wc, wctrans_t _Nonnull __transform) __INTRODUCED_IN(26);
 wctrans_t _Nullable wctrans(const char* _Nonnull __name) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
 
diff --git a/libc/include/complex.h b/libc/include/complex.h
index f205abd..1115862 100644
--- a/libc/include/complex.h
+++ b/libc/include/complex.h
@@ -53,76 +53,190 @@
 
 /* 7.3.5 Trigonometric functions */
 /* 7.3.5.1 The cacos functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex cacos(double complex __z) __INTRODUCED_IN(23);
 float complex cacosf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex cacosl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.5.2 The casin functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex casin(double complex __z) __INTRODUCED_IN(23);
 float complex casinf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex casinl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.5.1 The catan functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex catan(double complex __z) __INTRODUCED_IN(23);
 float complex catanf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex catanl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.5.1 The ccos functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex ccos(double complex __z) __INTRODUCED_IN(23);
 float complex ccosf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex ccosl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.5.1 The csin functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex csin(double complex __z) __INTRODUCED_IN(23);
 float complex csinf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex csinl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.5.1 The ctan functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex ctan(double complex __z) __INTRODUCED_IN(23);
 float complex ctanf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex ctanl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /* 7.3.6 Hyperbolic functions */
 /* 7.3.6.1 The cacosh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex cacosh(double complex __z) __INTRODUCED_IN(23);
 float complex cacoshf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex cacoshl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.6.2 The casinh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex casinh(double complex __z) __INTRODUCED_IN(23);
 float complex casinhf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex casinhl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.6.3 The catanh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex catanh(double complex __z) __INTRODUCED_IN(23);
 float complex catanhf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex catanhl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.6.4 The ccosh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex ccosh(double complex __z) __INTRODUCED_IN(23);
 float complex ccoshf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex ccoshl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.6.5 The csinh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex csinh(double complex __z) __INTRODUCED_IN(23);
 float complex csinhf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex csinhl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.6.6 The ctanh functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex ctanh(double complex __z) __INTRODUCED_IN(23);
 float complex ctanhf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex ctanhl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /* 7.3.7 Exponential and logarithmic functions */
 /* 7.3.7.1 The cexp functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex cexp(double complex __z) __INTRODUCED_IN(23);
 float complex cexpf(float complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 long double complex cexpl(long double complex __z) __INTRODUCED_IN(26);
 /* 7.3.7.2 The clog functions */
 double complex clog(double complex __z) __INTRODUCED_IN(26);
 float complex clogf(float complex __z) __INTRODUCED_IN(26);
 long double complex clogl(long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /* 7.3.8 Power and absolute-value functions */
 /* 7.3.8.1 The cabs functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double cabs(double complex __z) __INTRODUCED_IN(23);
 float cabsf(float complex __z) __INTRODUCED_IN(23);
 long double cabsl(long double complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 /* 7.3.8.2 The cpow functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 double complex cpow(double complex __x, double complex __z) __INTRODUCED_IN(26);
 float complex cpowf(float complex __x, float complex __z) __INTRODUCED_IN(26);
 long double complex cpowl(long double complex __x, long double complex __z) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 /* 7.3.8.3 The csqrt functions */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 double complex csqrt(double complex __z) __INTRODUCED_IN(23);
 float complex csqrtf(float complex __z) __INTRODUCED_IN(23);
 long double complex csqrtl(long double complex __z) __INTRODUCED_IN(23);
@@ -148,6 +262,8 @@
 double creal(double complex __z) __INTRODUCED_IN(23);
 float crealf(float complex __z) __INTRODUCED_IN(23);
 long double creall(long double complex __z) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
 
diff --git a/libc/include/ctype.h b/libc/include/ctype.h
index cb926a4..dc3f673 100644
--- a/libc/include/ctype.h
+++ b/libc/include/ctype.h
@@ -95,7 +95,7 @@
 
 /** Internal implementation detail. Do not use. */
 __attribute__((__no_sanitize__("unsigned-integer-overflow")))
-static inline int __bionic_ctype_in_range(unsigned __lo, int __ch, unsigned __hi) {
+__BIONIC_CTYPE_INLINE int __bionic_ctype_in_range(unsigned __lo, int __ch, unsigned __hi) {
   return (__BIONIC_CAST(static_cast, unsigned, __ch) - __lo) < (__hi - __lo + 1);
 }
 
diff --git a/libc/include/dirent.h b/libc/include/dirent.h
index 5333d78..8058cfb 100644
--- a/libc/include/dirent.h
+++ b/libc/include/dirent.h
@@ -33,8 +33,9 @@
  * @brief Directory entry iteration.
  */
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+
+#include <stdint.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
@@ -149,6 +150,8 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void seekdir(DIR* _Nonnull __dir, long __location) __INTRODUCED_IN(23);
 
 /**
@@ -161,6 +164,8 @@
  * Available since API level 23.
  */
 long telldir(DIR* _Nonnull __dir) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [dirfd(3)](https://man7.org/linux/man-pages/man3/dirfd.3.html)
@@ -221,6 +226,8 @@
  *
  * Available since API level 24.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int scandirat64(int __dir_fd, const char* _Nonnull __path, struct dirent64* _Nonnull * _Nonnull * _Nonnull __name_list, int (* _Nullable __filter)(const struct dirent64* _Nonnull), int (* _Nullable __comparator)(const struct dirent64* _Nonnull * _Nonnull, const struct dirent64* _Nonnull * _Nonnull)) __INTRODUCED_IN(24);
 
 /**
@@ -237,6 +244,8 @@
  * Available since API level 24.
  */
 int scandirat(int __dir_fd, const char* _Nonnull __path, struct dirent* _Nonnull * _Nonnull * _Nonnull __name_list, int (* _Nullable __filter)(const struct dirent* _Nonnull), int (* _Nullable __comparator)(const struct dirent* _Nonnull * _Nonnull, const struct dirent* _Nonnull * _Nonnull)) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 #endif
 
diff --git a/libc/include/dlfcn.h b/libc/include/dlfcn.h
index 071d50a..81045fd 100644
--- a/libc/include/dlfcn.h
+++ b/libc/include/dlfcn.h
@@ -28,9 +28,21 @@
 
 #pragma once
 
-#include <stdint.h>
 #include <sys/cdefs.h>
 
+#include <stdint.h>
+
+/**
+ * @addtogroup libdl Dynamic Linker
+ * @{
+ */
+
+/**
+ * \file
+ * Standard dynamic library support.
+ * See also the Android-specific functionality in `<android/dlext.h>`.
+ */
+
 __BEGIN_DECLS
 
 /**
@@ -51,6 +63,8 @@
  * [dlopen(3)](https://man7.org/linux/man-pages/man3/dlopen.3.html)
  * loads the given shared library.
  *
+ * See also android_dlopen_ext().
+ *
  * Returns a pointer to an opaque handle for use with other <dlfcn.h> functions
  * on success, and returns NULL on failure, in which case dlerror() can be used
  * to retrieve the specific error.
@@ -116,7 +130,11 @@
  * Returns the address of the symbol on success, and returns NULL on failure,
  * in which case dlerror() can be used to retrieve the specific error.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 void* _Nullable dlvsym(void* __BIONIC_COMPLICATED_NULLNESS __handle, const char* _Nullable __symbol, const char* _Nullable __version) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 /**
  * [dladdr(3)](https://man7.org/linux/man-pages/man3/dladdr.3.html)
@@ -186,3 +204,5 @@
 #endif
 
 __END_DECLS
+
+/** @} */
diff --git a/libc/include/err.h b/libc/include/err.h
index d8122d7..4a1841b 100644
--- a/libc/include/err.h
+++ b/libc/include/err.h
@@ -36,8 +36,9 @@
  * @brief BSD error reporting functions. See `<error.h>` for the GNU equivalent.
  */
 
-#include <stdarg.h>
 #include <sys/cdefs.h>
+
+#include <stdarg.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/error.h b/libc/include/error.h
index cb867cd..a9bdc24 100644
--- a/libc/include/error.h
+++ b/libc/include/error.h
@@ -44,6 +44,8 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 extern void (* _Nullable error_print_progname)(void) __INTRODUCED_IN(23);
 
 /**
@@ -81,5 +83,7 @@
  * Available since API level 23.
  */
 void error_at_line(int __status, int __errno, const char* _Nonnull __filename, unsigned int __line_number, const char* _Nonnull __fmt, ...) __printflike(5, 6) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/execinfo.h b/libc/include/execinfo.h
index 88f4ae7..84b637c 100644
--- a/libc/include/execinfo.h
+++ b/libc/include/execinfo.h
@@ -47,6 +47,8 @@
  *
  * Available since API level 33.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(33)
 int backtrace(void* _Nonnull * _Nonnull buffer, int size) __INTRODUCED_IN(33);
 
 /**
@@ -70,5 +72,7 @@
  * Available since API level 33.
  */
 void backtrace_symbols_fd(void* _Nonnull const* _Nonnull buffer, int size, int fd) __INTRODUCED_IN(33);
+#endif /* __BIONIC_AVAILABILITY_GUARD(33) */
+
 
 __END_DECLS
diff --git a/libc/include/fcntl.h b/libc/include/fcntl.h
index 1e9a285..2bd1fc6 100644
--- a/libc/include/fcntl.h
+++ b/libc/include/fcntl.h
@@ -227,7 +227,11 @@
  *
  * Returns 0 on success and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int sync_file_range(int __fd, off64_t __offset, off64_t __length, unsigned int __flags) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 #endif
 
diff --git a/libc/include/glob.h b/libc/include/glob.h
index 2c2b8d1..ccdf2e9 100644
--- a/libc/include/glob.h
+++ b/libc/include/glob.h
@@ -92,8 +92,12 @@
 
 __BEGIN_DECLS
 
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int glob(const char* _Nonnull __pattern, int __flags, int (* _Nullable __error_callback)(const char* _Nonnull __failure_path, int __failure_errno), glob_t* _Nonnull __result_ptr) __INTRODUCED_IN(28);
 void globfree(glob_t* _Nonnull __result_ptr) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
 
diff --git a/libc/include/grp.h b/libc/include/grp.h
index 2451db5..a48c046 100644
--- a/libc/include/grp.h
+++ b/libc/include/grp.h
@@ -51,12 +51,20 @@
 struct group* _Nullable getgrnam(const char* _Nonnull __name);
 
 /* Note: Android has thousands and thousands of ids to iterate through. */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 struct group* _Nullable getgrent(void) __INTRODUCED_IN(26);
 
 void setgrent(void) __INTRODUCED_IN(26);
 void endgrent(void) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int getgrgid_r(gid_t __gid, struct group* __BIONIC_COMPLICATED_NULLNESS __group, char* _Nonnull __buf, size_t __n, struct group* _Nullable * _Nonnull __result) __INTRODUCED_IN(24);
 int getgrnam_r(const char* _Nonnull __name, struct group* __BIONIC_COMPLICATED_NULLNESS __group, char* _Nonnull __buf, size_t __n, struct group* _Nullable *_Nonnull __result) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 int getgrouplist(const char* _Nonnull __user, gid_t __group, gid_t* __BIONIC_COMPLICATED_NULLNESS __groups, int* _Nonnull __group_count);
 int initgroups(const char* _Nonnull __user, gid_t __group);
 
diff --git a/libc/include/iconv.h b/libc/include/iconv.h
index 9da46b4..35328ee 100644
--- a/libc/include/iconv.h
+++ b/libc/include/iconv.h
@@ -60,6 +60,8 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 iconv_t _Nonnull iconv_open(const char* _Nonnull __dst_encoding, const char* _Nonnull __src_encoding) __INTRODUCED_IN(28);
 
 /**
@@ -82,5 +84,7 @@
  * Available since API level 28.
  */
 int iconv_close(iconv_t _Nonnull __converter) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/ifaddrs.h b/libc/include/ifaddrs.h
index c4d0e10..87d2947 100644
--- a/libc/include/ifaddrs.h
+++ b/libc/include/ifaddrs.h
@@ -80,6 +80,8 @@
  *
  * Available since API level 24.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int getifaddrs(struct ifaddrs* _Nullable * _Nonnull __list_ptr) __INTRODUCED_IN(24);
 
 /**
@@ -89,5 +91,7 @@
  * Available since API level 24.
  */
 void freeifaddrs(struct ifaddrs* _Nullable __ptr) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 __END_DECLS
diff --git a/libc/include/inttypes.h b/libc/include/inttypes.h
index 9fcd9f3..790030e 100644
--- a/libc/include/inttypes.h
+++ b/libc/include/inttypes.h
@@ -19,8 +19,8 @@
 #ifndef	_INTTYPES_H_
 #define	_INTTYPES_H_
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+#include <stdint.h>
 
 #ifdef __LP64__
 #define __PRI_64_prefix  "l"
diff --git a/libc/include/langinfo.h b/libc/include/langinfo.h
index 2b43892..b9d695c 100644
--- a/libc/include/langinfo.h
+++ b/libc/include/langinfo.h
@@ -92,8 +92,12 @@
 #define NOEXPR 54
 #define CRNCYSTR 55
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 char* _Nonnull nl_langinfo(nl_item __item) __INTRODUCED_IN(26);
 char* _Nonnull nl_langinfo_l(nl_item __item, locale_t _Nonnull __l) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
 
diff --git a/libc/include/limits.h b/libc/include/limits.h
index e1f566c..5e9ce59 100644
--- a/libc/include/limits.h
+++ b/libc/include/limits.h
@@ -1,6 +1,3 @@
-/*	$OpenBSD: limits.h,v 1.13 2005/12/31 19:29:38 millert Exp $	*/
-/*	$NetBSD: limits.h,v 1.7 1994/10/26 00:56:00 cgd Exp $	*/
-
 /*
  * Copyright (c) 1988 The Regents of the University of California.
  * All rights reserved.
@@ -32,108 +29,101 @@
  *	@(#)limits.h	5.9 (Berkeley) 4/3/91
  */
 
-#ifndef _LIMITS_H_
-#define _LIMITS_H_
+#pragma once
+
+/**
+ * @file limits.h
+ * @brief Constants relating to implementation limits.
+ *
+ * This file is included via `#include_next` from the clang header of the same
+ * name that provides all the limits that the compiler is responsible for,
+ * primarily those relating to integer types defined by the C standard.
+ * This file defines the additional limits defined by POSIX.
+ */
+
+/*
+ * The Android build system has bionic _before_ the clang headers,
+ * so although the claim above that clang does an `#include_next`
+ * of this file is true for the NDK, it's not true for the OS,
+ * and we need to paper over that difference here until/unless
+ * the OS build changes.
+ */
+#if __has_include_next(<limits.h>)
+#include_next <limits.h>
+#endif
 
 #include <sys/cdefs.h>
 
 /* Historically bionic exposed the content of <float.h> from <limits.h> and <sys/limits.h> too. */
 #include <float.h>
 
+/* Many of the POSIX limits come from the kernel. */
 #include <linux/limits.h>
 
-#define PASS_MAX		128	/* _PASSWORD_LEN from <pwd.h> */
-
-#define NL_ARGMAX		9
-#define NL_LANGMAX		14
-#define NL_MSGMAX		32767
-#define NL_NMAX			1
-#define NL_SETMAX		255
-#define NL_TEXTMAX		255
-
-#define TMP_MAX                 308915776
-
-/* TODO: get all these from the compiler's <limits.h>? */
-
-#define CHAR_BIT 8
-#ifdef __LP64__
-# define LONG_BIT 64
-#else
-# define LONG_BIT 32
+/*
+ * bionic always exposed these alternative names,
+ * but clang's <limits.h> considers them GNU extensions,
+ * and may or may not have defined them.
+ */
+#ifndef LONG_LONG_MIN
+/** Non-portable synonym; use LLONG_MIN directly instead. */
+#define LONG_LONG_MIN LLONG_MIN
 #endif
+#ifndef LONG_LONG_MAX
+/** Non-portable synonym; use LLONG_MAX directly instead. */
+#define LONG_LONG_MAX LLONG_MAX
+#endif
+#ifndef ULONG_LONG_MAX
+/** Non-portable synonym; use ULLONG_MAX directly instead. */
+#define ULONG_LONG_MAX ULLONG_MAX
+#endif
+
+/** Maximum number of positional arguments in a printf()/scanf() format string. */
+#define NL_ARGMAX 9
+/** Maximum number of bytes in a $LANG name. */
+#define NL_LANGMAX 14
+/** Irrelevant with Android's <nl_types.h>. */
+#define NL_MSGMAX 32767
+/** Obsolete; removed from POSIX. */
+#define NL_NMAX 1
+/** Irrelevant with Android's <nl_types.h>. */
+#define NL_SETMAX 255
+/** Irrelevant with Android's <nl_types.h>. */
+#define NL_TEXTMAX 255
+
+/** Obsolete; removed from POSIX. */
+#define PASS_MAX 128
+/** Obsolete; removed from POSIX. */
+#define TMP_MAX 308915776
+
+/** Number of bits in a `long` (POSIX). */
+#if __LP64__
+#define LONG_BIT 64
+#else
+#define LONG_BIT 32
+#endif
+/** Number of bits in a "word" of `int` (POSIX). */
 #define WORD_BIT 32
 
-#define	SCHAR_MAX	0x7f		/* max value for a signed char */
-#define SCHAR_MIN	(-0x7f-1)	/* min value for a signed char */
-
-#define	UCHAR_MAX	0xffU		/* max value for an unsigned char */
-#ifdef __CHAR_UNSIGNED__
-# define CHAR_MIN	0		/* min value for a char */
-# define CHAR_MAX	0xff		/* max value for a char */
-#else
-# define CHAR_MAX	0x7f
-# define CHAR_MIN	(-0x7f-1)
-#endif
-
-#define	USHRT_MAX	0xffffU		/* max value for an unsigned short */
-#define	SHRT_MAX	0x7fff		/* max value for a short */
-#define SHRT_MIN        (-0x7fff-1)     /* min value for a short */
-
-#define	UINT_MAX	0xffffffffU	/* max value for an unsigned int */
-#define	INT_MAX		0x7fffffff	/* max value for an int */
-#define	INT_MIN		(-0x7fffffff-1)	/* min value for an int */
-
-#ifdef __LP64__
-# define ULONG_MAX	0xffffffffffffffffUL     /* max value for unsigned long */
-# define LONG_MAX	0x7fffffffffffffffL      /* max value for a signed long */
-# define LONG_MIN	(-0x7fffffffffffffffL-1) /* min value for a signed long */
-#else
-# define ULONG_MAX	0xffffffffUL	/* max value for an unsigned long */
-# define LONG_MAX	0x7fffffffL	/* max value for a long */
-# define LONG_MIN	(-0x7fffffffL-1)/* min value for a long */
-#endif
-
-# define ULLONG_MAX	0xffffffffffffffffULL     /* max value for unsigned long long */
-# define LLONG_MAX	0x7fffffffffffffffLL      /* max value for a signed long long */
-# define LLONG_MIN	(-0x7fffffffffffffffLL-1) /* min value for a signed long long */
-
-/* GLibc compatibility definitions.
-   Note that these are defined by GCC's <limits.h>
-   only when __GNU_LIBRARY__ is defined, i.e. when
-   targetting GLibc. */
-#ifndef LONG_LONG_MIN
-#define LONG_LONG_MIN  LLONG_MIN
-#endif
-
-#ifndef LONG_LONG_MAX
-#define LONG_LONG_MAX  LLONG_MAX
-#endif
-
-#ifndef ULONG_LONG_MAX
-#define ULONG_LONG_MAX  ULLONG_MAX
-#endif
-
-#if defined(__USE_BSD) || defined(__BIONIC__) /* Historically bionic exposed these. */
-# define UID_MAX	UINT_MAX	/* max value for a uid_t */
-# define GID_MAX	UINT_MAX	/* max value for a gid_t */
-#if defined(__LP64__)
+/** Maximum value of a uid_t. */
+#define UID_MAX UINT_MAX
+/** Maximum value of a gid_t. */
+#define GID_MAX UINT_MAX
+/** Maximum value of a size_t. */
 #define SIZE_T_MAX ULONG_MAX
-#else
-#define SIZE_T_MAX UINT_MAX
-#endif
-#endif
-
-#if defined(__LP64__)
+/** Maximum value of a ssize_t. */
 #define SSIZE_MAX LONG_MAX
-#else
-#define SSIZE_MAX INT_MAX
-#endif
 
+/** Maximum number of bytes in a multibyte character. */
 #define MB_LEN_MAX 4
 
+/** Default process priority. */
 #define NZERO 20
 
+/** Maximum number of struct iovec that can be passed in a single readv()/writev(). */
 #define IOV_MAX 1024
+
+/** Maximum value for a semaphore. */
 #define SEM_VALUE_MAX 0x3fffffff
 
 /** Do not use: prefer getline() or asprintf() rather than hard-coding an arbitrary size. */
@@ -142,12 +132,17 @@
 /* POSIX says these belong in <unistd.h> but BSD has some in <limits.h>. */
 #include <bits/posix_limits.h>
 
+/** Maximum length of a hostname returned by gethostname(). */
 #define HOST_NAME_MAX _POSIX_HOST_NAME_MAX
+
+/** Maximum length of a login name. */
 #define LOGIN_NAME_MAX 256
+
+/** Maximum length of terminal device name. */
 #define TTY_NAME_MAX 32
 
-/* >= _POSIX_THREAD_DESTRUCTOR_ITERATIONS */
-#define PTHREAD_DESTRUCTOR_ITERATIONS 4
+/** Maximum number of attempts to destroy thread-specific data when a thread exits. */
+#define PTHREAD_DESTRUCTOR_ITERATIONS _POSIX_THREAD_DESTRUCTOR_ITERATIONS
 
 /**
  * The number of calls to pthread_key_create() without intervening calls to
@@ -156,7 +151,5 @@
  */
 #define PTHREAD_KEYS_MAX 128
 
-/** bionic has no specific limit on the number of threads. */
+/** bionic has no fixed limit on the number of threads. */
 #undef PTHREAD_THREADS_MAX
-
-#endif /* !_LIMITS_H_ */
diff --git a/libc/include/link.h b/libc/include/link.h
index 216502e..331070e 100644
--- a/libc/include/link.h
+++ b/libc/include/link.h
@@ -33,8 +33,9 @@
  * @brief Extra dynamic linker functionality (see also <dlfcn.h>).
  */
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+
+#include <stdint.h>
 #include <sys/types.h>
 
 #include <elf.h>
diff --git a/libc/include/malloc.h b/libc/include/malloc.h
index 2fa4b49..ac27467 100644
--- a/libc/include/malloc.h
+++ b/libc/include/malloc.h
@@ -89,9 +89,9 @@
  */
 #if __ANDROID_API__ >= 29
 __nodiscard void* _Nullable reallocarray(void* _Nullable __ptr, size_t __item_count, size_t __item_size) __BIONIC_ALLOC_SIZE(2, 3) __INTRODUCED_IN(29);
-#else
+#elif defined(__ANDROID_UNAVAILABLE_SYMBOLS_ARE_WEAK__)
 #include <errno.h>
-static __inline __nodiscard void* _Nullable reallocarray(void* _Nullable __ptr, size_t __item_count, size_t __item_size) {
+static __inline __nodiscard void* _Nullable reallocarray(void* _Nullable __ptr, size_t __item_count, size_t __item_size) __BIONIC_ALLOC_SIZE(2, 3) {
   size_t __new_size;
   if (__builtin_mul_overflow(__item_count, __item_size, &__new_size)) {
     errno = ENOMEM;
@@ -195,7 +195,11 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int malloc_info(int __must_be_zero, FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * mallopt() option to set the decay time. Valid values are -1, 0 and 1.
@@ -368,7 +372,11 @@
  *
  * Available since API level 26.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int mallopt(int __option, int __value) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /**
  * [__malloc_hook(3)](https://man7.org/linux/man-pages/man3/__malloc_hook.3.html)
@@ -379,6 +387,8 @@
  *
  * See also: [extra documentation](https://android.googlesource.com/platform/bionic/+/main/libc/malloc_hooks/README.md)
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 extern void* _Nonnull (*volatile _Nonnull __malloc_hook)(size_t __byte_count, const void* _Nonnull __caller) __INTRODUCED_IN(28);
 
 /**
@@ -413,5 +423,7 @@
  * See also: [extra documentation](https://android.googlesource.com/platform/bionic/+/main/libc/malloc_hooks/README.md)
  */
 extern void* _Nonnull (*volatile _Nonnull __memalign_hook)(size_t __alignment, size_t __byte_count, const void* _Nonnull __caller) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/math.h b/libc/include/math.h
index 343ab98..59161bf 100644
--- a/libc/include/math.h
+++ b/libc/include/math.h
@@ -350,7 +350,11 @@
 double gamma_r(double __x, int* _Nonnull __sign);
 double lgamma_r(double __x, int* _Nonnull __sign);
 double significand(double __x);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 long double lgammal_r(long double __x, int* _Nonnull __sign) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 long double significandl(long double __x);
 float dremf(float __x, float __y);
 int finitef(float __x) __attribute_const__;
diff --git a/libc/include/mntent.h b/libc/include/mntent.h
index 9a31838..4c03602 100644
--- a/libc/include/mntent.h
+++ b/libc/include/mntent.h
@@ -29,8 +29,9 @@
 #ifndef _MNTENT_H_
 #define _MNTENT_H_
 
-#include <stdio.h>
 #include <sys/cdefs.h>
+
+#include <stdio.h>
 #include <paths.h>  /* for _PATH_MOUNTED */
 
 #define MOUNTED _PATH_MOUNTED
@@ -61,7 +62,11 @@
 struct mntent* _Nullable getmntent(FILE* _Nonnull __fp);
 struct mntent* _Nullable getmntent_r(FILE* _Nonnull __fp, struct mntent* _Nonnull __entry, char* _Nonnull __buf, int __size);
 FILE* _Nullable setmntent(const char* _Nonnull __filename, const char* _Nonnull __type);
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 char* _Nullable hasmntopt(const struct mntent* _Nonnull __entry, const char* _Nonnull __option) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
 
diff --git a/libc/include/net/if.h b/libc/include/net/if.h
index 79b4195..50bc74c 100644
--- a/libc/include/net/if.h
+++ b/libc/include/net/if.h
@@ -29,9 +29,10 @@
 #ifndef _NET_IF_H_
 #define _NET_IF_H_
 
+#include <sys/cdefs.h>
+
 #include <sys/socket.h>
 #include <linux/if.h>
-#include <sys/cdefs.h>
 
 #ifndef IF_NAMESIZE
 #define IF_NAMESIZE IFNAMSIZ
@@ -46,8 +47,12 @@
 
 char* _Nullable if_indextoname(unsigned __index, char* _Nonnull __buf);
 unsigned if_nametoindex(const char* _Nonnull __name);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 struct if_nameindex* _Nullable if_nameindex(void) __INTRODUCED_IN(24);
 void if_freenameindex(struct if_nameindex* _Nullable __ptr) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 __END_DECLS
 
diff --git a/libc/include/netdb.h b/libc/include/netdb.h
index 88214d5..04aaf5c 100644
--- a/libc/include/netdb.h
+++ b/libc/include/netdb.h
@@ -212,28 +212,52 @@
 void herror(const char* _Nonnull __s);
 const char* _Nonnull hstrerror(int __error);
 struct hostent* _Nullable gethostbyaddr(const void* _Nonnull __addr, socklen_t __length, int __type);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int gethostbyaddr_r(const void* _Nonnull __addr, socklen_t __length, int __type, struct hostent* _Nonnull __ret, char* _Nonnull __buf, size_t __buf_size, struct hostent* _Nullable * _Nonnull __result, int* _Nonnull __h_errno_ptr) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 struct hostent* _Nullable gethostbyname(const char* _Nonnull __name);
 int gethostbyname_r(const char* _Nonnull __name, struct hostent* _Nonnull __ret, char* _Nonnull __buf, size_t __buf_size, struct hostent* _Nullable * _Nonnull __result, int* _Nonnull __h_errno_ptr);
 struct hostent* _Nullable gethostbyname2(const char* _Nonnull __name, int __af);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int gethostbyname2_r(const char* _Nonnull __name, int __af, struct hostent* _Nonnull __ret, char* _Nonnull __buf, size_t __buf_size, struct hostent* _Nullable * _Nonnull __result, int* _Nonnull __h_errno_ptr) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 void endhostent(void) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 struct hostent* _Nullable gethostent(void);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 void sethostent(int __stay_open) __INTRODUCED_IN(28);
 
 /* These functions are obsolete. None of these functions return anything but nullptr. */
 void endnetent(void) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 struct netent* _Nullable getnetbyaddr(uint32_t __net, int __type);
 struct netent* _Nullable getnetbyname(const char* _Nonnull __name);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 struct netent* _Nullable getnetent(void) __INTRODUCED_IN(28);
 void setnetent(int __stay_open) __INTRODUCED_IN(28);
 
 /* None of these functions return anything but nullptr. */
 void endprotoent(void) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 struct protoent* _Nullable getprotobyname(const char* _Nonnull __name);
 struct protoent* _Nullable getprotobynumber(int __proto);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 struct protoent* _Nullable getprotoent(void) __INTRODUCED_IN(28);
 void setprotoent(int __stay_open) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /* These functions return entries from a built-in database. */
 void endservent(void);
diff --git a/libc/include/netinet/icmp6.h b/libc/include/netinet/icmp6.h
index 2b237a8..ebd9f6c 100644
--- a/libc/include/netinet/icmp6.h
+++ b/libc/include/netinet/icmp6.h
@@ -65,9 +65,10 @@
 #ifndef _NETINET_ICMP6_H_
 #define _NETINET_ICMP6_H_
 
-#include <netinet/in.h> /* android-added: glibc source compatibility. */
 #include <sys/cdefs.h>
 
+#include <netinet/in.h> /* android-added: glibc source compatibility. */
+
 #define ICMPV6_PLD_MAXLEN	1232	/* IPV6_MMTU - sizeof(struct ip6_hdr)
 					   - sizeof(struct icmp6_hdr) */
 
diff --git a/libc/include/netinet/in.h b/libc/include/netinet/in.h
index 163e614..d4ce302 100644
--- a/libc/include/netinet/in.h
+++ b/libc/include/netinet/in.h
@@ -28,9 +28,10 @@
 
 #pragma once
 
+#include <sys/cdefs.h>
+
 #include <endian.h>
 #include <netinet/in6.h>
-#include <sys/cdefs.h>
 #include <sys/socket.h>
 
 #include <linux/in.h>
diff --git a/libc/include/nl_types.h b/libc/include/nl_types.h
index 6c9935d..172d80d 100644
--- a/libc/include/nl_types.h
+++ b/libc/include/nl_types.h
@@ -62,6 +62,8 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 nl_catd _Nonnull catopen(const char* _Nonnull __name, int __flag) __INTRODUCED_IN(26);
 
 /**
@@ -80,5 +82,7 @@
  * On Android, this always returns -1 with `errno` set to `EBADF`.
  */
 int catclose(nl_catd _Nonnull __catalog) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
diff --git a/libc/include/poll.h b/libc/include/poll.h
index 0dda3da..e57f812 100644
--- a/libc/include/poll.h
+++ b/libc/include/poll.h
@@ -64,7 +64,11 @@
 /**
  * Like ppoll() but allows setting a signal mask with RT signals even from a 32-bit process.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int ppoll64(struct pollfd* _Nullable  __fds, nfds_t __count, const struct timespec* _Nullable __timeout, const sigset64_t* _Nullable __mask) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 #if defined(__BIONIC_INCLUDE_FORTIFY_HEADERS)
 #define _POLL_H_
diff --git a/libc/include/pthread.h b/libc/include/pthread.h
index d718b40..cdf1b8c 100644
--- a/libc/include/pthread.h
+++ b/libc/include/pthread.h
@@ -33,11 +33,12 @@
  * @brief POSIX threads.
  */
 
+#include <sys/cdefs.h>
+
 #include <limits.h>
 #include <bits/page_size.h>
 #include <bits/pthread_types.h>
 #include <sched.h>
-#include <sys/cdefs.h>
 #include <sys/types.h>
 #include <time.h>
 
@@ -98,7 +99,11 @@
 int pthread_attr_destroy(pthread_attr_t* _Nonnull __attr);
 int pthread_attr_getdetachstate(const pthread_attr_t* _Nonnull __attr, int* _Nonnull __state);
 int pthread_attr_getguardsize(const pthread_attr_t* _Nonnull __attr, size_t* _Nonnull __size);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_attr_getinheritsched(const pthread_attr_t* _Nonnull __attr, int* _Nonnull __flag) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_attr_getschedparam(const pthread_attr_t* _Nonnull __attr, struct sched_param* _Nonnull __param);
 int pthread_attr_getschedpolicy(const pthread_attr_t* _Nonnull __attr, int* _Nonnull __policy);
 int pthread_attr_getscope(const pthread_attr_t* _Nonnull __attr, int* _Nonnull __scope);
@@ -107,7 +112,11 @@
 int pthread_attr_init(pthread_attr_t* _Nonnull __attr);
 int pthread_attr_setdetachstate(pthread_attr_t* _Nonnull __attr, int __state);
 int pthread_attr_setguardsize(pthread_attr_t* _Nonnull __attr, size_t __size);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_attr_setinheritsched(pthread_attr_t* _Nonnull __attr, int __flag) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_attr_setschedparam(pthread_attr_t* _Nonnull __attr, const struct sched_param* _Nonnull __param);
 int pthread_attr_setschedpolicy(pthread_attr_t* _Nonnull __attr, int __policy);
 int pthread_attr_setscope(pthread_attr_t* _Nonnull __attr, int __scope);
@@ -122,8 +131,12 @@
 int pthread_condattr_setpshared(pthread_condattr_t* _Nonnull __attr, int __shared);
 
 int pthread_cond_broadcast(pthread_cond_t* _Nonnull __cond);
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int pthread_cond_clockwait(pthread_cond_t* _Nonnull __cond, pthread_mutex_t* _Nonnull __mutex, clockid_t __clock,
                            const struct timespec* _Nullable __timeout) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 int pthread_cond_destroy(pthread_cond_t* _Nonnull __cond);
 int pthread_cond_init(pthread_cond_t* _Nonnull __cond, const pthread_condattr_t* _Nullable __attr);
 int pthread_cond_signal(pthread_cond_t* _Nonnull __cond);
@@ -138,8 +151,12 @@
  * Note that pthread_cond_clockwait() allows specifying an arbitrary clock and has superseded this
  * function.
  */
+
+#if (!defined(__LP64__)) || (defined(__LP64__) && __ANDROID_API__ >= 28)
 int pthread_cond_timedwait_monotonic_np(pthread_cond_t* _Nonnull __cond, pthread_mutex_t* _Nonnull __mutex,
                                         const struct timespec* _Nullable __timeout) __INTRODUCED_IN_64(28);
+#endif /* (!defined(__LP64__)) || (defined(__LP64__) && __ANDROID_API__ >= 28) */
+
 int pthread_cond_wait(pthread_cond_t* _Nonnull __cond, pthread_mutex_t* _Nonnull __mutex);
 
 int pthread_create(pthread_t* _Nonnull __pthread_ptr, pthread_attr_t const* _Nullable __attr, void* _Nullable (* _Nonnull __start_routine)(void* _Nullable), void* _Nullable);
@@ -187,14 +204,26 @@
 int pthread_mutexattr_destroy(pthread_mutexattr_t* _Nonnull __attr);
 int pthread_mutexattr_getpshared(const pthread_mutexattr_t* _Nonnull __attr, int* _Nonnull __shared);
 int pthread_mutexattr_gettype(const pthread_mutexattr_t* _Nonnull __attr, int* _Nonnull __type);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_mutexattr_getprotocol(const pthread_mutexattr_t* _Nonnull __attr, int* _Nonnull __protocol) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_mutexattr_init(pthread_mutexattr_t* _Nonnull __attr);
 int pthread_mutexattr_setpshared(pthread_mutexattr_t* _Nonnull __attr, int __shared);
 int pthread_mutexattr_settype(pthread_mutexattr_t* _Nonnull __attr, int __type);
-int pthread_mutexattr_setprotocol(pthread_mutexattr_t* _Nonnull __attr, int __protocol) __INTRODUCED_IN(28);
 
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int pthread_mutexattr_setprotocol(pthread_mutexattr_t* _Nonnull __attr, int __protocol) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int pthread_mutex_clocklock(pthread_mutex_t* _Nonnull __mutex, clockid_t __clock,
                             const struct timespec* _Nullable __abstime) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 int pthread_mutex_destroy(pthread_mutex_t* _Nonnull __mutex);
 int pthread_mutex_init(pthread_mutex_t* _Nonnull __mutex, const pthread_mutexattr_t* _Nullable __attr);
 int pthread_mutex_lock(pthread_mutex_t* _Nonnull __mutex);
@@ -209,8 +238,12 @@
  * Note that pthread_mutex_clocklock() allows specifying an arbitrary clock and has superseded this
  * function.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_mutex_timedlock_monotonic_np(pthread_mutex_t* _Nonnull __mutex, const struct timespec* _Nullable __timeout)
     __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_mutex_trylock(pthread_mutex_t* _Nonnull __mutex);
 int pthread_mutex_unlock(pthread_mutex_t* _Nonnull __mutex);
 
@@ -220,30 +253,48 @@
 int pthread_rwlockattr_destroy(pthread_rwlockattr_t* _Nonnull __attr);
 int pthread_rwlockattr_getpshared(const pthread_rwlockattr_t* _Nonnull __attr, int* _Nonnull __shared);
 int pthread_rwlockattr_setpshared(pthread_rwlockattr_t* _Nonnull __attr, int __shared);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int pthread_rwlockattr_getkind_np(const pthread_rwlockattr_t* _Nonnull __attr, int* _Nonnull __kind)
   __INTRODUCED_IN(23);
 int pthread_rwlockattr_setkind_np(pthread_rwlockattr_t* _Nonnull __attr, int __kind) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
 
+
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int pthread_rwlock_clockrdlock(pthread_rwlock_t* _Nonnull __rwlock, clockid_t __clock,
                                const struct timespec* _Nullable __timeout) __INTRODUCED_IN(30);
 int pthread_rwlock_clockwrlock(pthread_rwlock_t* _Nonnull __rwlock, clockid_t __clock,
                                const struct timespec* _Nullable __timeout) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 int pthread_rwlock_destroy(pthread_rwlock_t* _Nonnull __rwlock);
 int pthread_rwlock_init(pthread_rwlock_t* _Nonnull __rwlock, const pthread_rwlockattr_t* _Nullable __attr);
 int pthread_rwlock_rdlock(pthread_rwlock_t* _Nonnull __rwlock);
 int pthread_rwlock_timedrdlock(pthread_rwlock_t* _Nonnull __rwlock, const struct timespec* _Nullable __timeout);
 /* See the comment on pthread_mutex_timedlock_monotonic_np for usage of this function. */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_rwlock_timedrdlock_monotonic_np(pthread_rwlock_t* _Nonnull __rwlock,
                                             const struct timespec* _Nullable __timeout) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_rwlock_timedwrlock(pthread_rwlock_t* _Nonnull __rwlock, const struct timespec* _Nullable __timeout);
 /* See the comment on pthread_mutex_timedlock_monotonic_np for usage of this function. */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_rwlock_timedwrlock_monotonic_np(pthread_rwlock_t* _Nonnull __rwlock,
                                             const struct timespec* _Nullable __timeout) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int pthread_rwlock_tryrdlock(pthread_rwlock_t* _Nonnull __rwlock);
 int pthread_rwlock_trywrlock(pthread_rwlock_t* _Nonnull __rwlock);
 int pthread_rwlock_unlock(pthread_rwlock_t* _Nonnull __rwlock);
 int pthread_rwlock_wrlock(pthread_rwlock_t* _Nonnull __rwlock);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int pthread_barrierattr_init(pthread_barrierattr_t* _Nonnull __attr) __INTRODUCED_IN(24);
 int pthread_barrierattr_destroy(pthread_barrierattr_t* _Nonnull __attr) __INTRODUCED_IN(24);
 int pthread_barrierattr_getpshared(const pthread_barrierattr_t* _Nonnull __attr, int* _Nonnull __shared) __INTRODUCED_IN(24);
@@ -258,16 +309,65 @@
 int pthread_spin_lock(pthread_spinlock_t* _Nonnull __spinlock) __INTRODUCED_IN(24);
 int pthread_spin_trylock(pthread_spinlock_t* _Nonnull __spinlock) __INTRODUCED_IN(24);
 int pthread_spin_unlock(pthread_spinlock_t* _Nonnull __spinlock) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 pthread_t pthread_self(void) __attribute_const__;
 
-#if defined(__USE_GNU)
+#if defined(__USE_GNU) && __BIONIC_AVAILABILITY_GUARD(26)
+/**
+ * [pthread_getname_np(3)](https://man7.org/linux/man-pages/man3/pthread_getname_np.3.html)
+ * gets the name of the given thread.
+ * Names are at most 16 bytes (including '\0').
+ *
+ * Returns 0 on success and returns an error number on failure.
+ *
+ * Available since API level 26.
+ */
 int pthread_getname_np(pthread_t __pthread, char* _Nonnull __buf, size_t __n) __INTRODUCED_IN(26);
 #endif
-/* TODO: this should be __USE_GNU too. */
+
+/**
+ * [pthread_setname_np(3)](https://man7.org/linux/man-pages/man3/pthread_setname_np.3.html)
+ * sets the name of the given thread.
+ * Names are at most 16 bytes (including '\0').
+ * Truncation must be done by the caller;
+ * calls with longer names will fail with ERANGE.
+ *
+ * Returns 0 on success and returns an error number on failure.
+ *
+ * This should only have been available under _GNU_SOURCE,
+ * but is always available on Android by historical accident.
+ */
 int pthread_setname_np(pthread_t __pthread, const char* _Nonnull __name);
 
 /**
+ * [pthread_getaffinity_np(3)](https://man7.org/linux/man-pages/man3/pthread_getaffinity_np.3.html)
+ * gets the CPU affinity mask for the given thread.
+ *
+ * Returns 0 on success and returns an error number on failure.
+ *
+ * Available since API level 36.
+ * See sched_getaffinity() and pthread_gettid_np() for greater portability.
+ */
+#if defined(__USE_GNU) && __BIONIC_AVAILABILITY_GUARD(36)
+int pthread_getaffinity_np(pthread_t __pthread, size_t __cpu_set_size, cpu_set_t* __cpu_set) __INTRODUCED_IN(36);
+#endif
+
+/**
+ * [pthread_setaffinity_np(3)](https://man7.org/linux/man-pages/man3/pthread_setaffinity_np.3.html)
+ * sets the CPU affinity mask for the given thread.
+ *
+ * Returns 0 on success and returns an error number on failure.
+ *
+ * Available since API level 36.
+ * See sched_getaffinity() and pthread_gettid_np() for greater portability.
+ */
+#if defined(__USE_GNU) && __BIONIC_AVAILABILITY_GUARD(36)
+int pthread_setaffinity_np(pthread_t __pthread, size_t __cpu_set_size, const cpu_set_t* __cpu_set) __INTRODUCED_IN(36);
+#endif
+
+/**
  * [pthread_setschedparam(3)](https://man7.org/linux/man-pages/man3/pthread_setschedparam.3.html)
  * sets the scheduler policy and parameters of the given thread.
  *
@@ -301,7 +401,11 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pthread_setschedprio(pthread_t __pthread, int __priority) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 int pthread_setspecific(pthread_key_t __key, const void* _Nullable __value);
 
diff --git a/libc/include/pty.h b/libc/include/pty.h
index 1cfb772..92d7fbb 100644
--- a/libc/include/pty.h
+++ b/libc/include/pty.h
@@ -49,6 +49,8 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int openpty(int* _Nonnull __pty_fd, int* _Nonnull __tty_fd, char* _Nullable __tty_name, const struct termios* _Nullable __termios_ptr, const struct winsize* _Nullable __winsize_ptr) __INTRODUCED_IN(23);
 
 /**
@@ -61,5 +63,7 @@
  * Available since API level 23.
  */
 int forkpty(int* _Nonnull __parent_pty_fd, char* _Nullable __child_tty_name, const struct termios* _Nullable __termios_ptr, const struct winsize* _Nullable __winsize_ptr) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/pwd.h b/libc/include/pwd.h
index 2b17fbf..09592bc 100644
--- a/libc/include/pwd.h
+++ b/libc/include/pwd.h
@@ -84,10 +84,14 @@
 struct passwd* _Nullable getpwuid(uid_t __uid);
 
 /* Note: Android has thousands and thousands of ids to iterate through */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 struct passwd* _Nullable getpwent(void) __INTRODUCED_IN(26);
 
 void setpwent(void) __INTRODUCED_IN(26);
 void endpwent(void) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 int getpwnam_r(const char* _Nonnull __name, struct passwd* _Nonnull __pwd, char* _Nonnull __buf, size_t __n, struct passwd* _Nullable * _Nonnull __result);
 int getpwuid_r(uid_t __uid, struct passwd* _Nonnull __pwd, char* _Nonnull __buf, size_t __n, struct passwd* _Nullable * _Nonnull __result);
diff --git a/libc/include/resolv.h b/libc/include/resolv.h
index f25484a..c49cefc 100644
--- a/libc/include/resolv.h
+++ b/libc/include/resolv.h
@@ -29,9 +29,10 @@
 #ifndef _RESOLV_H_
 #define _RESOLV_H_
 
+#include <sys/cdefs.h>
+
 #include <sys/param.h>
 #include <sys/types.h>
-#include <sys/cdefs.h>
 #include <sys/socket.h>
 #include <stdio.h>
 #include <arpa/nameser.h>
@@ -60,7 +61,11 @@
 int res_search(const char* _Nonnull __name, int __class, int __type, u_char* _Nonnull __answer, int __answer_size);
 
 #define res_randomid __res_randomid
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 u_int __res_randomid(void) __INTRODUCED_IN(29);
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 
 __END_DECLS
 
diff --git a/libc/include/sched.h b/libc/include/sched.h
index e8f7736..7a2dcad 100644
--- a/libc/include/sched.h
+++ b/libc/include/sched.h
@@ -33,9 +33,10 @@
  * @brief Thread execution scheduling.
  */
 
+#include <sys/cdefs.h>
+
 #include <bits/timespec.h>
 #include <linux/sched.h>
-#include <sys/cdefs.h>
 
 __BEGIN_DECLS
 
diff --git a/libc/include/search.h b/libc/include/search.h
index 85e31ee..2f43d91 100644
--- a/libc/include/search.h
+++ b/libc/include/search.h
@@ -85,6 +85,8 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int hcreate(size_t __n) __INTRODUCED_IN(28);
 
 /**
@@ -109,6 +111,8 @@
  * Available since API level 28.
  */
 ENTRY* _Nullable hsearch(ENTRY __entry, ACTION __action) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 #if defined(__USE_BSD) || defined(__USE_GNU)
 
@@ -120,6 +124,8 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int hcreate_r(size_t __n, struct hsearch_data* _Nonnull __table) __INTRODUCED_IN(28);
 
 /**
@@ -140,6 +146,8 @@
  * Available since API level 28.
  */
 int hsearch_r(ENTRY __entry, ACTION __action, ENTRY* _Nullable * _Nonnull __result, struct hsearch_data* _Nonnull __table) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 #endif
 
diff --git a/libc/include/semaphore.h b/libc/include/semaphore.h
index 6ad9ea3..9c4702d 100644
--- a/libc/include/semaphore.h
+++ b/libc/include/semaphore.h
@@ -45,7 +45,11 @@
 
 #define SEM_FAILED __BIONIC_CAST(reinterpret_cast, sem_t*, 0)
 
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int sem_clockwait(sem_t* _Nonnull __sem, clockid_t __clock, const struct timespec* _Nonnull __ts) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 int sem_destroy(sem_t* _Nonnull __sem);
 int sem_getvalue(sem_t* _Nonnull __sem, int* _Nonnull __value);
 int sem_init(sem_t* _Nonnull __sem, int __shared, unsigned int __value);
@@ -59,7 +63,11 @@
  * Note that sem_clockwait() allows specifying an arbitrary clock and has superseded this
  * function.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sem_timedwait_monotonic_np(sem_t* _Nonnull __sem, const struct timespec* _Nonnull __ts) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int sem_trywait(sem_t* _Nonnull __sem);
 int sem_wait(sem_t* _Nonnull __sem);
 
diff --git a/libc/include/signal.h b/libc/include/signal.h
index 893fa9d..38dcbde 100644
--- a/libc/include/signal.h
+++ b/libc/include/signal.h
@@ -60,31 +60,73 @@
 #define si_timerid si_tid /* glibc compatibility. */
 
 int sigaction(int __signal, const struct sigaction* _Nullable __new_action, struct sigaction* _Nullable __old_action);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigaction64(int __signal, const struct sigaction64* _Nullable __new_action, struct sigaction64* _Nullable __old_action) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 int siginterrupt(int __signal, int __flag);
 
 sighandler_t _Nonnull signal(int __signal, sighandler_t _Nullable __handler);
 int sigaddset(sigset_t* _Nonnull __set, int __signal);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigaddset64(sigset64_t* _Nonnull __set, int __signal) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int sigdelset(sigset_t* _Nonnull __set, int __signal);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigdelset64(sigset64_t* _Nonnull __set, int __signal) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int sigemptyset(sigset_t* _Nonnull __set);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigemptyset64(sigset64_t* _Nonnull __set) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int sigfillset(sigset_t* _Nonnull __set);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigfillset64(sigset64_t* _Nonnull __set) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 int sigismember(const sigset_t* _Nonnull __set, int __signal);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigismember64(const sigset64_t* _Nonnull __set, int __signal) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 int sigpending(sigset_t* _Nonnull __set);
-int sigpending64(sigset64_t* _Nonnull __set) __INTRODUCED_IN(28);
-int sigprocmask(int __how, const sigset_t* _Nullable __new_set, sigset_t* _Nullable __old_set);
-int sigprocmask64(int __how, const sigset64_t* _Nullable __new_set, sigset64_t* _Nullable __old_set) __INTRODUCED_IN(28);
-int sigsuspend(const sigset_t* _Nonnull __mask);
-int sigsuspend64(const sigset64_t* _Nonnull __mask) __INTRODUCED_IN(28);
-int sigwait(const sigset_t* _Nonnull __set, int* _Nonnull __signal);
-int sigwait64(const sigset64_t* _Nonnull __set, int* _Nonnull __signal) __INTRODUCED_IN(28);
 
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int sigpending64(sigset64_t* _Nonnull __set) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+int sigprocmask(int __how, const sigset_t* _Nullable __new_set, sigset_t* _Nullable __old_set);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int sigprocmask64(int __how, const sigset64_t* _Nullable __new_set, sigset64_t* _Nullable __old_set) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+int sigsuspend(const sigset_t* _Nonnull __mask);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int sigsuspend64(const sigset64_t* _Nonnull __mask) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+int sigwait(const sigset_t* _Nonnull __set, int* _Nonnull __signal);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int sigwait64(const sigset64_t* _Nonnull __set, int* _Nonnull __signal) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int sighold(int __signal)
   __attribute__((__deprecated__("use sigprocmask() or pthread_sigmask() instead")))
   __INTRODUCED_IN(26);
@@ -97,6 +139,8 @@
   __INTRODUCED_IN(26);
 sighandler_t _Nonnull sigset(int __signal, sighandler_t _Nullable __handler)
   __attribute__((__deprecated__("use sigaction() instead"))) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 int raise(int __signal);
 int kill(pid_t __pid, int __signal);
@@ -110,17 +154,41 @@
 
 int pthread_kill(pthread_t __pthread, int __signal);
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 int pthread_sigqueue(pthread_t __pthread, int __signal, const union sigval __value) __INTRODUCED_IN(29);
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 #endif
 
 int pthread_sigmask(int __how, const sigset_t* _Nullable __new_set, sigset_t* _Nullable __old_set);
-int pthread_sigmask64(int __how, const sigset64_t* _Nullable __new_set, sigset64_t* _Nullable __old_set) __INTRODUCED_IN(28);
 
+#if __BIONIC_AVAILABILITY_GUARD(28)
+int pthread_sigmask64(int __how, const sigset64_t* _Nullable __new_set, sigset64_t* _Nullable __old_set) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int sigqueue(pid_t __pid, int __signal, const union sigval __value) __INTRODUCED_IN(23);
 int sigtimedwait(const sigset_t* _Nonnull __set, siginfo_t* _Nullable __info, const struct timespec* _Nullable __timeout) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigtimedwait64(const sigset64_t* _Nonnull __set, siginfo_t* _Nullable __info, const struct timespec* _Nullable __timeout) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int sigwaitinfo(const sigset_t* _Nonnull __set, siginfo_t* _Nullable __info) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int sigwaitinfo64(const sigset64_t* _Nonnull __set, siginfo_t* _Nullable __info) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /**
  * Buffer size suitable for any call to sig2str().
@@ -137,6 +205,8 @@
  *
  * Available since API level 36.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(36)
 int sig2str(int __signal, char* _Nonnull __buf) __INTRODUCED_IN(36);
 
 /**
@@ -149,6 +219,8 @@
  * Available since API level 36.
  */
 int str2sig(const char* _Nonnull __name, int* _Nonnull __signal) __INTRODUCED_IN(36);
+#endif /* __BIONIC_AVAILABILITY_GUARD(36) */
+
 
 __END_DECLS
 
diff --git a/libc/include/spawn.h b/libc/include/spawn.h
index f366239..b105754 100644
--- a/libc/include/spawn.h
+++ b/libc/include/spawn.h
@@ -55,6 +55,8 @@
 typedef struct __posix_spawnattr* posix_spawnattr_t;
 typedef struct __posix_spawn_file_actions* posix_spawn_file_actions_t;
 
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int posix_spawn(pid_t* _Nullable __pid, const char* _Nonnull __path, const posix_spawn_file_actions_t _Nullable * _Nullable __actions, const posix_spawnattr_t _Nullable * _Nullable __attr, char* const _Nullable __argv[_Nullable], char* const _Nullable __env[_Nullable]) __INTRODUCED_IN(28);
 int posix_spawnp(pid_t* _Nullable __pid, const char* _Nonnull __file, const posix_spawn_file_actions_t _Nullable * _Nullable __actions, const posix_spawnattr_t _Nullable * _Nullable __attr, char* const _Nullable __argv[_Nullable], char* const _Nullable __env[_Nullable]) __INTRODUCED_IN(28);
 
@@ -89,9 +91,15 @@
 int posix_spawn_file_actions_addopen(posix_spawn_file_actions_t _Nonnull * _Nonnull __actions, int __fd, const char* _Nonnull __path, int __flags, mode_t __mode) __INTRODUCED_IN(28);
 int posix_spawn_file_actions_addclose(posix_spawn_file_actions_t _Nonnull * _Nonnull __actions, int __fd) __INTRODUCED_IN(28);
 int posix_spawn_file_actions_adddup2(posix_spawn_file_actions_t _Nonnull * _Nonnull __actions, int __fd, int __new_fd) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
 
+
+
+#if __BIONIC_AVAILABILITY_GUARD(34)
 int posix_spawn_file_actions_addchdir_np(posix_spawn_file_actions_t _Nonnull * _Nonnull __actions, const char* _Nonnull __path) __INTRODUCED_IN(34);
 int posix_spawn_file_actions_addfchdir_np(posix_spawn_file_actions_t _Nonnull * _Nonnull __actions, int __fd) __INTRODUCED_IN(34);
+#endif /* __BIONIC_AVAILABILITY_GUARD(34) */
+
 
 __END_DECLS
 
diff --git a/libc/include/stdint.h b/libc/include/stdint.h
index 322a81c..772fe8b 100644
--- a/libc/include/stdint.h
+++ b/libc/include/stdint.h
@@ -29,9 +29,10 @@
 #ifndef _STDINT_H
 #define _STDINT_H
 
+#include <sys/cdefs.h>
+
 #include <bits/wchar_limits.h>
 #include <stddef.h>
-#include <sys/cdefs.h>
 
 typedef signed char __int8_t;
 typedef unsigned char __uint8_t;
diff --git a/libc/include/stdio.h b/libc/include/stdio.h
index d24f6af..2c2dc01 100644
--- a/libc/include/stdio.h
+++ b/libc/include/stdio.h
@@ -196,7 +196,11 @@
  *
  * Returns 0 on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int renameat2(int __old_dir_fd, const char* _Nonnull __old_path, int __new_dir_fd, const char* _Nonnull __new_path, unsigned __flags) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 
 #endif
 
@@ -205,17 +209,25 @@
 
 /* See https://android.googlesource.com/platform/bionic/+/main/docs/32-bit-abi.md */
 #if defined(__USE_FILE_OFFSET64)
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int fgetpos(FILE* _Nonnull __fp, fpos_t* _Nonnull __pos) __RENAME(fgetpos64) __INTRODUCED_IN(24);
 int fsetpos(FILE* _Nonnull __fp, const fpos_t* _Nonnull __pos) __RENAME(fsetpos64) __INTRODUCED_IN(24);
 int fseeko(FILE* _Nonnull __fp, off_t __offset, int __whence) __RENAME(fseeko64) __INTRODUCED_IN(24);
 __nodiscard off_t ftello(FILE* _Nonnull __fp) __RENAME(ftello64) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #  if defined(__USE_BSD)
 /* If __read_fn and __write_fn are both nullptr, it will cause EINVAL */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 __nodiscard FILE* _Nullable funopen(const void* _Nullable __cookie,
               int (* __BIONIC_COMPLICATED_NULLNESS __read_fn)(void* _Nonnull, char* _Nonnull, int),
               int (* __BIONIC_COMPLICATED_NULLNESS __write_fn)(void* _Nonnull, const char* _Nonnull, int),
               fpos_t (* _Nullable __seek_fn)(void* _Nonnull, fpos_t, int),
               int (* _Nullable __close_fn)(void* _Nonnull)) __RENAME(funopen64) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #  endif
 #else
 int fgetpos(FILE* _Nonnull __fp, fpos_t* _Nonnull __pos);
@@ -231,25 +243,45 @@
               int (* _Nullable __close_fn)(void* _Nonnull));
 #  endif
 #endif
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int fgetpos64(FILE* _Nonnull __fp, fpos64_t* _Nonnull __pos) __INTRODUCED_IN(24);
 int fsetpos64(FILE* _Nonnull __fp, const fpos64_t* _Nonnull __pos) __INTRODUCED_IN(24);
 int fseeko64(FILE* _Nonnull __fp, off64_t __offset, int __whence) __INTRODUCED_IN(24);
 __nodiscard off64_t ftello64(FILE* _Nonnull __fp) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #if defined(__USE_BSD)
 /* If __read_fn and __write_fn are both nullptr, it will cause EINVAL */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 __nodiscard FILE* _Nullable funopen64(const void* _Nullable __cookie,
                 int (* __BIONIC_COMPLICATED_NULLNESS __read_fn)(void* _Nonnull, char* _Nonnull, int),
                 int (* __BIONIC_COMPLICATED_NULLNESS __write_fn)(void* _Nonnull, const char* _Nonnull, int),
                 fpos64_t (* _Nullable __seek_fn)(void* _Nonnull, fpos64_t, int),
                 int (* _Nullable __close_fn)(void* _Nonnull)) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #endif
 
 __nodiscard FILE* _Nullable fopen(const char* _Nonnull __path, const char* _Nonnull __mode);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 __nodiscard FILE* _Nullable fopen64(const char* _Nonnull __path, const char* _Nonnull __mode) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 FILE* _Nullable freopen(const char* _Nullable __path, const char* _Nonnull __mode, FILE* _Nonnull __fp);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 FILE* _Nullable freopen64(const char* _Nullable __path, const char* _Nonnull __mode, FILE* _Nonnull __fp) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 __nodiscard FILE* _Nullable tmpfile(void);
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 __nodiscard FILE* _Nullable tmpfile64(void) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 int snprintf(char* __BIONIC_COMPLICATED_NULLNESS __buf, size_t __size, const char* _Nonnull __fmt, ...) __printflike(3, 4);
 int vfscanf(FILE* _Nonnull __fp, const char* _Nonnull __fmt, va_list __args) __scanflike(2, 0);
@@ -258,7 +290,11 @@
 int vsscanf(const char* _Nonnull __s, const char* _Nonnull __fmt, va_list __args) __scanflike(2, 0);
 
 #define L_ctermid 1024 /* size for ctermid() */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 char* _Nonnull ctermid(char* _Nullable __buf) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __nodiscard FILE* _Nullable fdopen(int __fd, const char* _Nonnull __mode);
 __nodiscard int fileno(FILE* _Nonnull __fp);
@@ -272,8 +308,12 @@
 int putc_unlocked(int __ch, FILE* _Nonnull __fp);
 int putchar_unlocked(int __ch);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 __nodiscard FILE* _Nullable fmemopen(void* _Nullable __buf, size_t __size, const char* _Nonnull __mode) __INTRODUCED_IN(23);
 __nodiscard FILE* _Nullable open_memstream(char* _Nonnull * _Nonnull __ptr, size_t* _Nonnull __size_ptr) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 #if defined(__USE_BSD) || defined(__BIONIC__) /* Historically bionic exposed these. */
 int  asprintf(char* _Nullable * _Nonnull __s_ptr, const char* _Nonnull __fmt, ...) __printflike(2, 3);
@@ -282,25 +322,41 @@
 void setbuffer(FILE* _Nonnull __fp, char* _Nullable __buf, int __size);
 int setlinebuf(FILE* _Nonnull __fp);
 int vasprintf(char* _Nullable * _Nonnull __s_ptr, const char* _Nonnull __fmt, va_list __args) __printflike(2, 0);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void clearerr_unlocked(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
 __nodiscard int feof_unlocked(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
 __nodiscard int ferror_unlocked(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 __nodiscard int fileno_unlocked(FILE* _Nonnull __fp) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #define fropen(cookie, fn) funopen(cookie, fn, 0, 0, 0)
 #define fwopen(cookie, fn) funopen(cookie, 0, fn, 0, 0)
 #endif
 
 #if defined(__USE_BSD)
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int fflush_unlocked(FILE* _Nullable __fp) __INTRODUCED_IN(28);
 __nodiscard int fgetc_unlocked(FILE* _Nonnull __fp) __INTRODUCED_IN(28);
 int fputc_unlocked(int __ch, FILE* _Nonnull __fp) __INTRODUCED_IN(28);
 size_t fread_unlocked(void* _Nonnull __buf, size_t __size, size_t __count, FILE* _Nonnull __fp) __INTRODUCED_IN(28);
 size_t fwrite_unlocked(const void* _Nonnull __buf, size_t __size, size_t __count, FILE* _Nonnull __fp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 #endif
 
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int fputs_unlocked(const char* _Nonnull __s, FILE* _Nonnull __fp) __INTRODUCED_IN(28);
 char* _Nullable fgets_unlocked(char* _Nonnull __buf, int __size, FILE* _Nonnull __fp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 #endif
 
 #if defined(__BIONIC_INCLUDE_FORTIFY_HEADERS)
diff --git a/libc/include/stdio_ext.h b/libc/include/stdio_ext.h
index d426a4a..9ff07da 100644
--- a/libc/include/stdio_ext.h
+++ b/libc/include/stdio_ext.h
@@ -44,6 +44,8 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 size_t __fbufsize(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
 
 /**
@@ -53,6 +55,8 @@
  * Available since API level 23.
  */
 int __freadable(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [__freading(3)](https://man7.org/linux/man-pages/man3/__freading.3.html) returns non-zero if
@@ -60,7 +64,11 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int __freading(FILE* _Nonnull __fp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /**
  * [__fwritable(3)](https://man7.org/linux/man-pages/man3/__fwritable.3.html) returns non-zero if
@@ -68,7 +76,11 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int __fwritable(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [__fwriting(3)](https://man7.org/linux/man-pages/man3/__fwriting.3.html) returns non-zero if
@@ -76,7 +88,11 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int __fwriting(FILE* _Nonnull __fp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /**
  * [__flbf(3)](https://man7.org/linux/man-pages/man3/__flbf.3.html) returns non-zero if
@@ -84,7 +100,11 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int __flbf(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [__fpurge(3)](https://man7.org/linux/man-pages/man3/__fpurge.3.html) discards the contents of
@@ -98,7 +118,11 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 size_t __fpending(FILE* _Nonnull __fp) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * __freadahead(3) returns the number of bytes in the input buffer.
@@ -106,7 +130,11 @@
  *
  * Available since API level 34.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(34)
 size_t __freadahead(FILE* _Nonnull __fp) __INTRODUCED_IN(34);
+#endif /* __BIONIC_AVAILABILITY_GUARD(34) */
+
 
 /**
  * [_flushlbf(3)](https://man7.org/linux/man-pages/man3/_flushlbf.3.html) flushes all
@@ -114,7 +142,11 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void _flushlbf(void) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * `__fseterr` sets the
@@ -122,7 +154,11 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 void __fseterr(FILE* _Nonnull __fp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /** __fsetlocking() constant to query locking type. */
 #define FSETLOCKING_QUERY 0
@@ -139,6 +175,10 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int __fsetlocking(FILE* _Nonnull __fp, int __type) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/stdlib.h b/libc/include/stdlib.h
index 076a978..7081d7c 100644
--- a/libc/include/stdlib.h
+++ b/libc/include/stdlib.h
@@ -29,11 +29,12 @@
 #ifndef _STDLIB_H
 #define _STDLIB_H
 
+#include <sys/cdefs.h>
+
 #include <alloca.h>
 #include <bits/wait.h>
 #include <malloc.h>
 #include <stddef.h>
-#include <sys/cdefs.h>
 #include <xlocale.h>
 
 __BEGIN_DECLS
@@ -59,13 +60,21 @@
 char* _Nullable mkdtemp(char* _Nonnull __template);
 char* _Nullable mktemp(char* _Nonnull __template) __attribute__((__deprecated__("mktemp is unsafe, use mkstemp or tmpfile instead")));
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int mkostemp64(char* _Nonnull __template, int __flags) __INTRODUCED_IN(23);
 int mkostemp(char* _Nonnull __template, int __flags) __INTRODUCED_IN(23);
 int mkostemps64(char* _Nonnull __template, int __suffix_length, int __flags) __INTRODUCED_IN(23);
 int mkostemps(char* _Nonnull __template, int __suffix_length, int __flags) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 int mkstemp64(char* _Nonnull __template);
 int mkstemp(char* _Nonnull __template);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int mkstemps64(char* _Nonnull __template, int __flags) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 int mkstemps(char* _Nonnull __template, int __flags);
 
 int posix_memalign(void* _Nullable * _Nullable __memptr, size_t __alignment, size_t __size);
@@ -79,7 +88,11 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 __nodiscard void* _Nullable aligned_alloc(size_t __alignment, size_t __size) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __nodiscard char* _Nullable realpath(const char* _Nonnull __path, char* _Nullable __resolved);
 
@@ -122,7 +135,11 @@
  *
  * Available since API level 36.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(36)
 void qsort_r(void* _Nullable __array, size_t __n, size_t __size, int (* _Nonnull __comparator)(const void* _Nullable __lhs, const void* _Nullable __rhs, void* _Nullable __context), void* _Nullable __context) __INTRODUCED_IN(36);
+#endif /* __BIONIC_AVAILABILITY_GUARD(36) */
+
 
 uint32_t arc4random(void);
 uint32_t arc4random_uniform(uint32_t __upper_bound);
@@ -135,7 +152,11 @@
 double drand48(void);
 double erand48(unsigned short __xsubi[_Nonnull 3]);
 long jrand48(unsigned short __xsubi[_Nonnull 3]);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void lcong48(unsigned short __param[_Nonnull 7]) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 long lrand48(void);
 long mrand48(void);
 long nrand48(unsigned short __xsubi[_Nonnull 3]);
@@ -151,7 +172,11 @@
 int ptsname_r(int __fd, char* _Nonnull __buf, size_t __n);
 int unlockpt(int __fd);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int getsubopt(char* _Nonnull * _Nonnull __option, char* _Nonnull const* _Nonnull __tokens, char* _Nullable * _Nonnull __value_ptr) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 typedef struct {
   int quot;
@@ -181,13 +206,21 @@
  *
  * Returns the number of samples written to `__averages` (at most 3), and returns -1 on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 int getloadavg(double __averages[_Nonnull], int __n) __INTRODUCED_IN(29);
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 
 /* BSD compatibility. */
 const char* _Nullable getprogname(void);
 void setprogname(const char* _Nonnull __name);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int mblen(const char* _Nullable __s, size_t __n) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 size_t mbstowcs(wchar_t* _Nullable __dst, const char* _Nullable __src, size_t __n);
 int mbtowc(wchar_t* _Nullable __wc_ptr, const char*  _Nullable __s, size_t __n);
 int wctomb(char* _Nullable __dst, wchar_t __wc);
diff --git a/libc/include/string.h b/libc/include/string.h
index 7c1c3be..79aac91 100644
--- a/libc/include/string.h
+++ b/libc/include/string.h
@@ -52,7 +52,11 @@
 int memcmp(const void* _Nonnull __lhs, const void* _Nonnull __rhs, size_t __n) __attribute_pure__;
 void* _Nonnull memcpy(void* _Nonnull, const void* _Nonnull, size_t);
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 void* _Nonnull mempcpy(void* _Nonnull __dst, const void* _Nonnull __src, size_t __n) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 #endif
 void* _Nonnull memmove(void* _Nonnull __dst, const void* _Nonnull __src, size_t __n);
 
@@ -71,7 +75,11 @@
  *
  * Returns `dst`.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(34)
 void* _Nonnull memset_explicit(void* _Nonnull __dst, int __ch, size_t __n) __INTRODUCED_IN(34);
+#endif /* __BIONIC_AVAILABILITY_GUARD(34) */
+
 
 void* _Nullable memmem(const void* _Nonnull __haystack, size_t __haystack_size, const void* _Nonnull __needle, size_t __needle_size) __attribute_pure__;
 
@@ -79,10 +87,18 @@
 char* _Nullable __strchr_chk(const char* _Nonnull __s, int __ch, size_t __n);
 #if defined(__USE_GNU)
 #if defined(__cplusplus)
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 extern "C++" char* _Nonnull strchrnul(char* _Nonnull __s, int __ch) __RENAME(strchrnul) __attribute_pure__ __INTRODUCED_IN(24);
 extern "C++" const char* _Nonnull strchrnul(const char* _Nonnull __s, int __ch) __RENAME(strchrnul) __attribute_pure__ __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #else
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 char* _Nonnull strchrnul(const char* _Nonnull __s, int __ch) __attribute_pure__ __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 #endif
 #endif
 
@@ -156,7 +172,11 @@
  * Available since API level 35.
  */
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 const char* _Nullable strerrorname_np(int __errno_value) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 #endif
 
 /**
@@ -199,10 +219,18 @@
  * It doesn't modify its argument, and in C++ it's const-correct.
  */
 #if defined(__cplusplus)
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 extern "C++" char* _Nonnull basename(char* _Nullable __path) __RENAME(__gnu_basename) __INTRODUCED_IN(23);
 extern "C++" const char* _Nonnull basename(const char* _Nonnull __path) __RENAME(__gnu_basename) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 #else
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 char* _Nonnull basename(const char* _Nonnull __path) __RENAME(__gnu_basename) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 #endif
 #endif
 
diff --git a/libc/include/strings.h b/libc/include/strings.h
index d203bd2..7543edc 100644
--- a/libc/include/strings.h
+++ b/libc/include/strings.h
@@ -43,8 +43,9 @@
  * @brief Extra string functions.
  */
 
-#include <sys/types.h>
 #include <sys/cdefs.h>
+
+#include <sys/types.h>
 #include <xlocale.h>
 
 #include <bits/strcasecmp.h>
diff --git a/libc/include/sys/cachectl.h b/libc/include/sys/cachectl.h
index b5fabe3..d06d683 100644
--- a/libc/include/sys/cachectl.h
+++ b/libc/include/sys/cachectl.h
@@ -48,7 +48,7 @@
 
 /**
  * __riscv_flush_icache(2) flushes the instruction cache for the given range of addresses.
- * The address range is currently (Linux 6.4) ignored, so both pointers may be null.
+ * The address range is currently (Linux 6.12) ignored, so both pointers may be null.
  *
  * Returns 0 on success, and returns -1 and sets `errno` on failure.
  */
diff --git a/libc/include/sys/epoll.h b/libc/include/sys/epoll.h
index a5e3c14..bec7c64 100644
--- a/libc/include/sys/epoll.h
+++ b/libc/include/sys/epoll.h
@@ -88,13 +88,19 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int epoll_pwait64(int __epoll_fd, struct epoll_event* _Nonnull __events, int __event_count, int __timeout_ms, const sigset64_t* _Nullable __mask) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 /**
  * Like epoll_pwait() but with a `struct timespec` timeout, for nanosecond resolution.
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 int epoll_pwait2(int __epoll_fd, struct epoll_event* _Nonnull __events, int __event_count, const struct timespec* _Nullable __timeout, const sigset_t* _Nullable __mask) __INTRODUCED_IN(35);
 
 /**
@@ -103,5 +109,7 @@
  * Available since API level 35.
  */
 int epoll_pwait2_64(int __epoll_fd, struct epoll_event* _Nonnull __events, int __event_count, const struct timespec* _Nullable __timeout, const sigset64_t* _Nullable __mask) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/mman.h b/libc/include/sys/mman.h
index 1a0e7f6..38cbf2f 100644
--- a/libc/include/sys/mman.h
+++ b/libc/include/sys/mman.h
@@ -133,7 +133,11 @@
  *
  * Returns 0 on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int mlock2(const void* _Nonnull __addr, size_t __size, int __flags) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 
 /**
  * [munlock(2)](https://man7.org/linux/man-pages/man2/munlock.2.html)
@@ -171,7 +175,11 @@
  *
  * Returns the number of bytes advised on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(31)
 ssize_t process_madvise(int __pid_fd, const struct iovec* _Nonnull __iov, size_t __count, int __advice, unsigned __flags) __INTRODUCED_IN(31);
+#endif /* __BIONIC_AVAILABILITY_GUARD(31) */
+
 
 #if defined(__USE_GNU)
 
@@ -183,7 +191,11 @@
  *
  * Returns an fd on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int memfd_create(const char* _Nonnull __name, unsigned __flags) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 
 #endif
 
@@ -220,7 +232,11 @@
  *
  * Returns 0 on success, and returns a positive error number on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int posix_madvise(void* _Nonnull __addr, size_t __size, int __advice) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [mseal(2)](https://man7.org/linux/man-pages/man2/mseal.2.html)
@@ -232,6 +248,10 @@
  *
  * Returns 0 on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(36)
 int mseal(void* _Nonnull __addr, size_t __size, unsigned long __flags) __INTRODUCED_IN(36);
+#endif /* __BIONIC_AVAILABILITY_GUARD(36) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/msg.h b/libc/include/sys/msg.h
index 26071b1..8b619be 100644
--- a/libc/include/sys/msg.h
+++ b/libc/include/sys/msg.h
@@ -46,6 +46,8 @@
 typedef __kernel_ulong_t msglen_t;
 
 /** Not useful on Android; disallowed by SELinux. */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int msgctl(int __msg_id, int __op, struct msqid_ds* _Nullable __buf) __INTRODUCED_IN(26);
 /** Not useful on Android; disallowed by SELinux. */
 int msgget(key_t __key, int __flags) __INTRODUCED_IN(26);
@@ -53,5 +55,7 @@
 ssize_t msgrcv(int __msg_id, void* _Nonnull __msgbuf_ptr, size_t __size, long __type, int __flags) __INTRODUCED_IN(26);
 /** Not useful on Android; disallowed by SELinux. */
 int msgsnd(int __msg_id, const void* _Nonnull __msgbuf_ptr, size_t __size, int __flags) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/param.h b/libc/include/sys/param.h
index 1c991ae..99b6a07 100644
--- a/libc/include/sys/param.h
+++ b/libc/include/sys/param.h
@@ -33,10 +33,11 @@
  * @brief Various macros.
  */
 
+#include <sys/cdefs.h>
+
 #include <endian.h>
 #include <limits.h>
 #include <linux/param.h>
-#include <sys/cdefs.h>
 
 /** The unit of `st_blocks` in `struct stat`. */
 #define DEV_BSIZE 512
diff --git a/libc/include/sys/pidfd.h b/libc/include/sys/pidfd.h
index 30455bb..aaf49c9 100644
--- a/libc/include/sys/pidfd.h
+++ b/libc/include/sys/pidfd.h
@@ -49,6 +49,8 @@
  *
  * Available since API level 31.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(31)
 int pidfd_open(pid_t __pid, unsigned int __flags) __INTRODUCED_IN(31);
 
 /**
@@ -72,5 +74,7 @@
  * Available since API level 31.
  */
 int pidfd_send_signal(int __pidfd, int __sig, siginfo_t * _Nullable __info, unsigned int __flags) __INTRODUCED_IN(31);
+#endif /* __BIONIC_AVAILABILITY_GUARD(31) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/quota.h b/libc/include/sys/quota.h
index 6e32705..af09674 100644
--- a/libc/include/sys/quota.h
+++ b/libc/include/sys/quota.h
@@ -51,6 +51,10 @@
  *
  * Available since API level 26.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int quotactl(int __op, const char* _Nullable __special, int __id, char* __BIONIC_COMPLICATED_NULLNESS __addr) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/random.h b/libc/include/sys/random.h
index b4a9993..23d2c3a 100644
--- a/libc/include/sys/random.h
+++ b/libc/include/sys/random.h
@@ -52,6 +52,10 @@
  *
  * See also arc4random_buf() which is available in all API levels.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 __nodiscard ssize_t getrandom(void* _Nonnull __buffer, size_t __buffer_size, unsigned int __flags) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/resource.h b/libc/include/sys/resource.h
index 6743343..05ef2c2 100644
--- a/libc/include/sys/resource.h
+++ b/libc/include/sys/resource.h
@@ -54,7 +54,11 @@
 
 int getrusage(int __who, struct rusage* _Nonnull __usage);
 
+
+#if (!defined(__LP64__) && __ANDROID_API__ >= 24) || (defined(__LP64__))
 int prlimit(pid_t __pid, int __resource, const struct rlimit* _Nullable __new_limit, struct rlimit* _Nullable __old_limit) __INTRODUCED_IN_32(24) __INTRODUCED_IN_64(21);
+#endif /* (!defined(__LP64__) && __ANDROID_API__ >= 24) || (defined(__LP64__)) */
+
 int prlimit64(pid_t __pid, int __resource, const struct rlimit64* _Nullable __new_limit, struct rlimit64* _Nullable __old_limit);
 
 __END_DECLS
diff --git a/libc/include/sys/select.h b/libc/include/sys/select.h
index d5b3495..a7227b0 100644
--- a/libc/include/sys/select.h
+++ b/libc/include/sys/select.h
@@ -119,6 +119,10 @@
  *
  * Available since API level 28.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int pselect64(int __max_fd_plus_one, fd_set* _Nullable __read_fds, fd_set* _Nullable __write_fds, fd_set* _Nullable __exception_fds, const struct timespec* _Nullable __timeout, const sigset64_t* _Nullable __mask) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/sem.h b/libc/include/sys/sem.h
index 5682282..72f567e 100644
--- a/libc/include/sys/sem.h
+++ b/libc/include/sys/sem.h
@@ -51,12 +51,20 @@
   void* _Nullable __pad;
 };
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int semctl(int __sem_id, int __sem_num, int __op, ...) __INTRODUCED_IN(26);
 int semget(key_t __key, int __sem_count, int __flags) __INTRODUCED_IN(26);
 int semop(int __sem_id, struct sembuf* _Nonnull __ops, size_t __op_count) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int semtimedop(int __sem_id, struct sembuf* _Nonnull __ops, size_t __op_count, const struct timespec* _Nullable __timeout) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 #endif
 
 __END_DECLS
diff --git a/libc/include/sys/shm.h b/libc/include/sys/shm.h
index 8ab3d9a..a960781 100644
--- a/libc/include/sys/shm.h
+++ b/libc/include/sys/shm.h
@@ -48,6 +48,8 @@
 typedef unsigned long shmatt_t;
 
 /** Not useful on Android; disallowed by SELinux. */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 void* _Nonnull shmat(int __shm_id, const void* _Nullable __addr, int __flags) __INTRODUCED_IN(26);
 /** Not useful on Android; disallowed by SELinux. */
 int shmctl(int __shm_id, int __op, struct shmid_ds* _Nullable __buf) __INTRODUCED_IN(26);
@@ -55,5 +57,7 @@
 int shmdt(const void* _Nonnull __addr) __INTRODUCED_IN(26);
 /** Not useful on Android; disallowed by SELinux. */
 int shmget(key_t __key, size_t __size, int __flags) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/signalfd.h b/libc/include/sys/signalfd.h
index 5568b7d..eaea525 100644
--- a/libc/include/sys/signalfd.h
+++ b/libc/include/sys/signalfd.h
@@ -51,6 +51,10 @@
 /**
  * Like signalfd() but allows setting a signal mask with RT signals even from a 32-bit process.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int signalfd64(int __fd, const sigset64_t* _Nonnull __mask, int __flags) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/stat.h b/libc/include/sys/stat.h
index 2633b69..0b4b248 100644
--- a/libc/include/sys/stat.h
+++ b/libc/include/sys/stat.h
@@ -33,9 +33,10 @@
  * @brief File status.
  */
 
+#include <sys/cdefs.h>
+
 #include <bits/timespec.h>
 #include <linux/stat.h>
-#include <sys/cdefs.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
@@ -176,7 +177,11 @@
  *
  * Returns 0 on success and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(36)
 int lchmod(const char* _Nonnull __path, mode_t __mode) __INTRODUCED_IN(36);
+#endif /* __BIONIC_AVAILABILITY_GUARD(36) */
+
 
 /**
  * [mkdir(2)](https://man7.org/linux/man-pages/man2/mkdir.2.html)
@@ -280,7 +285,11 @@
  *
  * Returns 0 on success and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int mkfifoat(int __dir_fd, const char* _Nonnull __path, mode_t __mode) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * Used in the tv_nsec field of an argument to utimensat()/futimens()
@@ -331,7 +340,11 @@
  *
  * Available since API level 30.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(30)
 int statx(int __dir_fd, const char* _Nullable __path, int __flags, unsigned __mask, struct statx* _Nonnull __buf) __INTRODUCED_IN(30);
+#endif /* __BIONIC_AVAILABILITY_GUARD(30) */
+
 #endif
 
 __END_DECLS
diff --git a/libc/include/sys/statvfs.h b/libc/include/sys/statvfs.h
index 2feca81..860824b 100644
--- a/libc/include/sys/statvfs.h
+++ b/libc/include/sys/statvfs.h
@@ -21,8 +21,9 @@
  * @brief Filesystem statistics.
  */
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+
+#include <stdint.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/sys/syscall.h b/libc/include/sys/syscall.h
index a49323d..9341ffb 100644
--- a/libc/include/sys/syscall.h
+++ b/libc/include/sys/syscall.h
@@ -29,9 +29,10 @@
 #ifndef _SYS_SYSCALL_H_
 #define _SYS_SYSCALL_H_
 
+#include <sys/cdefs.h>
+
 #include <asm/unistd.h> /* Linux kernel __NR_* names. */
 #include <bits/glibc-syscalls.h> /* glibc-compatible SYS_* aliases. */
-#include <sys/cdefs.h>
 
 /* The syscall function itself is declared in <unistd.h>, not here. */
 
diff --git a/libc/include/sys/sysinfo.h b/libc/include/sys/sysinfo.h
index 5956feb..ed6a007 100644
--- a/libc/include/sys/sysinfo.h
+++ b/libc/include/sys/sysinfo.h
@@ -53,6 +53,8 @@
  *
  * See also sysconf().
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int get_nprocs_conf(void) __INTRODUCED_IN(23);
 
 /**
@@ -84,5 +86,7 @@
  * See also sysconf().
  */
 long get_avphys_pages(void) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/system_properties.h b/libc/include/sys/system_properties.h
index e8b6e34..1303079 100644
--- a/libc/include/sys/system_properties.h
+++ b/libc/include/sys/system_properties.h
@@ -71,9 +71,13 @@
  *
  * Available since API level 26.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 void __system_property_read_callback(const prop_info* _Nonnull __pi,
     void (* _Nonnull __callback)(void* _Nullable __cookie, const char* _Nonnull __name, const char* _Nonnull __value, uint32_t __serial),
     void* _Nullable __cookie) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /**
  * Passes a `prop_info` for each system property to the provided
@@ -101,8 +105,12 @@
  * Available since API level 26.
  */
 struct timespec;
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 bool __system_property_wait(const prop_info* _Nullable __pi, uint32_t __old_serial, uint32_t* _Nonnull __new_serial_ptr, const struct timespec* _Nullable __relative_timeout)
     __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /**
  * Deprecated: there's no limit on the length of a property name since
@@ -233,7 +241,11 @@
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 int __system_properties_zygote_reload(void) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 /**
  * Deprecated: previously for testing, but now that SystemProperties is its own
diff --git a/libc/include/sys/thread_properties.h b/libc/include/sys/thread_properties.h
index efd212a..b6214ee 100644
--- a/libc/include/sys/thread_properties.h
+++ b/libc/include/sys/thread_properties.h
@@ -50,6 +50,8 @@
  *
  * Available since API level 31.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(31)
 void __libc_get_static_tls_bounds(void* _Nonnull * _Nonnull __static_tls_begin,
                                   void* _Nonnull * _Nonnull __static_tls_end) __INTRODUCED_IN(31);
 
@@ -93,5 +95,7 @@
                           void* _Nonnull __dynamic_tls_end),
     void (* _Nonnull __on_destruction)(void* _Nonnull __dynamic_tls_begin,
                              void* _Nonnull __dynamic_tls_end)) __INTRODUCED_IN(31);
+#endif /* __BIONIC_AVAILABILITY_GUARD(31) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/time.h b/libc/include/sys/time.h
index 6ba7a37..d12c306 100644
--- a/libc/include/sys/time.h
+++ b/libc/include/sys/time.h
@@ -47,8 +47,12 @@
 int utimes(const char* _Nonnull __path, const struct timeval __times[_Nullable 2]);
 
 #if defined(__USE_BSD)
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int futimes(int __fd, const struct timeval __times[_Nullable 2]) __INTRODUCED_IN(26);
 int lutimes(const char* _Nonnull __path, const struct timeval __times[_Nullable 2]) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 #endif
 
 #if defined(__USE_GNU)
@@ -65,7 +69,11 @@
  *
  * Available since API level 26.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int futimesat(int __dir_fd, const char* __BIONIC_COMPLICATED_NULLNESS __path, const struct timeval __times[_Nullable 2]) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 #endif
 
 #define timerclear(a)   \
diff --git a/libc/include/sys/timerfd.h b/libc/include/sys/timerfd.h
index bfa9a55..f7f1ffa 100644
--- a/libc/include/sys/timerfd.h
+++ b/libc/include/sys/timerfd.h
@@ -33,10 +33,11 @@
  * @brief Timer file descriptors.
  */
 
+#include <sys/cdefs.h>
+
 #include <fcntl.h>
 #include <linux/timerfd.h>
 #include <time.h>
-#include <sys/cdefs.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/sys/timex.h b/libc/include/sys/timex.h
index 828eb47..6fb58e4 100644
--- a/libc/include/sys/timex.h
+++ b/libc/include/sys/timex.h
@@ -46,6 +46,8 @@
  *
  * Available since API level 24.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 int adjtimex(struct timex* _Nonnull __buf) __INTRODUCED_IN(24);
 
 /**
@@ -56,5 +58,7 @@
  * Available since API level 24.
  */
 int clock_adjtime(clockid_t __clock, struct timex* _Nonnull __tx) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 __END_DECLS
diff --git a/libc/include/sys/types.h b/libc/include/sys/types.h
index 4622a4e..0446260 100644
--- a/libc/include/sys/types.h
+++ b/libc/include/sys/types.h
@@ -29,9 +29,10 @@
 #ifndef _SYS_TYPES_H_
 #define _SYS_TYPES_H_
 
+#include <sys/cdefs.h>
+
 #include <stddef.h>
 #include <stdint.h>
-#include <sys/cdefs.h>
 
 #include <linux/types.h>
 #include <linux/posix_types.h>
diff --git a/libc/include/sys/uio.h b/libc/include/sys/uio.h
index d3e6561..eff3b14 100644
--- a/libc/include/sys/uio.h
+++ b/libc/include/sys/uio.h
@@ -69,6 +69,8 @@
  *
  * Available since API level 24.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(24)
 ssize_t preadv(int __fd, const struct iovec* _Nonnull __iov, int __count, off_t __offset) __RENAME_IF_FILE_OFFSET64(preadv64) __INTRODUCED_IN(24);
 
 /**
@@ -96,6 +98,8 @@
  * Available since API level 24.
  */
 ssize_t pwritev64(int __fd, const struct iovec* _Nonnull __iov, int __count, off64_t __offset) __INTRODUCED_IN(24);
+#endif /* __BIONIC_AVAILABILITY_GUARD(24) */
+
 
 /**
  * [preadv2(2)](https://man7.org/linux/man-pages/man2/preadv2.2.html) reads
@@ -107,6 +111,8 @@
  *
  * Available since API level 33.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(33)
 ssize_t preadv2(int __fd, const struct iovec* _Nonnull __iov, int __count, off_t __offset, int __flags) __RENAME_IF_FILE_OFFSET64(preadv64v2) __INTRODUCED_IN(33);
 
 /**
@@ -134,6 +140,8 @@
  * Available since API level 33.
  */
 ssize_t pwritev64v2(int __fd, const struct iovec* _Nonnull __iov, int __count, off64_t __offset, int __flags) __INTRODUCED_IN(33);
+#endif /* __BIONIC_AVAILABILITY_GUARD(33) */
+
 
 /**
  * [process_vm_readv(2)](https://man7.org/linux/man-pages/man2/process_vm_readv.2.html)
@@ -144,6 +152,8 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 ssize_t process_vm_readv(pid_t __pid, const struct iovec* __BIONIC_COMPLICATED_NULLNESS __local_iov, unsigned long __local_iov_count, const struct iovec* __BIONIC_COMPLICATED_NULLNESS __remote_iov, unsigned long __remote_iov_count, unsigned long __flags) __INTRODUCED_IN(23);
 
 /**
@@ -156,6 +166,8 @@
  * Available since API level 23.
  */
 ssize_t process_vm_writev(pid_t __pid, const struct iovec* __BIONIC_COMPLICATED_NULLNESS __local_iov, unsigned long __local_iov_count, const struct iovec* __BIONIC_COMPLICATED_NULLNESS __remote_iov, unsigned long __remote_iov_count, unsigned long __flags) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 #endif
 
diff --git a/libc/include/sys/un.h b/libc/include/sys/un.h
index 83c1d17..c2bfcb0 100644
--- a/libc/include/sys/un.h
+++ b/libc/include/sys/un.h
@@ -33,9 +33,10 @@
  * @brief Unix domain sockets.
  */
 
+#include <sys/cdefs.h>
+
 #include <bits/sa_family_t.h>
 #include <linux/un.h>
-#include <sys/cdefs.h>
 
 #if defined(__USE_BSD) || defined(__USE_GNU)
 #include <string.h>
diff --git a/libc/include/sys/vfs.h b/libc/include/sys/vfs.h
index 1a640ba..5d078be 100644
--- a/libc/include/sys/vfs.h
+++ b/libc/include/sys/vfs.h
@@ -29,8 +29,9 @@
 #ifndef _SYS_VFS_H_
 #define _SYS_VFS_H_
 
-#include <stdint.h>
 #include <sys/cdefs.h>
+
+#include <stdint.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/sys/wait.h b/libc/include/sys/wait.h
index 5208366..632aa43 100644
--- a/libc/include/sys/wait.h
+++ b/libc/include/sys/wait.h
@@ -28,8 +28,9 @@
 
 #pragma once
 
-#include <bits/wait.h>
 #include <sys/cdefs.h>
+
+#include <bits/wait.h>
 #include <sys/types.h>
 #include <sys/resource.h>
 #include <linux/wait.h>
diff --git a/libc/include/sys/xattr.h b/libc/include/sys/xattr.h
index 38c11e2..ebe4eb8 100644
--- a/libc/include/sys/xattr.h
+++ b/libc/include/sys/xattr.h
@@ -33,8 +33,9 @@
  * @brief Extended attribute functions.
  */
 
-#include <linux/xattr.h>
 #include <sys/cdefs.h>
+
+#include <linux/xattr.h>
 #include <sys/types.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/syslog.h b/libc/include/syslog.h
index 33979f0..7a594f1 100644
--- a/libc/include/syslog.h
+++ b/libc/include/syslog.h
@@ -56,8 +56,9 @@
 
 #pragma once
 
-#include <stdio.h>
 #include <sys/cdefs.h>
+
+#include <stdio.h>
 #include <stdarg.h>
 
 __BEGIN_DECLS
diff --git a/libc/include/time.h b/libc/include/time.h
index e9d6569..6c9b761 100644
--- a/libc/include/time.h
+++ b/libc/include/time.h
@@ -166,7 +166,11 @@
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 time_t mktime_z(timezone_t _Nonnull __tz, struct tm* _Nonnull __tm) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 /**
  * [localtime(3)](https://man7.org/linux/man-pages/man3/localtime.3p.html) converts
@@ -200,7 +204,11 @@
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 struct tm* _Nullable localtime_rz(timezone_t _Nonnull __tz, const time_t* _Nonnull __t, struct tm* _Nonnull __tm) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 /**
  * Inverse of localtime().
@@ -314,6 +322,8 @@
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 timezone_t _Nullable tzalloc(const char* _Nullable __id) __INTRODUCED_IN(35);
 
 /**
@@ -326,6 +336,8 @@
  * Available since API level 35.
  */
 void tzfree(timezone_t _Nullable __tz) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 /**
  * [clock(3)](https://man7.org/linux/man-pages/man3/clock.3.html)
@@ -345,7 +357,11 @@
  *
  * Returns 0 on success, and returns -1 and returns an error number on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int clock_getcpuclockid(pid_t __pid, clockid_t* _Nonnull __clock) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 /**
  * [clock_getres(2)](https://man7.org/linux/man-pages/man2/clock_getres.2.html)
@@ -459,7 +475,11 @@
  * Available since API level 29 for TIME_UTC; other bases arrived later.
  * Code for Android should prefer clock_gettime().
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(29)
 int timespec_get(struct timespec* _Nonnull __ts, int __base) __INTRODUCED_IN(29);
+#endif /* __BIONIC_AVAILABILITY_GUARD(29) */
+
 
 /**
  * timespec_getres(3) is equivalent to clock_getres() for the clock corresponding to the given base.
@@ -469,6 +489,10 @@
  * Available since API level 35.
  * Code for Android should prefer clock_gettime().
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 int timespec_getres(struct timespec* _Nonnull __ts, int __base) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 __END_DECLS
diff --git a/libc/include/uchar.h b/libc/include/uchar.h
index 55a36e7..94efb2d 100644
--- a/libc/include/uchar.h
+++ b/libc/include/uchar.h
@@ -33,9 +33,10 @@
  * @brief Unicode functions.
  */
 
-#include <stddef.h>
 #include <sys/cdefs.h>
 
+#include <stddef.h>
+
 #include <bits/bionic_multibyte_result.h>
 #include <bits/mbstate_t.h>
 
diff --git a/libc/include/unistd.h b/libc/include/unistd.h
index e1c268f..e623339 100644
--- a/libc/include/unistd.h
+++ b/libc/include/unistd.h
@@ -28,8 +28,9 @@
 
 #pragma once
 
-#include <stddef.h>
 #include <sys/cdefs.h>
+
+#include <stddef.h>
 #include <sys/types.h>
 #include <sys/select.h>
 
@@ -100,7 +101,11 @@
  *
  * Available since API level 35.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(35)
 pid_t _Fork(void) __INTRODUCED_IN(35);
+#endif /* __BIONIC_AVAILABILITY_GUARD(35) */
+
 
 /**
  * [vfork(2)](https://man7.org/linux/man-pages/man2/vfork.2.html) creates a new
@@ -145,7 +150,11 @@
 int execlp(const char* _Nonnull __file, const char* _Nullable __arg0, ...) __attribute__((__sentinel__));
 int execle(const char* _Nonnull __path, const char* _Nullable __arg0, ... /*,  char* const* __envp */)
     __attribute__((__sentinel__(1)));
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int fexecve(int __fd, char* _Nullable const* _Nullable __argv, char* _Nullable const* _Nullable __envp) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 int nice(int __incr);
 
@@ -246,7 +255,11 @@
 int getresuid(uid_t* _Nonnull __ruid, uid_t* _Nonnull __euid, uid_t* _Nonnull __suid);
 int getresgid(gid_t* _Nonnull __rgid, gid_t* _Nonnull __egid, gid_t* _Nonnull __sgid);
 char* _Nullable getlogin(void);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int getlogin_r(char* _Nonnull __buffer, size_t __buffer_size) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 
 long fpathconf(int __fd, int __name);
 long pathconf(const char* _Nonnull __path, int __name);
@@ -257,8 +270,29 @@
 int linkat(int __old_dir_fd, const char* _Nonnull __old_path, int __new_dir_fd, const char* _Nonnull __new_path, int __flags);
 int unlink(const char* _Nonnull __path);
 int unlinkat(int __dirfd, const char* _Nonnull __path, int __flags);
+
+/**
+ * [chdir(2)](https://man7.org/linux/man-pages/man2/chdir.2.html) changes
+ * the current working directory to the given path.
+ *
+ * This function affects all threads in the process, so is generally a bad idea
+ * on Android where most code will be running in a multi-threaded context.
+ *
+ * Returns 0 on success, and returns -1 and sets `errno` on failure.
+ */
 int chdir(const char* _Nonnull __path);
+
+/**
+ * [fchdir(2)](https://man7.org/linux/man-pages/man2/chdir.2.html) changes
+ * the current working directory to the given fd.
+ *
+ * This function affects all threads in the process, so is generally a bad idea
+ * on Android where most code will be running in a multi-threaded context.
+ *
+ * Returns 0 on success, and returns -1 and sets `errno` on failure.
+ */
 int fchdir(int __fd);
+
 int rmdir(const char* _Nonnull __path);
 int pipe(int __fds[_Nonnull 2]);
 #if defined(__USE_GNU)
@@ -277,7 +311,11 @@
 
 void sync(void);
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 int syncfs(int __fd) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 #endif
 
 int close(int __fd);
@@ -339,7 +377,11 @@
 int usleep(useconds_t __microseconds);
 
 int gethostname(char* _Nonnull _buf, size_t __buf_size);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int sethostname(const char* _Nonnull __name, size_t __n) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 int brk(void* _Nonnull __addr);
 void* _Nullable sbrk(ptrdiff_t __increment);
@@ -382,8 +424,12 @@
     } while (_rc == -1 && errno == EINTR); \
     _rc; })
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 int getdomainname(char* _Nonnull __buf, size_t __buf_size) __INTRODUCED_IN(26);
 int setdomainname(const char* _Nonnull __name, size_t __n) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 /**
  * [copy_file_range(2)](https://man7.org/linux/man-pages/man2/copy_file_range.2.html) copies
@@ -394,7 +440,11 @@
  * Returns the number of bytes copied on success, and returns -1 and sets
  * `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(34)
 ssize_t copy_file_range(int __fd_in, off64_t* _Nullable __off_in, int __fd_out, off64_t* _Nullable __off_out, size_t __length, unsigned int __flags) __INTRODUCED_IN(34);
+#endif /* __BIONIC_AVAILABILITY_GUARD(34) */
+
 
 #if __ANDROID_API__ >= 28
 void swab(const void* _Nonnull __src, void* _Nonnull __dst, ssize_t __byte_count) __INTRODUCED_IN(28);
@@ -414,7 +464,11 @@
  *
  * Returns 0 on success, and returns -1 and sets `errno` on failure.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(34)
 int close_range(unsigned int __min_fd, unsigned int __max_fd, int __flags) __INTRODUCED_IN(34);
+#endif /* __BIONIC_AVAILABILITY_GUARD(34) */
+
 
 #if defined(__BIONIC_INCLUDE_FORTIFY_HEADERS)
 #define _UNISTD_H_
diff --git a/libc/include/utmp.h b/libc/include/utmp.h
index d249f8a..1674491 100644
--- a/libc/include/utmp.h
+++ b/libc/include/utmp.h
@@ -131,6 +131,10 @@
  *
  * Available since API level 23.
  */
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int login_tty(int __fd) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 
 __END_DECLS
diff --git a/libc/include/wchar.h b/libc/include/wchar.h
index e86f94d..56594dc 100644
--- a/libc/include/wchar.h
+++ b/libc/include/wchar.h
@@ -75,7 +75,11 @@
 wchar_t* _Nonnull wcpncpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 size_t wcrtomb(char* _Nullable __buf, wchar_t __wc, mbstate_t* _Nullable __ps);
 int wcscasecmp(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int wcscasecmp_l(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs, locale_t _Nonnull __l) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 wchar_t* _Nonnull wcscat(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src);
 wchar_t* _Nullable wcschr(const wchar_t * _Nonnull __s, wchar_t __wc);
 int wcscmp(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs);
@@ -83,10 +87,18 @@
 wchar_t* _Nonnull wcscpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src);
 size_t wcscspn(const wchar_t* _Nonnull __s, const wchar_t* _Nonnull __accept);
 size_t wcsftime(wchar_t* _Nonnull __buf, size_t __n, const wchar_t* _Nullable __fmt, const struct tm* _Nonnull __tm);
+
+#if __BIONIC_AVAILABILITY_GUARD(28)
 size_t wcsftime_l(wchar_t* _Nonnull __buf, size_t __n, const wchar_t* _Nullable __fmt, const struct tm* _Nonnull __tm, locale_t _Nonnull __l) __INTRODUCED_IN(28);
+#endif /* __BIONIC_AVAILABILITY_GUARD(28) */
+
 size_t wcslen(const wchar_t* _Nonnull __s);
 int wcsncasecmp(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs, size_t __n);
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 int wcsncasecmp_l(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs, size_t __n, locale_t _Nonnull __l) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 wchar_t* _Nonnull wcsncat(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 int wcsncmp(const wchar_t* _Nonnull __lhs, const wchar_t* _Nonnull __rhs, size_t __n);
 wchar_t* _Nonnull wcsncpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
@@ -117,7 +129,11 @@
 int wmemcmp(const wchar_t* _Nullable __lhs, const wchar_t* _Nullable __rhs, size_t __n);
 wchar_t* _Nonnull wmemcpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 #if defined(__USE_GNU)
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 wchar_t* _Nonnull wmempcpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 #endif
 wchar_t* _Nonnull wmemmove(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 wchar_t* _Nonnull wmemset(wchar_t* _Nonnull __dst, wchar_t __wc, size_t __n);
@@ -133,7 +149,11 @@
 size_t wcslcat(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 size_t wcslcpy(wchar_t* _Nonnull __dst, const wchar_t* _Nonnull __src, size_t __n);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(23)
 FILE* _Nullable open_wmemstream(wchar_t* _Nonnull * _Nonnull __ptr, size_t* _Nonnull  __size_ptr) __INTRODUCED_IN(23);
+#endif /* __BIONIC_AVAILABILITY_GUARD(23) */
+
 wchar_t* _Nullable wcsdup(const wchar_t* _Nonnull __s);
 size_t wcsnlen(const wchar_t* _Nonnull __s, size_t __n);
 
diff --git a/libc/include/wctype.h b/libc/include/wctype.h
index 4f6f81f..30ec04f 100644
--- a/libc/include/wctype.h
+++ b/libc/include/wctype.h
@@ -29,8 +29,9 @@
 #ifndef _WCTYPE_H_
 #define _WCTYPE_H_
 
-#include <bits/wctype.h>
 #include <sys/cdefs.h>
+
+#include <bits/wctype.h>
 #include <xlocale.h>
 
 __BEGIN_DECLS
@@ -51,8 +52,12 @@
 wint_t towlower_l(wint_t __wc, locale_t _Nonnull __l);
 wint_t towupper_l(wint_t __wc, locale_t _Nonnull __l);
 
+
+#if __BIONIC_AVAILABILITY_GUARD(26)
 wint_t towctrans_l(wint_t __wc, wctrans_t _Nonnull __transform, locale_t _Nonnull __l) __INTRODUCED_IN(26);
 wctrans_t _Nonnull wctrans_l(const char* _Nonnull __name, locale_t _Nonnull __l) __INTRODUCED_IN(26);
+#endif /* __BIONIC_AVAILABILITY_GUARD(26) */
+
 
 wctype_t wctype_l(const char* _Nonnull __name, locale_t _Nonnull __l);
 int iswctype_l(wint_t __wc, wctype_t __transform, locale_t _Nonnull __l);
diff --git a/libc/kernel/android/scsi/scsi/scsi_proto.h b/libc/kernel/android/scsi/scsi/scsi_proto.h
index d873fad..754e12a 100644
--- a/libc/kernel/android/scsi/scsi/scsi_proto.h
+++ b/libc/kernel/android/scsi/scsi/scsi_proto.h
@@ -109,6 +109,7 @@
 #define WRITE_SAME_16 0x93
 #define ZBC_OUT 0x94
 #define ZBC_IN 0x95
+#define WRITE_ATOMIC_16 0x9c
 #define SERVICE_ACTION_BIDIRECTIONAL 0x9d
 #define SERVICE_ACTION_IN_16 0x9e
 #define SERVICE_ACTION_OUT_16 0x9f
diff --git a/libc/kernel/tools/cpp.py b/libc/kernel/tools/cpp.py
index 08b786a..df50806 100755
--- a/libc/kernel/tools/cpp.py
+++ b/libc/kernel/tools/cpp.py
@@ -1430,7 +1430,7 @@
                         state = VAR_DECL
                     elif state == NORMAL and token_id in ['struct', 'typedef',
                                                           'enum', 'union',
-                                                          '__extension__']:
+                                                          '__extension__', '=']:
                         state = OTHER_DECL
                         state_token = token_id
                     elif block.tokens[i].kind == TokenKind.IDENTIFIER:
@@ -2057,7 +2057,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_struct_from_end(self):
         text = """\
@@ -2076,7 +2076,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_minimal_struct(self):
         text = """\
@@ -2084,7 +2084,7 @@
 };
 """
         expected = "";
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_struct_with_struct_fields(self):
         text = """\
@@ -2104,7 +2104,7 @@
   struct remove val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_consecutive_structs(self):
         text = """\
@@ -2136,7 +2136,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove1": True, "remove2": True}), expected)
+        self.assertEqual(self.parse(text, {"remove1": None, "remove2": None}), expected)
 
     def test_remove_multiple_structs(self):
         text = """\
@@ -2169,7 +2169,7 @@
   int val;
 };
 """
-        self.assertEqual(self.parse(text, {"remove1": True, "remove2": True}), expected)
+        self.assertEqual(self.parse(text, {"remove1": None, "remove2": None}), expected)
 
     def test_remove_struct_with_inline_structs(self):
         text = """\
@@ -2194,7 +2194,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_struct_across_blocks(self):
         text = """\
@@ -2219,7 +2219,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove": True}), expected)
+        self.assertEqual(self.parse(text, {"remove": None}), expected)
 
     def test_remove_struct_across_blocks_multiple_structs(self):
         text = """\
@@ -2246,7 +2246,7 @@
   struct timeval val2;
 };
 """
-        self.assertEqual(self.parse(text, {"remove1": True, "remove2": True}), expected)
+        self.assertEqual(self.parse(text, {"remove1": None, "remove2": None}), expected)
 
     def test_remove_multiple_struct_and_add_includes(self):
         text = """\
@@ -2263,7 +2263,7 @@
 #include <bits/remove1.h>
 #include <bits/remove2.h>
 """
-        self.assertEqual(self.parse(text, {"remove1": False, "remove2": False}), expected)
+        self.assertEqual(self.parse(text, {"remove1": "bits/remove1.h", "remove2": "bits/remove2.h"}), expected)
 
 
 class FullPathTest(unittest.TestCase):
@@ -2580,6 +2580,71 @@
   struct timeval timeval;
   struct itimerval itimerval;
 };
+#include <linux/time.h>
+"""
+        self.assertEqual(self.parse(text), expected)
+
+    def test_var_definition(self):
+        # If we're definining the whole thing, it's probably worth keeping.
+        text = """\
+static const char *kString = "hello world";
+static const int kInteger = 42;
+"""
+        expected = """\
+static const char * kString = "hello world";
+static const int kInteger = 42;
+"""
+        self.assertEqual(self.parse(text), expected)
+
+    def test_struct_array_definition(self):
+        text = """\
+struct descriptor {
+  int args;
+  int size;
+};
+static const struct descriptor[] = {
+  {0, 0},
+  {1, 12},
+  {0, 42},
+};
+"""
+        expected = """\
+struct descriptor {
+  int args;
+  int size;
+};
+static const struct descriptor[] = {
+ {
+    0, 0
+  }
+ , {
+    1, 12
+  }
+ , {
+    0, 42
+  }
+ ,
+};
+"""
+        self.assertEqual(self.parse(text), expected)
+
+    def test_array_definition(self):
+        text = """\
+static const char *arr[] = {
+  "foo",
+  "bar",
+  "baz",
+};
+
+static int another_arr[5] = { 1, 2, 3, 4, 5};
+"""
+        expected = """\
+static const char * arr[] = {
+  "foo", "bar", "baz",
+};
+static int another_arr[5] = {
+  1, 2, 3, 4, 5
+};
 """
         self.assertEqual(self.parse(text), expected)
 
diff --git a/libc/kernel/tools/defaults.py b/libc/kernel/tools/defaults.py
index 06afb25..2994e5e 100644
--- a/libc/kernel/tools/defaults.py
+++ b/libc/kernel/tools/defaults.py
@@ -25,6 +25,23 @@
     # Otherwise, there will be two struct timeval definitions when
     # __kernel_old_timeval is renamed to timeval.
     "__kernel_old_timeval": "1",
+    # Drop the custom byte swap functions and just use the clang builtins.
+    # https://github.com/android/ndk/issues/2107
+    "__arch_swab16": kCppUndefinedMacro,
+    "__arch_swab16p": kCppUndefinedMacro,
+    "__arch_swab16s": kCppUndefinedMacro,
+    "__arch_swab32": kCppUndefinedMacro,
+    "__arch_swab32p": kCppUndefinedMacro,
+    "__arch_swab32s": kCppUndefinedMacro,
+    "__arch_swab64": kCppUndefinedMacro,
+    "__arch_swab64p": kCppUndefinedMacro,
+    "__arch_swab64s": kCppUndefinedMacro,
+    "__arch_swahb32": kCppUndefinedMacro,
+    "__arch_swahb32p": kCppUndefinedMacro,
+    "__arch_swahb32s": kCppUndefinedMacro,
+    "__arch_swahw32": kCppUndefinedMacro,
+    "__arch_swahw32p": kCppUndefinedMacro,
+    "__arch_swahw32s": kCppUndefinedMacro,
     }
 
 # This is the set of known kernel data structures we want to remove from
diff --git a/libc/kernel/tools/update_all.py b/libc/kernel/tools/update_all.py
index ae89a80..331a957 100755
--- a/libc/kernel/tools/update_all.py
+++ b/libc/kernel/tools/update_all.py
@@ -88,15 +88,8 @@
     # Collect the set of all syscalls for all architectures.
     syscalls = set()
     pattern = re.compile(r'^\s*#\s*define\s*__NR_([a-z_]\S+)')
-    for unistd_h in ['kernel/uapi/asm-generic/unistd.h',
-                     'kernel/uapi/asm-arm/asm/unistd.h',
-                     'kernel/uapi/asm-arm/asm/unistd-eabi.h',
-                     'kernel/uapi/asm-arm/asm/unistd-oabi.h',
-                     'kernel/uapi/asm-riscv/asm/unistd.h',
-                     'kernel/uapi/asm-x86/asm/unistd_32.h',
-                     'kernel/uapi/asm-x86/asm/unistd_64.h',
-                     'kernel/uapi/asm-x86/asm/unistd_x32.h']:
-        for line in open(os.path.join(libc_root, unistd_h)):
+    for unistd_h in glob.glob('%s/kernel/uapi/asm-*/asm/unistd*.h' % libc_root):
+        for line in open(unistd_h):
             m = re.search(pattern, line)
             if m:
                 nr_name = m.group(1)
diff --git a/libc/kernel/uapi/asm-arm64/asm/unistd.h b/libc/kernel/uapi/asm-arm64/asm/unistd.h
index 7457ebc..178578f 100644
--- a/libc/kernel/uapi/asm-arm64/asm/unistd.h
+++ b/libc/kernel/uapi/asm-arm64/asm/unistd.h
@@ -4,10 +4,4 @@
  * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
  * for more information.
  */
-#define __ARCH_WANT_RENAMEAT
-#define __ARCH_WANT_NEW_STAT
-#define __ARCH_WANT_SET_GET_RLIMIT
-#define __ARCH_WANT_TIME32_SYSCALLS
-#define __ARCH_WANT_SYS_CLONE3
-#define __ARCH_WANT_MEMFD_SECRET
-#include <asm-generic/unistd.h>
+#include <asm/unistd_64.h>
diff --git a/libc/kernel/uapi/asm-arm64/asm/unistd_64.h b/libc/kernel/uapi/asm-arm64/asm/unistd_64.h
new file mode 100644
index 0000000..0a0a1c0
--- /dev/null
+++ b/libc/kernel/uapi/asm-arm64/asm/unistd_64.h
@@ -0,0 +1,327 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef _UAPI_ASM_UNISTD_64_H
+#define _UAPI_ASM_UNISTD_64_H
+#define __NR_io_setup 0
+#define __NR_io_destroy 1
+#define __NR_io_submit 2
+#define __NR_io_cancel 3
+#define __NR_io_getevents 4
+#define __NR_setxattr 5
+#define __NR_lsetxattr 6
+#define __NR_fsetxattr 7
+#define __NR_getxattr 8
+#define __NR_lgetxattr 9
+#define __NR_fgetxattr 10
+#define __NR_listxattr 11
+#define __NR_llistxattr 12
+#define __NR_flistxattr 13
+#define __NR_removexattr 14
+#define __NR_lremovexattr 15
+#define __NR_fremovexattr 16
+#define __NR_getcwd 17
+#define __NR_lookup_dcookie 18
+#define __NR_eventfd2 19
+#define __NR_epoll_create1 20
+#define __NR_epoll_ctl 21
+#define __NR_epoll_pwait 22
+#define __NR_dup 23
+#define __NR_dup3 24
+#define __NR_fcntl 25
+#define __NR_inotify_init1 26
+#define __NR_inotify_add_watch 27
+#define __NR_inotify_rm_watch 28
+#define __NR_ioctl 29
+#define __NR_ioprio_set 30
+#define __NR_ioprio_get 31
+#define __NR_flock 32
+#define __NR_mknodat 33
+#define __NR_mkdirat 34
+#define __NR_unlinkat 35
+#define __NR_symlinkat 36
+#define __NR_linkat 37
+#define __NR_renameat 38
+#define __NR_umount2 39
+#define __NR_mount 40
+#define __NR_pivot_root 41
+#define __NR_nfsservctl 42
+#define __NR_statfs 43
+#define __NR_fstatfs 44
+#define __NR_truncate 45
+#define __NR_ftruncate 46
+#define __NR_fallocate 47
+#define __NR_faccessat 48
+#define __NR_chdir 49
+#define __NR_fchdir 50
+#define __NR_chroot 51
+#define __NR_fchmod 52
+#define __NR_fchmodat 53
+#define __NR_fchownat 54
+#define __NR_fchown 55
+#define __NR_openat 56
+#define __NR_close 57
+#define __NR_vhangup 58
+#define __NR_pipe2 59
+#define __NR_quotactl 60
+#define __NR_getdents64 61
+#define __NR_lseek 62
+#define __NR_read 63
+#define __NR_write 64
+#define __NR_readv 65
+#define __NR_writev 66
+#define __NR_pread64 67
+#define __NR_pwrite64 68
+#define __NR_preadv 69
+#define __NR_pwritev 70
+#define __NR_sendfile 71
+#define __NR_pselect6 72
+#define __NR_ppoll 73
+#define __NR_signalfd4 74
+#define __NR_vmsplice 75
+#define __NR_splice 76
+#define __NR_tee 77
+#define __NR_readlinkat 78
+#define __NR_newfstatat 79
+#define __NR_fstat 80
+#define __NR_sync 81
+#define __NR_fsync 82
+#define __NR_fdatasync 83
+#define __NR_sync_file_range 84
+#define __NR_timerfd_create 85
+#define __NR_timerfd_settime 86
+#define __NR_timerfd_gettime 87
+#define __NR_utimensat 88
+#define __NR_acct 89
+#define __NR_capget 90
+#define __NR_capset 91
+#define __NR_personality 92
+#define __NR_exit 93
+#define __NR_exit_group 94
+#define __NR_waitid 95
+#define __NR_set_tid_address 96
+#define __NR_unshare 97
+#define __NR_futex 98
+#define __NR_set_robust_list 99
+#define __NR_get_robust_list 100
+#define __NR_nanosleep 101
+#define __NR_getitimer 102
+#define __NR_setitimer 103
+#define __NR_kexec_load 104
+#define __NR_init_module 105
+#define __NR_delete_module 106
+#define __NR_timer_create 107
+#define __NR_timer_gettime 108
+#define __NR_timer_getoverrun 109
+#define __NR_timer_settime 110
+#define __NR_timer_delete 111
+#define __NR_clock_settime 112
+#define __NR_clock_gettime 113
+#define __NR_clock_getres 114
+#define __NR_clock_nanosleep 115
+#define __NR_syslog 116
+#define __NR_ptrace 117
+#define __NR_sched_setparam 118
+#define __NR_sched_setscheduler 119
+#define __NR_sched_getscheduler 120
+#define __NR_sched_getparam 121
+#define __NR_sched_setaffinity 122
+#define __NR_sched_getaffinity 123
+#define __NR_sched_yield 124
+#define __NR_sched_get_priority_max 125
+#define __NR_sched_get_priority_min 126
+#define __NR_sched_rr_get_interval 127
+#define __NR_restart_syscall 128
+#define __NR_kill 129
+#define __NR_tkill 130
+#define __NR_tgkill 131
+#define __NR_sigaltstack 132
+#define __NR_rt_sigsuspend 133
+#define __NR_rt_sigaction 134
+#define __NR_rt_sigprocmask 135
+#define __NR_rt_sigpending 136
+#define __NR_rt_sigtimedwait 137
+#define __NR_rt_sigqueueinfo 138
+#define __NR_rt_sigreturn 139
+#define __NR_setpriority 140
+#define __NR_getpriority 141
+#define __NR_reboot 142
+#define __NR_setregid 143
+#define __NR_setgid 144
+#define __NR_setreuid 145
+#define __NR_setuid 146
+#define __NR_setresuid 147
+#define __NR_getresuid 148
+#define __NR_setresgid 149
+#define __NR_getresgid 150
+#define __NR_setfsuid 151
+#define __NR_setfsgid 152
+#define __NR_times 153
+#define __NR_setpgid 154
+#define __NR_getpgid 155
+#define __NR_getsid 156
+#define __NR_setsid 157
+#define __NR_getgroups 158
+#define __NR_setgroups 159
+#define __NR_uname 160
+#define __NR_sethostname 161
+#define __NR_setdomainname 162
+#define __NR_getrlimit 163
+#define __NR_setrlimit 164
+#define __NR_getrusage 165
+#define __NR_umask 166
+#define __NR_prctl 167
+#define __NR_getcpu 168
+#define __NR_gettimeofday 169
+#define __NR_settimeofday 170
+#define __NR_adjtimex 171
+#define __NR_getpid 172
+#define __NR_getppid 173
+#define __NR_getuid 174
+#define __NR_geteuid 175
+#define __NR_getgid 176
+#define __NR_getegid 177
+#define __NR_gettid 178
+#define __NR_sysinfo 179
+#define __NR_mq_open 180
+#define __NR_mq_unlink 181
+#define __NR_mq_timedsend 182
+#define __NR_mq_timedreceive 183
+#define __NR_mq_notify 184
+#define __NR_mq_getsetattr 185
+#define __NR_msgget 186
+#define __NR_msgctl 187
+#define __NR_msgrcv 188
+#define __NR_msgsnd 189
+#define __NR_semget 190
+#define __NR_semctl 191
+#define __NR_semtimedop 192
+#define __NR_semop 193
+#define __NR_shmget 194
+#define __NR_shmctl 195
+#define __NR_shmat 196
+#define __NR_shmdt 197
+#define __NR_socket 198
+#define __NR_socketpair 199
+#define __NR_bind 200
+#define __NR_listen 201
+#define __NR_accept 202
+#define __NR_connect 203
+#define __NR_getsockname 204
+#define __NR_getpeername 205
+#define __NR_sendto 206
+#define __NR_recvfrom 207
+#define __NR_setsockopt 208
+#define __NR_getsockopt 209
+#define __NR_shutdown 210
+#define __NR_sendmsg 211
+#define __NR_recvmsg 212
+#define __NR_readahead 213
+#define __NR_brk 214
+#define __NR_munmap 215
+#define __NR_mremap 216
+#define __NR_add_key 217
+#define __NR_request_key 218
+#define __NR_keyctl 219
+#define __NR_clone 220
+#define __NR_execve 221
+#define __NR_mmap 222
+#define __NR_fadvise64 223
+#define __NR_swapon 224
+#define __NR_swapoff 225
+#define __NR_mprotect 226
+#define __NR_msync 227
+#define __NR_mlock 228
+#define __NR_munlock 229
+#define __NR_mlockall 230
+#define __NR_munlockall 231
+#define __NR_mincore 232
+#define __NR_madvise 233
+#define __NR_remap_file_pages 234
+#define __NR_mbind 235
+#define __NR_get_mempolicy 236
+#define __NR_set_mempolicy 237
+#define __NR_migrate_pages 238
+#define __NR_move_pages 239
+#define __NR_rt_tgsigqueueinfo 240
+#define __NR_perf_event_open 241
+#define __NR_accept4 242
+#define __NR_recvmmsg 243
+#define __NR_wait4 260
+#define __NR_prlimit64 261
+#define __NR_fanotify_init 262
+#define __NR_fanotify_mark 263
+#define __NR_name_to_handle_at 264
+#define __NR_open_by_handle_at 265
+#define __NR_clock_adjtime 266
+#define __NR_syncfs 267
+#define __NR_setns 268
+#define __NR_sendmmsg 269
+#define __NR_process_vm_readv 270
+#define __NR_process_vm_writev 271
+#define __NR_kcmp 272
+#define __NR_finit_module 273
+#define __NR_sched_setattr 274
+#define __NR_sched_getattr 275
+#define __NR_renameat2 276
+#define __NR_seccomp 277
+#define __NR_getrandom 278
+#define __NR_memfd_create 279
+#define __NR_bpf 280
+#define __NR_execveat 281
+#define __NR_userfaultfd 282
+#define __NR_membarrier 283
+#define __NR_mlock2 284
+#define __NR_copy_file_range 285
+#define __NR_preadv2 286
+#define __NR_pwritev2 287
+#define __NR_pkey_mprotect 288
+#define __NR_pkey_alloc 289
+#define __NR_pkey_free 290
+#define __NR_statx 291
+#define __NR_io_pgetevents 292
+#define __NR_rseq 293
+#define __NR_kexec_file_load 294
+#define __NR_pidfd_send_signal 424
+#define __NR_io_uring_setup 425
+#define __NR_io_uring_enter 426
+#define __NR_io_uring_register 427
+#define __NR_open_tree 428
+#define __NR_move_mount 429
+#define __NR_fsopen 430
+#define __NR_fsconfig 431
+#define __NR_fsmount 432
+#define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
+#define __NR_close_range 436
+#define __NR_openat2 437
+#define __NR_pidfd_getfd 438
+#define __NR_faccessat2 439
+#define __NR_process_madvise 440
+#define __NR_epoll_pwait2 441
+#define __NR_mount_setattr 442
+#define __NR_quotactl_fd 443
+#define __NR_landlock_create_ruleset 444
+#define __NR_landlock_add_rule 445
+#define __NR_landlock_restrict_self 446
+#define __NR_memfd_secret 447
+#define __NR_process_mrelease 448
+#define __NR_futex_waitv 449
+#define __NR_set_mempolicy_home_node 450
+#define __NR_cachestat 451
+#define __NR_fchmodat2 452
+#define __NR_map_shadow_stack 453
+#define __NR_futex_wake 454
+#define __NR_futex_wait 455
+#define __NR_futex_requeue 456
+#define __NR_statmount 457
+#define __NR_listmount 458
+#define __NR_lsm_get_self_attr 459
+#define __NR_lsm_set_self_attr 460
+#define __NR_lsm_list_modules 461
+#define __NR_mseal 462
+#endif
diff --git a/libc/kernel/uapi/asm-generic/unistd.h b/libc/kernel/uapi/asm-generic/unistd.h
index 7eaa89a..652e7a2 100644
--- a/libc/kernel/uapi/asm-generic/unistd.h
+++ b/libc/kernel/uapi/asm-generic/unistd.h
@@ -381,9 +381,7 @@
 #define __NR_fsmount 432
 #define __NR_fspick 433
 #define __NR_pidfd_open 434
-#ifdef __ARCH_WANT_SYS_CLONE3
 #define __NR_clone3 435
-#endif
 #define __NR_close_range 436
 #define __NR_openat2 437
 #define __NR_pidfd_getfd 438
diff --git a/libc/kernel/uapi/asm-riscv/asm/hwprobe.h b/libc/kernel/uapi/asm-riscv/asm/hwprobe.h
index 3f30c88..2e5f9a4 100644
--- a/libc/kernel/uapi/asm-riscv/asm/hwprobe.h
+++ b/libc/kernel/uapi/asm-riscv/asm/hwprobe.h
@@ -54,6 +54,18 @@
 #define RISCV_HWPROBE_EXT_ZACAS (1ULL << 34)
 #define RISCV_HWPROBE_EXT_ZICOND (1ULL << 35)
 #define RISCV_HWPROBE_EXT_ZIHINTPAUSE (1ULL << 36)
+#define RISCV_HWPROBE_EXT_ZVE32X (1ULL << 37)
+#define RISCV_HWPROBE_EXT_ZVE32F (1ULL << 38)
+#define RISCV_HWPROBE_EXT_ZVE64X (1ULL << 39)
+#define RISCV_HWPROBE_EXT_ZVE64F (1ULL << 40)
+#define RISCV_HWPROBE_EXT_ZVE64D (1ULL << 41)
+#define RISCV_HWPROBE_EXT_ZIMOP (1ULL << 42)
+#define RISCV_HWPROBE_EXT_ZCA (1ULL << 43)
+#define RISCV_HWPROBE_EXT_ZCB (1ULL << 44)
+#define RISCV_HWPROBE_EXT_ZCD (1ULL << 45)
+#define RISCV_HWPROBE_EXT_ZCF (1ULL << 46)
+#define RISCV_HWPROBE_EXT_ZCMOP (1ULL << 47)
+#define RISCV_HWPROBE_EXT_ZAWRS (1ULL << 48)
 #define RISCV_HWPROBE_KEY_CPUPERF_0 5
 #define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)
 #define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0)
@@ -62,5 +74,13 @@
 #define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0)
 #define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0)
 #define RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE 6
+#define RISCV_HWPROBE_KEY_HIGHEST_VIRT_ADDRESS 7
+#define RISCV_HWPROBE_KEY_TIME_CSR_FREQ 8
+#define RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF 9
+#define RISCV_HWPROBE_MISALIGNED_SCALAR_UNKNOWN 0
+#define RISCV_HWPROBE_MISALIGNED_SCALAR_EMULATED 1
+#define RISCV_HWPROBE_MISALIGNED_SCALAR_SLOW 2
+#define RISCV_HWPROBE_MISALIGNED_SCALAR_FAST 3
+#define RISCV_HWPROBE_MISALIGNED_SCALAR_UNSUPPORTED 4
 #define RISCV_HWPROBE_WHICH_CPUS (1 << 0)
 #endif
diff --git a/libc/kernel/uapi/asm-riscv/asm/kvm.h b/libc/kernel/uapi/asm-riscv/asm/kvm.h
index 12d8f61..51f4977 100644
--- a/libc/kernel/uapi/asm-riscv/asm/kvm.h
+++ b/libc/kernel/uapi/asm-riscv/asm/kvm.h
@@ -128,6 +128,13 @@
   KVM_RISCV_ISA_EXT_ZTSO,
   KVM_RISCV_ISA_EXT_ZACAS,
   KVM_RISCV_ISA_EXT_SSCOFPMF,
+  KVM_RISCV_ISA_EXT_ZIMOP,
+  KVM_RISCV_ISA_EXT_ZCA,
+  KVM_RISCV_ISA_EXT_ZCB,
+  KVM_RISCV_ISA_EXT_ZCD,
+  KVM_RISCV_ISA_EXT_ZCF,
+  KVM_RISCV_ISA_EXT_ZCMOP,
+  KVM_RISCV_ISA_EXT_ZAWRS,
   KVM_RISCV_ISA_EXT_MAX,
 };
 enum KVM_RISCV_SBI_EXT_ID {
diff --git a/libc/kernel/uapi/asm-riscv/asm/unistd.h b/libc/kernel/uapi/asm-riscv/asm/unistd.h
index 0c58887..f395f71 100644
--- a/libc/kernel/uapi/asm-riscv/asm/unistd.h
+++ b/libc/kernel/uapi/asm-riscv/asm/unistd.h
@@ -4,18 +4,9 @@
  * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
  * for more information.
  */
-#if defined(__LP64__) && !defined(__SYSCALL_COMPAT)
-#define __ARCH_WANT_NEW_STAT
-#define __ARCH_WANT_SET_GET_RLIMIT
+#include <asm/bitsperlong.h>
+#if __BITS_PER_LONG == 64
+#include <asm/unistd_64.h>
+#else
+#include <asm/unistd_32.h>
 #endif
-#define __ARCH_WANT_SYS_CLONE3
-#define __ARCH_WANT_MEMFD_SECRET
-#include <asm-generic/unistd.h>
-#ifndef __NR_riscv_flush_icache
-#define __NR_riscv_flush_icache (__NR_arch_specific_syscall + 15)
-#endif
-__SYSCALL(__NR_riscv_flush_icache, sys_riscv_flush_icache)
-#ifndef __NR_riscv_hwprobe
-#define __NR_riscv_hwprobe (__NR_arch_specific_syscall + 14)
-#endif
-__SYSCALL(__NR_riscv_hwprobe, sys_riscv_hwprobe)
diff --git a/libc/kernel/uapi/asm-riscv/asm/unistd_32.h b/libc/kernel/uapi/asm-riscv/asm/unistd_32.h
new file mode 100644
index 0000000..864a556
--- /dev/null
+++ b/libc/kernel/uapi/asm-riscv/asm/unistd_32.h
@@ -0,0 +1,318 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef _UAPI_ASM_UNISTD_32_H
+#define _UAPI_ASM_UNISTD_32_H
+#define __NR_io_setup 0
+#define __NR_io_destroy 1
+#define __NR_io_submit 2
+#define __NR_io_cancel 3
+#define __NR_setxattr 5
+#define __NR_lsetxattr 6
+#define __NR_fsetxattr 7
+#define __NR_getxattr 8
+#define __NR_lgetxattr 9
+#define __NR_fgetxattr 10
+#define __NR_listxattr 11
+#define __NR_llistxattr 12
+#define __NR_flistxattr 13
+#define __NR_removexattr 14
+#define __NR_lremovexattr 15
+#define __NR_fremovexattr 16
+#define __NR_getcwd 17
+#define __NR_lookup_dcookie 18
+#define __NR_eventfd2 19
+#define __NR_epoll_create1 20
+#define __NR_epoll_ctl 21
+#define __NR_epoll_pwait 22
+#define __NR_dup 23
+#define __NR_dup3 24
+#define __NR_fcntl64 25
+#define __NR_inotify_init1 26
+#define __NR_inotify_add_watch 27
+#define __NR_inotify_rm_watch 28
+#define __NR_ioctl 29
+#define __NR_ioprio_set 30
+#define __NR_ioprio_get 31
+#define __NR_flock 32
+#define __NR_mknodat 33
+#define __NR_mkdirat 34
+#define __NR_unlinkat 35
+#define __NR_symlinkat 36
+#define __NR_linkat 37
+#define __NR_umount2 39
+#define __NR_mount 40
+#define __NR_pivot_root 41
+#define __NR_nfsservctl 42
+#define __NR_statfs64 43
+#define __NR_fstatfs64 44
+#define __NR_truncate64 45
+#define __NR_ftruncate64 46
+#define __NR_fallocate 47
+#define __NR_faccessat 48
+#define __NR_chdir 49
+#define __NR_fchdir 50
+#define __NR_chroot 51
+#define __NR_fchmod 52
+#define __NR_fchmodat 53
+#define __NR_fchownat 54
+#define __NR_fchown 55
+#define __NR_openat 56
+#define __NR_close 57
+#define __NR_vhangup 58
+#define __NR_pipe2 59
+#define __NR_quotactl 60
+#define __NR_getdents64 61
+#define __NR_llseek 62
+#define __NR_read 63
+#define __NR_write 64
+#define __NR_readv 65
+#define __NR_writev 66
+#define __NR_pread64 67
+#define __NR_pwrite64 68
+#define __NR_preadv 69
+#define __NR_pwritev 70
+#define __NR_sendfile64 71
+#define __NR_signalfd4 74
+#define __NR_vmsplice 75
+#define __NR_splice 76
+#define __NR_tee 77
+#define __NR_readlinkat 78
+#define __NR_sync 81
+#define __NR_fsync 82
+#define __NR_fdatasync 83
+#define __NR_sync_file_range 84
+#define __NR_timerfd_create 85
+#define __NR_acct 89
+#define __NR_capget 90
+#define __NR_capset 91
+#define __NR_personality 92
+#define __NR_exit 93
+#define __NR_exit_group 94
+#define __NR_waitid 95
+#define __NR_set_tid_address 96
+#define __NR_unshare 97
+#define __NR_set_robust_list 99
+#define __NR_get_robust_list 100
+#define __NR_getitimer 102
+#define __NR_setitimer 103
+#define __NR_kexec_load 104
+#define __NR_init_module 105
+#define __NR_delete_module 106
+#define __NR_timer_create 107
+#define __NR_timer_getoverrun 109
+#define __NR_timer_delete 111
+#define __NR_syslog 116
+#define __NR_ptrace 117
+#define __NR_sched_setparam 118
+#define __NR_sched_setscheduler 119
+#define __NR_sched_getscheduler 120
+#define __NR_sched_getparam 121
+#define __NR_sched_setaffinity 122
+#define __NR_sched_getaffinity 123
+#define __NR_sched_yield 124
+#define __NR_sched_get_priority_max 125
+#define __NR_sched_get_priority_min 126
+#define __NR_restart_syscall 128
+#define __NR_kill 129
+#define __NR_tkill 130
+#define __NR_tgkill 131
+#define __NR_sigaltstack 132
+#define __NR_rt_sigsuspend 133
+#define __NR_rt_sigaction 134
+#define __NR_rt_sigprocmask 135
+#define __NR_rt_sigpending 136
+#define __NR_rt_sigqueueinfo 138
+#define __NR_rt_sigreturn 139
+#define __NR_setpriority 140
+#define __NR_getpriority 141
+#define __NR_reboot 142
+#define __NR_setregid 143
+#define __NR_setgid 144
+#define __NR_setreuid 145
+#define __NR_setuid 146
+#define __NR_setresuid 147
+#define __NR_getresuid 148
+#define __NR_setresgid 149
+#define __NR_getresgid 150
+#define __NR_setfsuid 151
+#define __NR_setfsgid 152
+#define __NR_times 153
+#define __NR_setpgid 154
+#define __NR_getpgid 155
+#define __NR_getsid 156
+#define __NR_setsid 157
+#define __NR_getgroups 158
+#define __NR_setgroups 159
+#define __NR_uname 160
+#define __NR_sethostname 161
+#define __NR_setdomainname 162
+#define __NR_getrusage 165
+#define __NR_umask 166
+#define __NR_prctl 167
+#define __NR_getcpu 168
+#define __NR_getpid 172
+#define __NR_getppid 173
+#define __NR_getuid 174
+#define __NR_geteuid 175
+#define __NR_getgid 176
+#define __NR_getegid 177
+#define __NR_gettid 178
+#define __NR_sysinfo 179
+#define __NR_mq_open 180
+#define __NR_mq_unlink 181
+#define __NR_mq_notify 184
+#define __NR_mq_getsetattr 185
+#define __NR_msgget 186
+#define __NR_msgctl 187
+#define __NR_msgrcv 188
+#define __NR_msgsnd 189
+#define __NR_semget 190
+#define __NR_semctl 191
+#define __NR_semop 193
+#define __NR_shmget 194
+#define __NR_shmctl 195
+#define __NR_shmat 196
+#define __NR_shmdt 197
+#define __NR_socket 198
+#define __NR_socketpair 199
+#define __NR_bind 200
+#define __NR_listen 201
+#define __NR_accept 202
+#define __NR_connect 203
+#define __NR_getsockname 204
+#define __NR_getpeername 205
+#define __NR_sendto 206
+#define __NR_recvfrom 207
+#define __NR_setsockopt 208
+#define __NR_getsockopt 209
+#define __NR_shutdown 210
+#define __NR_sendmsg 211
+#define __NR_recvmsg 212
+#define __NR_readahead 213
+#define __NR_brk 214
+#define __NR_munmap 215
+#define __NR_mremap 216
+#define __NR_add_key 217
+#define __NR_request_key 218
+#define __NR_keyctl 219
+#define __NR_clone 220
+#define __NR_execve 221
+#define __NR_mmap2 222
+#define __NR_fadvise64_64 223
+#define __NR_swapon 224
+#define __NR_swapoff 225
+#define __NR_mprotect 226
+#define __NR_msync 227
+#define __NR_mlock 228
+#define __NR_munlock 229
+#define __NR_mlockall 230
+#define __NR_munlockall 231
+#define __NR_mincore 232
+#define __NR_madvise 233
+#define __NR_remap_file_pages 234
+#define __NR_mbind 235
+#define __NR_get_mempolicy 236
+#define __NR_set_mempolicy 237
+#define __NR_migrate_pages 238
+#define __NR_move_pages 239
+#define __NR_rt_tgsigqueueinfo 240
+#define __NR_perf_event_open 241
+#define __NR_accept4 242
+#define __NR_riscv_hwprobe 258
+#define __NR_riscv_flush_icache 259
+#define __NR_prlimit64 261
+#define __NR_fanotify_init 262
+#define __NR_fanotify_mark 263
+#define __NR_name_to_handle_at 264
+#define __NR_open_by_handle_at 265
+#define __NR_syncfs 267
+#define __NR_setns 268
+#define __NR_sendmmsg 269
+#define __NR_process_vm_readv 270
+#define __NR_process_vm_writev 271
+#define __NR_kcmp 272
+#define __NR_finit_module 273
+#define __NR_sched_setattr 274
+#define __NR_sched_getattr 275
+#define __NR_renameat2 276
+#define __NR_seccomp 277
+#define __NR_getrandom 278
+#define __NR_memfd_create 279
+#define __NR_bpf 280
+#define __NR_execveat 281
+#define __NR_userfaultfd 282
+#define __NR_membarrier 283
+#define __NR_mlock2 284
+#define __NR_copy_file_range 285
+#define __NR_preadv2 286
+#define __NR_pwritev2 287
+#define __NR_pkey_mprotect 288
+#define __NR_pkey_alloc 289
+#define __NR_pkey_free 290
+#define __NR_statx 291
+#define __NR_rseq 293
+#define __NR_kexec_file_load 294
+#define __NR_clock_gettime64 403
+#define __NR_clock_settime64 404
+#define __NR_clock_adjtime64 405
+#define __NR_clock_getres_time64 406
+#define __NR_clock_nanosleep_time64 407
+#define __NR_timer_gettime64 408
+#define __NR_timer_settime64 409
+#define __NR_timerfd_gettime64 410
+#define __NR_timerfd_settime64 411
+#define __NR_utimensat_time64 412
+#define __NR_pselect6_time64 413
+#define __NR_ppoll_time64 414
+#define __NR_io_pgetevents_time64 416
+#define __NR_recvmmsg_time64 417
+#define __NR_mq_timedsend_time64 418
+#define __NR_mq_timedreceive_time64 419
+#define __NR_semtimedop_time64 420
+#define __NR_rt_sigtimedwait_time64 421
+#define __NR_futex_time64 422
+#define __NR_sched_rr_get_interval_time64 423
+#define __NR_pidfd_send_signal 424
+#define __NR_io_uring_setup 425
+#define __NR_io_uring_enter 426
+#define __NR_io_uring_register 427
+#define __NR_open_tree 428
+#define __NR_move_mount 429
+#define __NR_fsopen 430
+#define __NR_fsconfig 431
+#define __NR_fsmount 432
+#define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
+#define __NR_close_range 436
+#define __NR_openat2 437
+#define __NR_pidfd_getfd 438
+#define __NR_faccessat2 439
+#define __NR_process_madvise 440
+#define __NR_epoll_pwait2 441
+#define __NR_mount_setattr 442
+#define __NR_quotactl_fd 443
+#define __NR_landlock_create_ruleset 444
+#define __NR_landlock_add_rule 445
+#define __NR_landlock_restrict_self 446
+#define __NR_memfd_secret 447
+#define __NR_process_mrelease 448
+#define __NR_futex_waitv 449
+#define __NR_set_mempolicy_home_node 450
+#define __NR_cachestat 451
+#define __NR_fchmodat2 452
+#define __NR_map_shadow_stack 453
+#define __NR_futex_wake 454
+#define __NR_futex_wait 455
+#define __NR_futex_requeue 456
+#define __NR_statmount 457
+#define __NR_listmount 458
+#define __NR_lsm_get_self_attr 459
+#define __NR_lsm_set_self_attr 460
+#define __NR_lsm_list_modules 461
+#define __NR_mseal 462
+#endif
diff --git a/libc/kernel/uapi/asm-riscv/asm/unistd_64.h b/libc/kernel/uapi/asm-riscv/asm/unistd_64.h
new file mode 100644
index 0000000..f15b65b
--- /dev/null
+++ b/libc/kernel/uapi/asm-riscv/asm/unistd_64.h
@@ -0,0 +1,328 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef _UAPI_ASM_UNISTD_64_H
+#define _UAPI_ASM_UNISTD_64_H
+#define __NR_io_setup 0
+#define __NR_io_destroy 1
+#define __NR_io_submit 2
+#define __NR_io_cancel 3
+#define __NR_io_getevents 4
+#define __NR_setxattr 5
+#define __NR_lsetxattr 6
+#define __NR_fsetxattr 7
+#define __NR_getxattr 8
+#define __NR_lgetxattr 9
+#define __NR_fgetxattr 10
+#define __NR_listxattr 11
+#define __NR_llistxattr 12
+#define __NR_flistxattr 13
+#define __NR_removexattr 14
+#define __NR_lremovexattr 15
+#define __NR_fremovexattr 16
+#define __NR_getcwd 17
+#define __NR_lookup_dcookie 18
+#define __NR_eventfd2 19
+#define __NR_epoll_create1 20
+#define __NR_epoll_ctl 21
+#define __NR_epoll_pwait 22
+#define __NR_dup 23
+#define __NR_dup3 24
+#define __NR_fcntl 25
+#define __NR_inotify_init1 26
+#define __NR_inotify_add_watch 27
+#define __NR_inotify_rm_watch 28
+#define __NR_ioctl 29
+#define __NR_ioprio_set 30
+#define __NR_ioprio_get 31
+#define __NR_flock 32
+#define __NR_mknodat 33
+#define __NR_mkdirat 34
+#define __NR_unlinkat 35
+#define __NR_symlinkat 36
+#define __NR_linkat 37
+#define __NR_umount2 39
+#define __NR_mount 40
+#define __NR_pivot_root 41
+#define __NR_nfsservctl 42
+#define __NR_statfs 43
+#define __NR_fstatfs 44
+#define __NR_truncate 45
+#define __NR_ftruncate 46
+#define __NR_fallocate 47
+#define __NR_faccessat 48
+#define __NR_chdir 49
+#define __NR_fchdir 50
+#define __NR_chroot 51
+#define __NR_fchmod 52
+#define __NR_fchmodat 53
+#define __NR_fchownat 54
+#define __NR_fchown 55
+#define __NR_openat 56
+#define __NR_close 57
+#define __NR_vhangup 58
+#define __NR_pipe2 59
+#define __NR_quotactl 60
+#define __NR_getdents64 61
+#define __NR_lseek 62
+#define __NR_read 63
+#define __NR_write 64
+#define __NR_readv 65
+#define __NR_writev 66
+#define __NR_pread64 67
+#define __NR_pwrite64 68
+#define __NR_preadv 69
+#define __NR_pwritev 70
+#define __NR_sendfile 71
+#define __NR_pselect6 72
+#define __NR_ppoll 73
+#define __NR_signalfd4 74
+#define __NR_vmsplice 75
+#define __NR_splice 76
+#define __NR_tee 77
+#define __NR_readlinkat 78
+#define __NR_newfstatat 79
+#define __NR_fstat 80
+#define __NR_sync 81
+#define __NR_fsync 82
+#define __NR_fdatasync 83
+#define __NR_sync_file_range 84
+#define __NR_timerfd_create 85
+#define __NR_timerfd_settime 86
+#define __NR_timerfd_gettime 87
+#define __NR_utimensat 88
+#define __NR_acct 89
+#define __NR_capget 90
+#define __NR_capset 91
+#define __NR_personality 92
+#define __NR_exit 93
+#define __NR_exit_group 94
+#define __NR_waitid 95
+#define __NR_set_tid_address 96
+#define __NR_unshare 97
+#define __NR_futex 98
+#define __NR_set_robust_list 99
+#define __NR_get_robust_list 100
+#define __NR_nanosleep 101
+#define __NR_getitimer 102
+#define __NR_setitimer 103
+#define __NR_kexec_load 104
+#define __NR_init_module 105
+#define __NR_delete_module 106
+#define __NR_timer_create 107
+#define __NR_timer_gettime 108
+#define __NR_timer_getoverrun 109
+#define __NR_timer_settime 110
+#define __NR_timer_delete 111
+#define __NR_clock_settime 112
+#define __NR_clock_gettime 113
+#define __NR_clock_getres 114
+#define __NR_clock_nanosleep 115
+#define __NR_syslog 116
+#define __NR_ptrace 117
+#define __NR_sched_setparam 118
+#define __NR_sched_setscheduler 119
+#define __NR_sched_getscheduler 120
+#define __NR_sched_getparam 121
+#define __NR_sched_setaffinity 122
+#define __NR_sched_getaffinity 123
+#define __NR_sched_yield 124
+#define __NR_sched_get_priority_max 125
+#define __NR_sched_get_priority_min 126
+#define __NR_sched_rr_get_interval 127
+#define __NR_restart_syscall 128
+#define __NR_kill 129
+#define __NR_tkill 130
+#define __NR_tgkill 131
+#define __NR_sigaltstack 132
+#define __NR_rt_sigsuspend 133
+#define __NR_rt_sigaction 134
+#define __NR_rt_sigprocmask 135
+#define __NR_rt_sigpending 136
+#define __NR_rt_sigtimedwait 137
+#define __NR_rt_sigqueueinfo 138
+#define __NR_rt_sigreturn 139
+#define __NR_setpriority 140
+#define __NR_getpriority 141
+#define __NR_reboot 142
+#define __NR_setregid 143
+#define __NR_setgid 144
+#define __NR_setreuid 145
+#define __NR_setuid 146
+#define __NR_setresuid 147
+#define __NR_getresuid 148
+#define __NR_setresgid 149
+#define __NR_getresgid 150
+#define __NR_setfsuid 151
+#define __NR_setfsgid 152
+#define __NR_times 153
+#define __NR_setpgid 154
+#define __NR_getpgid 155
+#define __NR_getsid 156
+#define __NR_setsid 157
+#define __NR_getgroups 158
+#define __NR_setgroups 159
+#define __NR_uname 160
+#define __NR_sethostname 161
+#define __NR_setdomainname 162
+#define __NR_getrlimit 163
+#define __NR_setrlimit 164
+#define __NR_getrusage 165
+#define __NR_umask 166
+#define __NR_prctl 167
+#define __NR_getcpu 168
+#define __NR_gettimeofday 169
+#define __NR_settimeofday 170
+#define __NR_adjtimex 171
+#define __NR_getpid 172
+#define __NR_getppid 173
+#define __NR_getuid 174
+#define __NR_geteuid 175
+#define __NR_getgid 176
+#define __NR_getegid 177
+#define __NR_gettid 178
+#define __NR_sysinfo 179
+#define __NR_mq_open 180
+#define __NR_mq_unlink 181
+#define __NR_mq_timedsend 182
+#define __NR_mq_timedreceive 183
+#define __NR_mq_notify 184
+#define __NR_mq_getsetattr 185
+#define __NR_msgget 186
+#define __NR_msgctl 187
+#define __NR_msgrcv 188
+#define __NR_msgsnd 189
+#define __NR_semget 190
+#define __NR_semctl 191
+#define __NR_semtimedop 192
+#define __NR_semop 193
+#define __NR_shmget 194
+#define __NR_shmctl 195
+#define __NR_shmat 196
+#define __NR_shmdt 197
+#define __NR_socket 198
+#define __NR_socketpair 199
+#define __NR_bind 200
+#define __NR_listen 201
+#define __NR_accept 202
+#define __NR_connect 203
+#define __NR_getsockname 204
+#define __NR_getpeername 205
+#define __NR_sendto 206
+#define __NR_recvfrom 207
+#define __NR_setsockopt 208
+#define __NR_getsockopt 209
+#define __NR_shutdown 210
+#define __NR_sendmsg 211
+#define __NR_recvmsg 212
+#define __NR_readahead 213
+#define __NR_brk 214
+#define __NR_munmap 215
+#define __NR_mremap 216
+#define __NR_add_key 217
+#define __NR_request_key 218
+#define __NR_keyctl 219
+#define __NR_clone 220
+#define __NR_execve 221
+#define __NR_mmap 222
+#define __NR_fadvise64 223
+#define __NR_swapon 224
+#define __NR_swapoff 225
+#define __NR_mprotect 226
+#define __NR_msync 227
+#define __NR_mlock 228
+#define __NR_munlock 229
+#define __NR_mlockall 230
+#define __NR_munlockall 231
+#define __NR_mincore 232
+#define __NR_madvise 233
+#define __NR_remap_file_pages 234
+#define __NR_mbind 235
+#define __NR_get_mempolicy 236
+#define __NR_set_mempolicy 237
+#define __NR_migrate_pages 238
+#define __NR_move_pages 239
+#define __NR_rt_tgsigqueueinfo 240
+#define __NR_perf_event_open 241
+#define __NR_accept4 242
+#define __NR_recvmmsg 243
+#define __NR_riscv_hwprobe 258
+#define __NR_riscv_flush_icache 259
+#define __NR_wait4 260
+#define __NR_prlimit64 261
+#define __NR_fanotify_init 262
+#define __NR_fanotify_mark 263
+#define __NR_name_to_handle_at 264
+#define __NR_open_by_handle_at 265
+#define __NR_clock_adjtime 266
+#define __NR_syncfs 267
+#define __NR_setns 268
+#define __NR_sendmmsg 269
+#define __NR_process_vm_readv 270
+#define __NR_process_vm_writev 271
+#define __NR_kcmp 272
+#define __NR_finit_module 273
+#define __NR_sched_setattr 274
+#define __NR_sched_getattr 275
+#define __NR_renameat2 276
+#define __NR_seccomp 277
+#define __NR_getrandom 278
+#define __NR_memfd_create 279
+#define __NR_bpf 280
+#define __NR_execveat 281
+#define __NR_userfaultfd 282
+#define __NR_membarrier 283
+#define __NR_mlock2 284
+#define __NR_copy_file_range 285
+#define __NR_preadv2 286
+#define __NR_pwritev2 287
+#define __NR_pkey_mprotect 288
+#define __NR_pkey_alloc 289
+#define __NR_pkey_free 290
+#define __NR_statx 291
+#define __NR_io_pgetevents 292
+#define __NR_rseq 293
+#define __NR_kexec_file_load 294
+#define __NR_pidfd_send_signal 424
+#define __NR_io_uring_setup 425
+#define __NR_io_uring_enter 426
+#define __NR_io_uring_register 427
+#define __NR_open_tree 428
+#define __NR_move_mount 429
+#define __NR_fsopen 430
+#define __NR_fsconfig 431
+#define __NR_fsmount 432
+#define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
+#define __NR_close_range 436
+#define __NR_openat2 437
+#define __NR_pidfd_getfd 438
+#define __NR_faccessat2 439
+#define __NR_process_madvise 440
+#define __NR_epoll_pwait2 441
+#define __NR_mount_setattr 442
+#define __NR_quotactl_fd 443
+#define __NR_landlock_create_ruleset 444
+#define __NR_landlock_add_rule 445
+#define __NR_landlock_restrict_self 446
+#define __NR_memfd_secret 447
+#define __NR_process_mrelease 448
+#define __NR_futex_waitv 449
+#define __NR_set_mempolicy_home_node 450
+#define __NR_cachestat 451
+#define __NR_fchmodat2 452
+#define __NR_map_shadow_stack 453
+#define __NR_futex_wake 454
+#define __NR_futex_wait 455
+#define __NR_futex_requeue 456
+#define __NR_statmount 457
+#define __NR_listmount 458
+#define __NR_lsm_get_self_attr 459
+#define __NR_lsm_set_self_attr 460
+#define __NR_lsm_list_modules 461
+#define __NR_mseal 462
+#endif
diff --git a/libc/kernel/uapi/asm-x86/asm/kvm.h b/libc/kernel/uapi/asm-x86/asm/kvm.h
index 17b1c5d..cd647b6 100644
--- a/libc/kernel/uapi/asm-x86/asm/kvm.h
+++ b/libc/kernel/uapi/asm-x86/asm/kvm.h
@@ -94,6 +94,7 @@
 #define KVM_NR_IRQCHIPS 3
 #define KVM_RUN_X86_SMM (1 << 0)
 #define KVM_RUN_X86_BUS_LOCK (1 << 1)
+#define KVM_RUN_X86_GUEST_MODE (1 << 2)
 struct kvm_regs {
   __u64 rax, rbx, rcx, rdx;
   __u64 rsi, rdi, rsp, rbp;
@@ -532,6 +533,9 @@
   KVM_SEV_GET_ATTESTATION_REPORT,
   KVM_SEV_SEND_CANCEL,
   KVM_SEV_INIT2,
+  KVM_SEV_SNP_LAUNCH_START = 100,
+  KVM_SEV_SNP_LAUNCH_UPDATE,
+  KVM_SEV_SNP_LAUNCH_FINISH,
   KVM_SEV_NR_MAX,
 };
 struct kvm_sev_cmd {
@@ -644,6 +648,42 @@
   __u32 trans_len;
   __u32 pad2;
 };
+struct kvm_sev_snp_launch_start {
+  __u64 policy;
+  __u8 gosvw[16];
+  __u16 flags;
+  __u8 pad0[6];
+  __u64 pad1[4];
+};
+#define KVM_SEV_SNP_PAGE_TYPE_NORMAL 0x1
+#define KVM_SEV_SNP_PAGE_TYPE_ZERO 0x3
+#define KVM_SEV_SNP_PAGE_TYPE_UNMEASURED 0x4
+#define KVM_SEV_SNP_PAGE_TYPE_SECRETS 0x5
+#define KVM_SEV_SNP_PAGE_TYPE_CPUID 0x6
+struct kvm_sev_snp_launch_update {
+  __u64 gfn_start;
+  __u64 uaddr;
+  __u64 len;
+  __u8 type;
+  __u8 pad0;
+  __u16 flags;
+  __u32 pad1;
+  __u64 pad2[4];
+};
+#define KVM_SEV_SNP_ID_BLOCK_SIZE 96
+#define KVM_SEV_SNP_ID_AUTH_SIZE 4096
+#define KVM_SEV_SNP_FINISH_DATA_SIZE 32
+struct kvm_sev_snp_launch_finish {
+  __u64 id_block_uaddr;
+  __u64 id_auth_uaddr;
+  __u8 id_block_en;
+  __u8 auth_key_en;
+  __u8 vcek_disabled;
+  __u8 host_data[KVM_SEV_SNP_FINISH_DATA_SIZE];
+  __u8 pad0[3];
+  __u16 flags;
+  __u64 pad1[4];
+};
 #define KVM_X2APIC_API_USE_32BIT_IDS (1ULL << 0)
 #define KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK (1ULL << 1)
 struct kvm_hyperv_eventfd {
@@ -667,4 +707,5 @@
 #define KVM_X86_SW_PROTECTED_VM 1
 #define KVM_X86_SEV_VM 2
 #define KVM_X86_SEV_ES_VM 3
+#define KVM_X86_SNP_VM 4
 #endif
diff --git a/libc/kernel/uapi/asm-x86/asm/svm.h b/libc/kernel/uapi/asm-x86/asm/svm.h
index ffbf0b3..4f165fa 100644
--- a/libc/kernel/uapi/asm-x86/asm/svm.h
+++ b/libc/kernel/uapi/asm-x86/asm/svm.h
@@ -117,6 +117,7 @@
 #define SVM_VMGEXIT_AP_CREATE_ON_INIT 0
 #define SVM_VMGEXIT_AP_CREATE 1
 #define SVM_VMGEXIT_AP_DESTROY 2
+#define SVM_VMGEXIT_SNP_RUN_VMPL 0x80000018
 #define SVM_VMGEXIT_HV_FEATURES 0x8000fffd
 #define SVM_VMGEXIT_TERM_REQUEST 0x8000fffe
 #define SVM_VMGEXIT_TERM_REASON(reason_set,reason_code) (((((u64) reason_set) & 0xf)) | ((((u64) reason_code) & 0xff) << 4))
diff --git a/libc/kernel/uapi/asm-x86/asm/unistd_64.h b/libc/kernel/uapi/asm-x86/asm/unistd_64.h
index 5dd666c..d5408a3 100644
--- a/libc/kernel/uapi/asm-x86/asm/unistd_64.h
+++ b/libc/kernel/uapi/asm-x86/asm/unistd_64.h
@@ -341,6 +341,7 @@
 #define __NR_statx 332
 #define __NR_io_pgetevents 333
 #define __NR_rseq 334
+#define __NR_uretprobe 335
 #define __NR_pidfd_send_signal 424
 #define __NR_io_uring_setup 425
 #define __NR_io_uring_enter 426
diff --git a/libc/kernel/uapi/asm-x86/asm/unistd_x32.h b/libc/kernel/uapi/asm-x86/asm/unistd_x32.h
index a2ff6f4..fdcf7e6 100644
--- a/libc/kernel/uapi/asm-x86/asm/unistd_x32.h
+++ b/libc/kernel/uapi/asm-x86/asm/unistd_x32.h
@@ -294,6 +294,7 @@
 #define __NR_statx (__X32_SYSCALL_BIT + 332)
 #define __NR_io_pgetevents (__X32_SYSCALL_BIT + 333)
 #define __NR_rseq (__X32_SYSCALL_BIT + 334)
+#define __NR_uretprobe (__X32_SYSCALL_BIT + 335)
 #define __NR_pidfd_send_signal (__X32_SYSCALL_BIT + 424)
 #define __NR_io_uring_setup (__X32_SYSCALL_BIT + 425)
 #define __NR_io_uring_enter (__X32_SYSCALL_BIT + 426)
diff --git a/libc/kernel/uapi/drm/amdgpu_drm.h b/libc/kernel/uapi/drm/amdgpu_drm.h
index 0ad0bc2..7bbd5de 100644
--- a/libc/kernel/uapi/drm/amdgpu_drm.h
+++ b/libc/kernel/uapi/drm/amdgpu_drm.h
@@ -65,6 +65,7 @@
 #define AMDGPU_GEM_CREATE_COHERENT (1 << 13)
 #define AMDGPU_GEM_CREATE_UNCACHED (1 << 14)
 #define AMDGPU_GEM_CREATE_EXT_COHERENT (1 << 15)
+#define AMDGPU_GEM_CREATE_GFX12_DCC (1 << 16)
 struct drm_amdgpu_gem_create_in {
   __u64 bo_size;
   __u64 alignment;
@@ -216,6 +217,14 @@
 #define AMDGPU_TILING_DCC_INDEPENDENT_128B_MASK 0x1
 #define AMDGPU_TILING_SCANOUT_SHIFT 63
 #define AMDGPU_TILING_SCANOUT_MASK 0x1
+#define AMDGPU_TILING_GFX12_SWIZZLE_MODE_SHIFT 0
+#define AMDGPU_TILING_GFX12_SWIZZLE_MODE_MASK 0x7
+#define AMDGPU_TILING_GFX12_DCC_MAX_COMPRESSED_BLOCK_SHIFT 3
+#define AMDGPU_TILING_GFX12_DCC_MAX_COMPRESSED_BLOCK_MASK 0x3
+#define AMDGPU_TILING_GFX12_DCC_NUMBER_TYPE_SHIFT 5
+#define AMDGPU_TILING_GFX12_DCC_NUMBER_TYPE_MASK 0x7
+#define AMDGPU_TILING_GFX12_DCC_DATA_FORMAT_SHIFT 8
+#define AMDGPU_TILING_GFX12_DCC_DATA_FORMAT_MASK 0x3f
 #define AMDGPU_TILING_SET(field,value) (((__u64) (value) & AMDGPU_TILING_ ##field ##_MASK) << AMDGPU_TILING_ ##field ##_SHIFT)
 #define AMDGPU_TILING_GET(value,field) (((__u64) (value) >> AMDGPU_TILING_ ##field ##_SHIFT) & AMDGPU_TILING_ ##field ##_MASK)
 #define AMDGPU_GEM_METADATA_OP_SET_METADATA 1
@@ -744,6 +753,10 @@
 #define AMDGPU_FAMILY_GC_10_3_6 149
 #define AMDGPU_FAMILY_GC_10_3_7 151
 #define AMDGPU_FAMILY_GC_11_5_0 150
+#define AMDGPU_FAMILY_GC_12_0_0 152
+struct drm_color_ctm_3x4 {
+  __u64 matrix[12];
+};
 #ifdef __cplusplus
 }
 #endif
diff --git a/libc/kernel/uapi/drm/drm_fourcc.h b/libc/kernel/uapi/drm/drm_fourcc.h
index 6fd2eb8..4902d6c 100644
--- a/libc/kernel/uapi/drm/drm_fourcc.h
+++ b/libc/kernel/uapi/drm/drm_fourcc.h
@@ -255,12 +255,17 @@
 #define AMD_FMT_MOD_TILE_VER_GFX10 2
 #define AMD_FMT_MOD_TILE_VER_GFX10_RBPLUS 3
 #define AMD_FMT_MOD_TILE_VER_GFX11 4
+#define AMD_FMT_MOD_TILE_VER_GFX12 5
 #define AMD_FMT_MOD_TILE_GFX9_64K_S 9
 #define AMD_FMT_MOD_TILE_GFX9_64K_D 10
 #define AMD_FMT_MOD_TILE_GFX9_64K_S_X 25
 #define AMD_FMT_MOD_TILE_GFX9_64K_D_X 26
 #define AMD_FMT_MOD_TILE_GFX9_64K_R_X 27
 #define AMD_FMT_MOD_TILE_GFX11_256K_R_X 31
+#define AMD_FMT_MOD_TILE_GFX12_256B_2D 1
+#define AMD_FMT_MOD_TILE_GFX12_4K_2D 2
+#define AMD_FMT_MOD_TILE_GFX12_64K_2D 3
+#define AMD_FMT_MOD_TILE_GFX12_256K_2D 4
 #define AMD_FMT_MOD_DCC_BLOCK_64B 0
 #define AMD_FMT_MOD_DCC_BLOCK_128B 1
 #define AMD_FMT_MOD_DCC_BLOCK_256B 2
diff --git a/libc/kernel/uapi/drm/drm_mode.h b/libc/kernel/uapi/drm/drm_mode.h
index 8fccdaf..06c91c5 100644
--- a/libc/kernel/uapi/drm/drm_mode.h
+++ b/libc/kernel/uapi/drm/drm_mode.h
@@ -357,9 +357,6 @@
 struct drm_color_ctm {
   __u64 matrix[9];
 };
-struct drm_color_ctm_3x4 {
-  __u64 matrix[12];
-};
 struct drm_color_lut {
   __u16 red;
   __u16 green;
diff --git a/libc/kernel/uapi/drm/i915_drm.h b/libc/kernel/uapi/drm/i915_drm.h
index 13eda7c..b43d8df 100644
--- a/libc/kernel/uapi/drm/i915_drm.h
+++ b/libc/kernel/uapi/drm/i915_drm.h
@@ -745,6 +745,7 @@
 #define I915_CONTEXT_PARAM_RINGSIZE 0xc
 #define I915_CONTEXT_PARAM_PROTECTED_CONTENT 0xd
 #define I915_CONTEXT_PARAM_LOW_LATENCY 0xe
+#define I915_CONTEXT_PARAM_CONTEXT_IMAGE 0xf
   __u64 value;
 };
 struct drm_i915_gem_context_param_sseu {
@@ -799,6 +800,14 @@
 } __attribute__((packed));
 #define I915_DEFINE_CONTEXT_PARAM_ENGINES(name__,N__) struct { __u64 extensions; struct i915_engine_class_instance engines[N__]; \
 } __attribute__((packed)) name__
+struct i915_gem_context_param_context_image {
+  struct i915_engine_class_instance engine;
+  __u32 flags;
+#define I915_CONTEXT_IMAGE_FLAG_ENGINE_INDEX (1u << 0)
+  __u32 size;
+  __u32 mbz;
+  __u64 image;
+} __attribute__((packed));
 struct drm_i915_gem_context_create_ext_setparam {
   struct i915_user_extension base;
   struct drm_i915_gem_context_param param;
diff --git a/libc/kernel/uapi/drm/ivpu_accel.h b/libc/kernel/uapi/drm/ivpu_accel.h
index fcbf6f7..960bd43 100644
--- a/libc/kernel/uapi/drm/ivpu_accel.h
+++ b/libc/kernel/uapi/drm/ivpu_accel.h
@@ -18,12 +18,20 @@
 #define DRM_IVPU_BO_INFO 0x03
 #define DRM_IVPU_SUBMIT 0x05
 #define DRM_IVPU_BO_WAIT 0x06
+#define DRM_IVPU_METRIC_STREAMER_START 0x07
+#define DRM_IVPU_METRIC_STREAMER_STOP 0x08
+#define DRM_IVPU_METRIC_STREAMER_GET_DATA 0x09
+#define DRM_IVPU_METRIC_STREAMER_GET_INFO 0x0a
 #define DRM_IOCTL_IVPU_GET_PARAM DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_GET_PARAM, struct drm_ivpu_param)
 #define DRM_IOCTL_IVPU_SET_PARAM DRM_IOW(DRM_COMMAND_BASE + DRM_IVPU_SET_PARAM, struct drm_ivpu_param)
 #define DRM_IOCTL_IVPU_BO_CREATE DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_BO_CREATE, struct drm_ivpu_bo_create)
 #define DRM_IOCTL_IVPU_BO_INFO DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_BO_INFO, struct drm_ivpu_bo_info)
 #define DRM_IOCTL_IVPU_SUBMIT DRM_IOW(DRM_COMMAND_BASE + DRM_IVPU_SUBMIT, struct drm_ivpu_submit)
 #define DRM_IOCTL_IVPU_BO_WAIT DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_BO_WAIT, struct drm_ivpu_bo_wait)
+#define DRM_IOCTL_IVPU_METRIC_STREAMER_START DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_METRIC_STREAMER_START, struct drm_ivpu_metric_streamer_start)
+#define DRM_IOCTL_IVPU_METRIC_STREAMER_STOP DRM_IOW(DRM_COMMAND_BASE + DRM_IVPU_METRIC_STREAMER_STOP, struct drm_ivpu_metric_streamer_stop)
+#define DRM_IOCTL_IVPU_METRIC_STREAMER_GET_DATA DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_METRIC_STREAMER_GET_DATA, struct drm_ivpu_metric_streamer_get_data)
+#define DRM_IOCTL_IVPU_METRIC_STREAMER_GET_INFO DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_METRIC_STREAMER_GET_INFO, struct drm_ivpu_metric_streamer_get_data)
 #define DRM_IVPU_PARAM_DEVICE_ID 0
 #define DRM_IVPU_PARAM_DEVICE_REVISION 1
 #define DRM_IVPU_PARAM_PLATFORM_TYPE 2
@@ -96,6 +104,22 @@
   __u32 job_status;
   __u32 pad;
 };
+struct drm_ivpu_metric_streamer_start {
+  __u64 metric_group_mask;
+  __u64 sampling_period_ns;
+  __u32 read_period_samples;
+  __u32 sample_size;
+  __u32 max_data_size;
+};
+struct drm_ivpu_metric_streamer_get_data {
+  __u64 metric_group_mask;
+  __u64 buffer_ptr;
+  __u64 buffer_size;
+  __u64 data_size;
+};
+struct drm_ivpu_metric_streamer_stop {
+  __u64 metric_group_mask;
+};
 #ifdef __cplusplus
 }
 #endif
diff --git a/libc/kernel/uapi/drm/msm_drm.h b/libc/kernel/uapi/drm/msm_drm.h
index 4d83744..7ec5ed2 100644
--- a/libc/kernel/uapi/drm/msm_drm.h
+++ b/libc/kernel/uapi/drm/msm_drm.h
@@ -37,6 +37,7 @@
 #define MSM_PARAM_VA_START 0x0e
 #define MSM_PARAM_VA_SIZE 0x0f
 #define MSM_PARAM_HIGHEST_BANK_BIT 0x10
+#define MSM_PARAM_RAYTRACING 0x11
 #define MSM_PARAM_NR_RINGS MSM_PARAM_PRIORITIES
 struct drm_msm_param {
   __u32 pipe;
diff --git a/libc/kernel/uapi/drm/v3d_drm.h b/libc/kernel/uapi/drm/v3d_drm.h
index 4000fd3..b7aca21 100644
--- a/libc/kernel/uapi/drm/v3d_drm.h
+++ b/libc/kernel/uapi/drm/v3d_drm.h
@@ -22,6 +22,7 @@
 #define DRM_V3D_PERFMON_DESTROY 0x09
 #define DRM_V3D_PERFMON_GET_VALUES 0x0a
 #define DRM_V3D_SUBMIT_CPU 0x0b
+#define DRM_V3D_PERFMON_GET_COUNTER 0x0c
 #define DRM_IOCTL_V3D_SUBMIT_CL DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_SUBMIT_CL, struct drm_v3d_submit_cl)
 #define DRM_IOCTL_V3D_WAIT_BO DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_WAIT_BO, struct drm_v3d_wait_bo)
 #define DRM_IOCTL_V3D_CREATE_BO DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_CREATE_BO, struct drm_v3d_create_bo)
@@ -34,6 +35,7 @@
 #define DRM_IOCTL_V3D_PERFMON_DESTROY DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_PERFMON_DESTROY, struct drm_v3d_perfmon_destroy)
 #define DRM_IOCTL_V3D_PERFMON_GET_VALUES DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_PERFMON_GET_VALUES, struct drm_v3d_perfmon_get_values)
 #define DRM_IOCTL_V3D_SUBMIT_CPU DRM_IOW(DRM_COMMAND_BASE + DRM_V3D_SUBMIT_CPU, struct drm_v3d_submit_cpu)
+#define DRM_IOCTL_V3D_PERFMON_GET_COUNTER DRM_IOWR(DRM_COMMAND_BASE + DRM_V3D_PERFMON_GET_COUNTER, struct drm_v3d_perfmon_get_counter)
 #define DRM_V3D_SUBMIT_CL_FLUSH_CACHE 0x01
 #define DRM_V3D_SUBMIT_EXTENSION 0x02
 struct drm_v3d_extension {
@@ -119,6 +121,7 @@
   DRM_V3D_PARAM_SUPPORTS_PERFMON,
   DRM_V3D_PARAM_SUPPORTS_MULTISYNC_EXT,
   DRM_V3D_PARAM_SUPPORTS_CPU_QUEUE,
+  DRM_V3D_PARAM_MAX_PERF_COUNTERS,
 };
 struct drm_v3d_get_param {
   __u32 param;
@@ -324,6 +327,16 @@
   __u32 pad;
   __u64 values_ptr;
 };
+#define DRM_V3D_PERFCNT_MAX_NAME 64
+#define DRM_V3D_PERFCNT_MAX_CATEGORY 32
+#define DRM_V3D_PERFCNT_MAX_DESCRIPTION 256
+struct drm_v3d_perfmon_get_counter {
+  __u8 counter;
+  __u8 name[DRM_V3D_PERFCNT_MAX_NAME];
+  __u8 category[DRM_V3D_PERFCNT_MAX_CATEGORY];
+  __u8 description[DRM_V3D_PERFCNT_MAX_DESCRIPTION];
+  __u8 reserved[7];
+};
 #ifdef __cplusplus
 }
 #endif
diff --git a/libc/kernel/uapi/drm/xe_drm.h b/libc/kernel/uapi/drm/xe_drm.h
index d1b6dad..a034b29 100644
--- a/libc/kernel/uapi/drm/xe_drm.h
+++ b/libc/kernel/uapi/drm/xe_drm.h
@@ -21,6 +21,7 @@
 #define DRM_XE_EXEC_QUEUE_GET_PROPERTY 0x08
 #define DRM_XE_EXEC 0x09
 #define DRM_XE_WAIT_USER_FENCE 0x0a
+#define DRM_XE_OBSERVATION 0x0b
 #define DRM_IOCTL_XE_DEVICE_QUERY DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_DEVICE_QUERY, struct drm_xe_device_query)
 #define DRM_IOCTL_XE_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_GEM_CREATE, struct drm_xe_gem_create)
 #define DRM_IOCTL_XE_GEM_MMAP_OFFSET DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_GEM_MMAP_OFFSET, struct drm_xe_gem_mmap_offset)
@@ -32,6 +33,7 @@
 #define DRM_IOCTL_XE_EXEC_QUEUE_GET_PROPERTY DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_EXEC_QUEUE_GET_PROPERTY, struct drm_xe_exec_queue_get_property)
 #define DRM_IOCTL_XE_EXEC DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)
 #define DRM_IOCTL_XE_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence)
+#define DRM_IOCTL_XE_OBSERVATION DRM_IOW(DRM_COMMAND_BASE + DRM_XE_OBSERVATION, struct drm_xe_observation_param)
 struct drm_xe_user_extension {
   __u64 next_extension;
   __u32 name;
@@ -120,6 +122,7 @@
   __u16 gt_id;
 #define DRM_XE_TOPO_DSS_GEOMETRY 1
 #define DRM_XE_TOPO_DSS_COMPUTE 2
+#define DRM_XE_TOPO_L3_BANK 3
 #define DRM_XE_TOPO_EU_PER_DSS 4
   __u16 type;
   __u32 num_bytes;
@@ -155,6 +158,7 @@
 #define DRM_XE_DEVICE_QUERY_GT_TOPOLOGY 5
 #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES 6
 #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION 7
+#define DRM_XE_DEVICE_QUERY_OA_UNITS 8
   __u32 query;
   __u32 size;
   __u64 data;
@@ -309,6 +313,92 @@
   __u32 pad2;
   __u64 reserved[2];
 };
+enum drm_xe_observation_type {
+  DRM_XE_OBSERVATION_TYPE_OA,
+};
+enum drm_xe_observation_op {
+  DRM_XE_OBSERVATION_OP_STREAM_OPEN,
+  DRM_XE_OBSERVATION_OP_ADD_CONFIG,
+  DRM_XE_OBSERVATION_OP_REMOVE_CONFIG,
+};
+struct drm_xe_observation_param {
+  __u64 extensions;
+  __u64 observation_type;
+  __u64 observation_op;
+  __u64 param;
+};
+enum drm_xe_observation_ioctls {
+  DRM_XE_OBSERVATION_IOCTL_ENABLE = _IO('i', 0x0),
+  DRM_XE_OBSERVATION_IOCTL_DISABLE = _IO('i', 0x1),
+  DRM_XE_OBSERVATION_IOCTL_CONFIG = _IO('i', 0x2),
+  DRM_XE_OBSERVATION_IOCTL_STATUS = _IO('i', 0x3),
+  DRM_XE_OBSERVATION_IOCTL_INFO = _IO('i', 0x4),
+};
+enum drm_xe_oa_unit_type {
+  DRM_XE_OA_UNIT_TYPE_OAG,
+  DRM_XE_OA_UNIT_TYPE_OAM,
+};
+struct drm_xe_oa_unit {
+  __u64 extensions;
+  __u32 oa_unit_id;
+  __u32 oa_unit_type;
+  __u64 capabilities;
+#define DRM_XE_OA_CAPS_BASE (1 << 0)
+  __u64 oa_timestamp_freq;
+  __u64 reserved[4];
+  __u64 num_engines;
+  struct drm_xe_engine_class_instance eci[];
+};
+struct drm_xe_query_oa_units {
+  __u64 extensions;
+  __u32 num_oa_units;
+  __u32 pad;
+  __u64 oa_units[];
+};
+enum drm_xe_oa_format_type {
+  DRM_XE_OA_FMT_TYPE_OAG,
+  DRM_XE_OA_FMT_TYPE_OAR,
+  DRM_XE_OA_FMT_TYPE_OAM,
+  DRM_XE_OA_FMT_TYPE_OAC,
+  DRM_XE_OA_FMT_TYPE_OAM_MPEC,
+  DRM_XE_OA_FMT_TYPE_PEC,
+};
+enum drm_xe_oa_property_id {
+#define DRM_XE_OA_EXTENSION_SET_PROPERTY 0
+  DRM_XE_OA_PROPERTY_OA_UNIT_ID = 1,
+  DRM_XE_OA_PROPERTY_SAMPLE_OA,
+  DRM_XE_OA_PROPERTY_OA_METRIC_SET,
+  DRM_XE_OA_PROPERTY_OA_FORMAT,
+#define DRM_XE_OA_FORMAT_MASK_FMT_TYPE (0xffu << 0)
+#define DRM_XE_OA_FORMAT_MASK_COUNTER_SEL (0xffu << 8)
+#define DRM_XE_OA_FORMAT_MASK_COUNTER_SIZE (0xffu << 16)
+#define DRM_XE_OA_FORMAT_MASK_BC_REPORT (0xffu << 24)
+  DRM_XE_OA_PROPERTY_OA_PERIOD_EXPONENT,
+  DRM_XE_OA_PROPERTY_OA_DISABLED,
+  DRM_XE_OA_PROPERTY_EXEC_QUEUE_ID,
+  DRM_XE_OA_PROPERTY_OA_ENGINE_INSTANCE,
+  DRM_XE_OA_PROPERTY_NO_PREEMPT,
+};
+struct drm_xe_oa_config {
+  __u64 extensions;
+  char uuid[36];
+  __u32 n_regs;
+  __u64 regs_ptr;
+};
+struct drm_xe_oa_stream_status {
+  __u64 extensions;
+  __u64 oa_status;
+#define DRM_XE_OASTATUS_MMIO_TRG_Q_FULL (1 << 3)
+#define DRM_XE_OASTATUS_COUNTER_OVERFLOW (1 << 2)
+#define DRM_XE_OASTATUS_BUFFER_OVERFLOW (1 << 1)
+#define DRM_XE_OASTATUS_REPORT_LOST (1 << 0)
+  __u64 reserved[3];
+};
+struct drm_xe_oa_stream_info {
+  __u64 extensions;
+  __u64 oa_buf_size;
+  __u64 reserved[3];
+};
 #ifdef __cplusplus
 }
 #endif
diff --git a/libc/kernel/uapi/linux/bpf.h b/libc/kernel/uapi/linux/bpf.h
index c732920..8d64816 100644
--- a/libc/kernel/uapi/linux/bpf.h
+++ b/libc/kernel/uapi/linux/bpf.h
@@ -367,6 +367,7 @@
 #define BPF_F_QUERY_EFFECTIVE (1U << 0)
 #define BPF_F_TEST_RUN_ON_CPU (1U << 0)
 #define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
+#define BPF_F_TEST_SKB_CHECKSUM_COMPLETE (1U << 2)
 enum bpf_stats_type {
   BPF_STATS_RUN_TIME = 0,
 };
@@ -773,8 +774,11 @@
 #define __bpf_md_ptr(type,name) union { type name; __u64 : 64; \
 } __attribute__((aligned(8)))
 enum {
-  BPF_SKB_TSTAMP_UNSPEC,
-  BPF_SKB_TSTAMP_DELIVERY_MONO,
+  BPF_SKB_TSTAMP_UNSPEC = 0,
+  BPF_SKB_TSTAMP_DELIVERY_MONO = 1,
+  BPF_SKB_CLOCK_REALTIME = 0,
+  BPF_SKB_CLOCK_MONOTONIC = 1,
+  BPF_SKB_CLOCK_TAI = 2,
 };
 struct __sk_buff {
   __u32 len;
diff --git a/libc/kernel/uapi/linux/btrfs_tree.h b/libc/kernel/uapi/linux/btrfs_tree.h
index ea33eee..88f44d9 100644
--- a/libc/kernel/uapi/linux/btrfs_tree.h
+++ b/libc/kernel/uapi/linux/btrfs_tree.h
@@ -279,18 +279,8 @@
   __le64 devid;
   __le64 physical;
 } __attribute__((__packed__));
-#define BTRFS_STRIPE_RAID0 1
-#define BTRFS_STRIPE_RAID1 2
-#define BTRFS_STRIPE_DUP 3
-#define BTRFS_STRIPE_RAID10 4
-#define BTRFS_STRIPE_RAID5 5
-#define BTRFS_STRIPE_RAID6 6
-#define BTRFS_STRIPE_RAID1C3 7
-#define BTRFS_STRIPE_RAID1C4 8
 struct btrfs_stripe_extent {
-  __u8 encoding;
-  __u8 reserved[7];
-  struct btrfs_raid_stride strides[];
+  __DECLARE_FLEX_ARRAY(struct btrfs_raid_stride, strides);
 } __attribute__((__packed__));
 #define BTRFS_HEADER_FLAG_WRITTEN (1ULL << 0)
 #define BTRFS_HEADER_FLAG_RELOC (1ULL << 1)
@@ -300,6 +290,9 @@
 #define BTRFS_SUPER_FLAG_METADUMP_V2 (1ULL << 34)
 #define BTRFS_SUPER_FLAG_CHANGING_FSID (1ULL << 35)
 #define BTRFS_SUPER_FLAG_CHANGING_FSID_V2 (1ULL << 36)
+#define BTRFS_SUPER_FLAG_CHANGING_BG_TREE (1ULL << 38)
+#define BTRFS_SUPER_FLAG_CHANGING_DATA_CSUM (1ULL << 39)
+#define BTRFS_SUPER_FLAG_CHANGING_META_CSUM (1ULL << 40)
 struct btrfs_extent_item {
   __le64 refs;
   __le64 generation;
diff --git a/libc/kernel/uapi/linux/dlm.h b/libc/kernel/uapi/linux/dlm.h
index 3c73908..1344f9b 100644
--- a/libc/kernel/uapi/linux/dlm.h
+++ b/libc/kernel/uapi/linux/dlm.h
@@ -20,4 +20,5 @@
 };
 #define DLM_LSFL_TIMEWARN 0x00000002
 #define DLM_LSFL_NEWEXCL 0x00000008
+#define __DLM_LSFL_RESERVED0 0x00000010
 #endif
diff --git a/libc/kernel/uapi/linux/dma-heap.h b/libc/kernel/uapi/linux/dma-heap.h
index 467e336..ac86b59 100644
--- a/libc/kernel/uapi/linux/dma-heap.h
+++ b/libc/kernel/uapi/linux/dma-heap.h
@@ -9,7 +9,7 @@
 #include <linux/ioctl.h>
 #include <linux/types.h>
 #define DMA_HEAP_VALID_FD_FLAGS (O_CLOEXEC | O_ACCMODE)
-#define DMA_HEAP_VALID_HEAP_FLAGS (0)
+#define DMA_HEAP_VALID_HEAP_FLAGS (0ULL)
 struct dma_heap_allocation_data {
   __u64 len;
   __u32 fd;
diff --git a/libc/kernel/uapi/linux/ethtool.h b/libc/kernel/uapi/linux/ethtool.h
index 334c788..e213ba1 100644
--- a/libc/kernel/uapi/linux/ethtool.h
+++ b/libc/kernel/uapi/linux/ethtool.h
@@ -270,6 +270,56 @@
   ETHTOOL_MODULE_POWER_MODE_LOW = 1,
   ETHTOOL_MODULE_POWER_MODE_HIGH,
 };
+enum ethtool_c33_pse_ext_state {
+  ETHTOOL_C33_PSE_EXT_STATE_ERROR_CONDITION = 1,
+  ETHTOOL_C33_PSE_EXT_STATE_MR_MPS_VALID,
+  ETHTOOL_C33_PSE_EXT_STATE_MR_PSE_ENABLE,
+  ETHTOOL_C33_PSE_EXT_STATE_OPTION_DETECT_TED,
+  ETHTOOL_C33_PSE_EXT_STATE_OPTION_VPORT_LIM,
+  ETHTOOL_C33_PSE_EXT_STATE_OVLD_DETECTED,
+  ETHTOOL_C33_PSE_EXT_STATE_PD_DLL_POWER_TYPE,
+  ETHTOOL_C33_PSE_EXT_STATE_POWER_NOT_AVAILABLE,
+  ETHTOOL_C33_PSE_EXT_STATE_SHORT_DETECTED,
+};
+enum ethtool_c33_pse_ext_substate_mr_mps_valid {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_MR_MPS_VALID_DETECTED_UNDERLOAD = 1,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_MR_MPS_VALID_CONNECTION_OPEN,
+};
+enum ethtool_c33_pse_ext_substate_error_condition {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_NON_EXISTING_PORT = 1,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_UNDEFINED_PORT,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_INTERNAL_HW_FAULT,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_COMM_ERROR_AFTER_FORCE_ON,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_UNKNOWN_PORT_STATUS,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_HOST_CRASH_TURN_OFF,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_HOST_CRASH_FORCE_SHUTDOWN,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_CONFIG_CHANGE,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_ERROR_CONDITION_DETECTED_OVER_TEMP,
+};
+enum ethtool_c33_pse_ext_substate_mr_pse_enable {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_MR_PSE_ENABLE_DISABLE_PIN_ACTIVE = 1,
+};
+enum ethtool_c33_pse_ext_substate_option_detect_ted {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OPTION_DETECT_TED_DET_IN_PROCESS = 1,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OPTION_DETECT_TED_CONNECTION_CHECK_ERROR,
+};
+enum ethtool_c33_pse_ext_substate_option_vport_lim {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OPTION_VPORT_LIM_HIGH_VOLTAGE = 1,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OPTION_VPORT_LIM_LOW_VOLTAGE,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OPTION_VPORT_LIM_VOLTAGE_INJECTION,
+};
+enum ethtool_c33_pse_ext_substate_ovld_detected {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_OVLD_DETECTED_OVERLOAD = 1,
+};
+enum ethtool_c33_pse_ext_substate_power_not_available {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_POWER_NOT_AVAILABLE_BUDGET_EXCEEDED = 1,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_POWER_NOT_AVAILABLE_PORT_PW_LIMIT_EXCEEDS_CONTROLLER_BUDGET,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_POWER_NOT_AVAILABLE_PD_REQUEST_EXCEEDS_PORT_LIMIT,
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_POWER_NOT_AVAILABLE_HW_PW_LIMIT,
+};
+enum ethtool_c33_pse_ext_substate_short_detected {
+  ETHTOOL_C33_PSE_EXT_SUBSTATE_SHORT_DETECTED_SHORT_CONDITION = 1,
+};
 enum ethtool_pse_types {
   ETHTOOL_PSE_UNKNOWN = 1 << 0,
   ETHTOOL_PSE_PODL = 1 << 1,
@@ -311,6 +361,12 @@
   ETHTOOL_MM_VERIFY_STATUS_FAILED,
   ETHTOOL_MM_VERIFY_STATUS_DISABLED,
 };
+enum ethtool_module_fw_flash_status {
+  ETHTOOL_MODULE_FW_FLASH_STATUS_STARTED = 1,
+  ETHTOOL_MODULE_FW_FLASH_STATUS_IN_PROGRESS,
+  ETHTOOL_MODULE_FW_FLASH_STATUS_COMPLETED,
+  ETHTOOL_MODULE_FW_FLASH_STATUS_ERROR,
+};
 struct ethtool_gstrings {
   __u32 cmd;
   __u32 string_set;
@@ -749,6 +805,7 @@
   ETHTOOL_LINK_MODE_10baseT1S_Full_BIT = 99,
   ETHTOOL_LINK_MODE_10baseT1S_Half_BIT = 100,
   ETHTOOL_LINK_MODE_10baseT1S_P2MP_Half_BIT = 101,
+  ETHTOOL_LINK_MODE_10baseT1BRR_Full_BIT = 102,
   __ETHTOOL_LINK_MODE_MASK_NBITS
 };
 #define __ETHTOOL_LINK_MODE_LEGACY_MASK(base_name) (1UL << (ETHTOOL_LINK_MODE_ ##base_name ##_BIT))
diff --git a/libc/kernel/uapi/linux/ethtool_netlink.h b/libc/kernel/uapi/linux/ethtool_netlink.h
index 6757ef5..ac6391a 100644
--- a/libc/kernel/uapi/linux/ethtool_netlink.h
+++ b/libc/kernel/uapi/linux/ethtool_netlink.h
@@ -52,6 +52,7 @@
   ETHTOOL_MSG_PLCA_GET_STATUS,
   ETHTOOL_MSG_MM_GET,
   ETHTOOL_MSG_MM_SET,
+  ETHTOOL_MSG_MODULE_FW_FLASH_ACT,
   __ETHTOOL_MSG_USER_CNT,
   ETHTOOL_MSG_USER_MAX = __ETHTOOL_MSG_USER_CNT - 1
 };
@@ -100,6 +101,7 @@
   ETHTOOL_MSG_PLCA_NTF,
   ETHTOOL_MSG_MM_GET_REPLY,
   ETHTOOL_MSG_MM_NTF,
+  ETHTOOL_MSG_MODULE_FW_FLASH_NTF,
   __ETHTOOL_MSG_KERNEL_CNT,
   ETHTOOL_MSG_KERNEL_MAX = __ETHTOOL_MSG_KERNEL_CNT - 1
 };
@@ -316,10 +318,26 @@
   ETHTOOL_A_COALESCE_TX_AGGR_MAX_BYTES,
   ETHTOOL_A_COALESCE_TX_AGGR_MAX_FRAMES,
   ETHTOOL_A_COALESCE_TX_AGGR_TIME_USECS,
+  ETHTOOL_A_COALESCE_RX_PROFILE,
+  ETHTOOL_A_COALESCE_TX_PROFILE,
   __ETHTOOL_A_COALESCE_CNT,
   ETHTOOL_A_COALESCE_MAX = (__ETHTOOL_A_COALESCE_CNT - 1)
 };
 enum {
+  ETHTOOL_A_PROFILE_UNSPEC,
+  ETHTOOL_A_PROFILE_IRQ_MODERATION,
+  __ETHTOOL_A_PROFILE_CNT,
+  ETHTOOL_A_PROFILE_MAX = (__ETHTOOL_A_PROFILE_CNT - 1)
+};
+enum {
+  ETHTOOL_A_IRQ_MODERATION_UNSPEC,
+  ETHTOOL_A_IRQ_MODERATION_USEC,
+  ETHTOOL_A_IRQ_MODERATION_PKTS,
+  ETHTOOL_A_IRQ_MODERATION_COMPS,
+  __ETHTOOL_A_IRQ_MODERATION_CNT,
+  ETHTOOL_A_IRQ_MODERATION_MAX = (__ETHTOOL_A_IRQ_MODERATION_CNT - 1)
+};
+enum {
   ETHTOOL_A_PAUSE_UNSPEC,
   ETHTOOL_A_PAUSE_HEADER,
   ETHTOOL_A_PAUSE_AUTONEG,
@@ -635,6 +653,11 @@
   ETHTOOL_A_MODULE_MAX = (__ETHTOOL_A_MODULE_CNT - 1)
 };
 enum {
+  ETHTOOL_A_C33_PSE_PW_LIMIT_UNSPEC,
+  ETHTOOL_A_C33_PSE_PW_LIMIT_MIN,
+  ETHTOOL_A_C33_PSE_PW_LIMIT_MAX,
+};
+enum {
   ETHTOOL_A_PSE_UNSPEC,
   ETHTOOL_A_PSE_HEADER,
   ETHTOOL_A_PODL_PSE_ADMIN_STATE,
@@ -643,6 +666,12 @@
   ETHTOOL_A_C33_PSE_ADMIN_STATE,
   ETHTOOL_A_C33_PSE_ADMIN_CONTROL,
   ETHTOOL_A_C33_PSE_PW_D_STATUS,
+  ETHTOOL_A_C33_PSE_PW_CLASS,
+  ETHTOOL_A_C33_PSE_ACTUAL_PW,
+  ETHTOOL_A_C33_PSE_EXT_STATE,
+  ETHTOOL_A_C33_PSE_EXT_SUBSTATE,
+  ETHTOOL_A_C33_PSE_AVAIL_PW_LIMIT,
+  ETHTOOL_A_C33_PSE_PW_LIMIT_RANGES,
   __ETHTOOL_A_PSE_CNT,
   ETHTOOL_A_PSE_MAX = (__ETHTOOL_A_PSE_CNT - 1)
 };
@@ -699,6 +728,18 @@
   __ETHTOOL_A_MM_CNT,
   ETHTOOL_A_MM_MAX = (__ETHTOOL_A_MM_CNT - 1)
 };
+enum {
+  ETHTOOL_A_MODULE_FW_FLASH_UNSPEC,
+  ETHTOOL_A_MODULE_FW_FLASH_HEADER,
+  ETHTOOL_A_MODULE_FW_FLASH_FILE_NAME,
+  ETHTOOL_A_MODULE_FW_FLASH_PASSWORD,
+  ETHTOOL_A_MODULE_FW_FLASH_STATUS,
+  ETHTOOL_A_MODULE_FW_FLASH_STATUS_MSG,
+  ETHTOOL_A_MODULE_FW_FLASH_DONE,
+  ETHTOOL_A_MODULE_FW_FLASH_TOTAL,
+  __ETHTOOL_A_MODULE_FW_FLASH_CNT,
+  ETHTOOL_A_MODULE_FW_FLASH_MAX = (__ETHTOOL_A_MODULE_FW_FLASH_CNT - 1)
+};
 #define ETHTOOL_GENL_NAME "ethtool"
 #define ETHTOOL_GENL_VERSION 1
 #define ETHTOOL_MCGRP_MONITOR_NAME "monitor"
diff --git a/libc/kernel/uapi/linux/fs.h b/libc/kernel/uapi/linux/fs.h
index 38a2d9e..adab56f 100644
--- a/libc/kernel/uapi/linux/fs.h
+++ b/libc/kernel/uapi/linux/fs.h
@@ -194,8 +194,10 @@
 #define RWF_NOWAIT (( __kernel_rwf_t) 0x00000008)
 #define RWF_APPEND (( __kernel_rwf_t) 0x00000010)
 #define RWF_NOAPPEND (( __kernel_rwf_t) 0x00000020)
-#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT | RWF_APPEND | RWF_NOAPPEND)
-#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+#define RWF_ATOMIC (( __kernel_rwf_t) 0x00000040)
+#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT | RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC)
+#define PROCFS_IOCTL_MAGIC 'f'
+#define PAGEMAP_SCAN _IOWR(PROCFS_IOCTL_MAGIC, 16, struct pm_scan_arg)
 #define PAGE_IS_WPALLOWED (1 << 0)
 #define PAGE_IS_WRITTEN (1 << 1)
 #define PAGE_IS_FILE (1 << 2)
@@ -225,4 +227,30 @@
   __u64 category_anyof_mask;
   __u64 return_mask;
 };
+#define PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 17, struct procmap_query)
+enum procmap_query_flags {
+  PROCMAP_QUERY_VMA_READABLE = 0x01,
+  PROCMAP_QUERY_VMA_WRITABLE = 0x02,
+  PROCMAP_QUERY_VMA_EXECUTABLE = 0x04,
+  PROCMAP_QUERY_VMA_SHARED = 0x08,
+  PROCMAP_QUERY_COVERING_OR_NEXT_VMA = 0x10,
+  PROCMAP_QUERY_FILE_BACKED_VMA = 0x20,
+};
+struct procmap_query {
+  __u64 size;
+  __u64 query_flags;
+  __u64 query_addr;
+  __u64 vma_start;
+  __u64 vma_end;
+  __u64 vma_flags;
+  __u64 vma_page_size;
+  __u64 vma_offset;
+  __u64 inode;
+  __u32 dev_major;
+  __u32 dev_minor;
+  __u32 vma_name_size;
+  __u32 build_id_size;
+  __u64 vma_name_addr;
+  __u64 build_id_addr;
+};
 #endif
diff --git a/libc/kernel/uapi/linux/if_xdp.h b/libc/kernel/uapi/linux/if_xdp.h
index b7eec87..7201e06 100644
--- a/libc/kernel/uapi/linux/if_xdp.h
+++ b/libc/kernel/uapi/linux/if_xdp.h
@@ -14,6 +14,7 @@
 #define XDP_USE_SG (1 << 4)
 #define XDP_UMEM_UNALIGNED_CHUNK_FLAG (1 << 0)
 #define XDP_UMEM_TX_SW_CSUM (1 << 1)
+#define XDP_UMEM_TX_METADATA_LEN (1 << 2)
 struct sockaddr_xdp {
   __u16 sxdp_family;
   __u16 sxdp_flags;
diff --git a/libc/kernel/uapi/linux/iio/buffer.h b/libc/kernel/uapi/linux/iio/buffer.h
index 45c6f65..7e03a8c 100644
--- a/libc/kernel/uapi/linux/iio/buffer.h
+++ b/libc/kernel/uapi/linux/iio/buffer.h
@@ -6,5 +6,16 @@
  */
 #ifndef _UAPI_IIO_BUFFER_H_
 #define _UAPI_IIO_BUFFER_H_
+#include <linux/types.h>
+#define IIO_BUFFER_DMABUF_CYCLIC (1 << 0)
+#define IIO_BUFFER_DMABUF_SUPPORTED_FLAGS 0x00000001
+struct iio_dmabuf {
+  __u32 fd;
+  __u32 flags;
+  __u64 bytes_used;
+};
 #define IIO_BUFFER_GET_FD_IOCTL _IOWR('i', 0x91, int)
+#define IIO_BUFFER_DMABUF_ATTACH_IOCTL _IOW('i', 0x92, int)
+#define IIO_BUFFER_DMABUF_DETACH_IOCTL _IOW('i', 0x93, int)
+#define IIO_BUFFER_DMABUF_ENQUEUE_IOCTL _IOW('i', 0x94, struct iio_dmabuf)
 #endif
diff --git a/libc/kernel/uapi/linux/in.h b/libc/kernel/uapi/linux/in.h
index 44efdd8..97bf493 100644
--- a/libc/kernel/uapi/linux/in.h
+++ b/libc/kernel/uapi/linux/in.h
@@ -69,6 +69,8 @@
 #define IPPROTO_ETHERNET IPPROTO_ETHERNET
   IPPROTO_RAW = 255,
 #define IPPROTO_RAW IPPROTO_RAW
+  IPPROTO_SMC = 256,
+#define IPPROTO_SMC IPPROTO_SMC
   IPPROTO_MPTCP = 262,
 #define IPPROTO_MPTCP IPPROTO_MPTCP
   IPPROTO_MAX
diff --git a/libc/kernel/uapi/linux/io_uring.h b/libc/kernel/uapi/linux/io_uring.h
index 5505963..6b4f2ea 100644
--- a/libc/kernel/uapi/linux/io_uring.h
+++ b/libc/kernel/uapi/linux/io_uring.h
@@ -176,6 +176,8 @@
   IORING_OP_FUTEX_WAITV,
   IORING_OP_FIXED_FD_INSTALL,
   IORING_OP_FTRUNCATE,
+  IORING_OP_BIND,
+  IORING_OP_LISTEN,
   IORING_OP_LAST,
 };
 #define IORING_URING_CMD_FIXED (1U << 0)
diff --git a/libc/kernel/uapi/linux/iommufd.h b/libc/kernel/uapi/linux/iommufd.h
index 2570628..6f663b4 100644
--- a/libc/kernel/uapi/linux/iommufd.h
+++ b/libc/kernel/uapi/linux/iommufd.h
@@ -12,19 +12,20 @@
 enum {
   IOMMUFD_CMD_BASE = 0x80,
   IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
-  IOMMUFD_CMD_IOAS_ALLOC,
-  IOMMUFD_CMD_IOAS_ALLOW_IOVAS,
-  IOMMUFD_CMD_IOAS_COPY,
-  IOMMUFD_CMD_IOAS_IOVA_RANGES,
-  IOMMUFD_CMD_IOAS_MAP,
-  IOMMUFD_CMD_IOAS_UNMAP,
-  IOMMUFD_CMD_OPTION,
-  IOMMUFD_CMD_VFIO_IOAS,
-  IOMMUFD_CMD_HWPT_ALLOC,
-  IOMMUFD_CMD_GET_HW_INFO,
-  IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING,
-  IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP,
-  IOMMUFD_CMD_HWPT_INVALIDATE,
+  IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+  IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+  IOMMUFD_CMD_IOAS_COPY = 0x83,
+  IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+  IOMMUFD_CMD_IOAS_MAP = 0x85,
+  IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+  IOMMUFD_CMD_OPTION = 0x87,
+  IOMMUFD_CMD_VFIO_IOAS = 0x88,
+  IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+  IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+  IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+  IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+  IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+  IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
 };
 struct iommu_destroy {
   __u32 size;
@@ -122,6 +123,7 @@
 enum iommufd_hwpt_alloc_flags {
   IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
   IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+  IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
 };
 enum iommu_hwpt_vtd_s1_flags {
   IOMMU_VTD_S1_SRE = 1 << 0,
@@ -135,8 +137,8 @@
   __u32 __reserved;
 };
 enum iommu_hwpt_data_type {
-  IOMMU_HWPT_DATA_NONE,
-  IOMMU_HWPT_DATA_VTD_S1,
+  IOMMU_HWPT_DATA_NONE = 0,
+  IOMMU_HWPT_DATA_VTD_S1 = 1,
 };
 struct iommu_hwpt_alloc {
   __u32 size;
@@ -148,6 +150,8 @@
   __u32 data_type;
   __u32 data_len;
   __aligned_u64 data_uptr;
+  __u32 fault_id;
+  __u32 __reserved2;
 };
 #define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
 enum iommu_hw_info_vtd_flags {
@@ -160,8 +164,8 @@
   __aligned_u64 ecap_reg;
 };
 enum iommu_hw_info_type {
-  IOMMU_HW_INFO_TYPE_NONE,
-  IOMMU_HW_INFO_TYPE_INTEL_VTD,
+  IOMMU_HW_INFO_TYPE_NONE = 0,
+  IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
 };
 enum iommufd_hw_capabilities {
   IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
@@ -202,7 +206,7 @@
 };
 #define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
 enum iommu_hwpt_invalidate_data_type {
-  IOMMU_HWPT_INVALIDATE_DATA_VTD_S1,
+  IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
 };
 enum iommu_hwpt_vtd_s1_invalidate_flags {
   IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
@@ -223,4 +227,39 @@
   __u32 __reserved;
 };
 #define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+enum iommu_hwpt_pgfault_flags {
+  IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+  IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+enum iommu_hwpt_pgfault_perm {
+  IOMMU_PGFAULT_PERM_READ = (1 << 0),
+  IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+  IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+  IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+struct iommu_hwpt_pgfault {
+  __u32 flags;
+  __u32 dev_id;
+  __u32 pasid;
+  __u32 grpid;
+  __u32 perm;
+  __u64 addr;
+  __u32 length;
+  __u32 cookie;
+};
+enum iommufd_page_response_code {
+  IOMMUFD_PAGE_RESP_SUCCESS = 0,
+  IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+struct iommu_hwpt_page_response {
+  __u32 cookie;
+  __u32 code;
+};
+struct iommu_fault_alloc {
+  __u32 size;
+  __u32 flags;
+  __u32 out_fault_id;
+  __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
 #endif
diff --git a/libc/kernel/uapi/linux/kfd_ioctl.h b/libc/kernel/uapi/linux/kfd_ioctl.h
index 62c9872..193dd8e 100644
--- a/libc/kernel/uapi/linux/kfd_ioctl.h
+++ b/libc/kernel/uapi/linux/kfd_ioctl.h
@@ -9,7 +9,7 @@
 #include <drm/drm.h>
 #include <linux/ioctl.h>
 #define KFD_IOCTL_MAJOR_VERSION 1
-#define KFD_IOCTL_MINOR_VERSION 15
+#define KFD_IOCTL_MINOR_VERSION 16
 struct kfd_ioctl_get_version_args {
   __u32 major_version;
   __u32 minor_version;
@@ -269,6 +269,7 @@
 #define KFD_IOC_ALLOC_MEM_FLAGS_COHERENT (1 << 26)
 #define KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED (1 << 25)
 #define KFD_IOC_ALLOC_MEM_FLAGS_EXT_COHERENT (1 << 24)
+#define KFD_IOC_ALLOC_MEM_FLAGS_CONTIGUOUS (1 << 23)
 struct kfd_ioctl_alloc_memory_of_gpu_args {
   __u64 va_addr;
   __u64 size;
@@ -465,6 +466,7 @@
 };
 enum kfd_dbg_trap_flags {
   KFD_DBG_TRAP_FLAG_SINGLE_MEM_OP = 1,
+  KFD_DBG_TRAP_FLAG_SINGLE_ALU_OP = 2,
 };
 enum kfd_dbg_trap_exception_code {
   EC_NONE = 0,
diff --git a/libc/kernel/uapi/linux/kfd_sysfs.h b/libc/kernel/uapi/linux/kfd_sysfs.h
index e538cf2..7771582 100644
--- a/libc/kernel/uapi/linux/kfd_sysfs.h
+++ b/libc/kernel/uapi/linux/kfd_sysfs.h
@@ -35,7 +35,8 @@
 #define HSA_CAP_SVMAPI_SUPPORTED 0x08000000
 #define HSA_CAP_FLAGS_COHERENTHOSTACCESS 0x10000000
 #define HSA_CAP_TRAP_DEBUG_FIRMWARE_SUPPORTED 0x20000000
-#define HSA_CAP_RESERVED 0xe00f8000
+#define HSA_CAP_TRAP_DEBUG_PRECISE_ALU_OPERATIONS_SUPPORTED 0x40000000
+#define HSA_CAP_RESERVED 0x800f8000
 #define HSA_DBG_WATCH_ADDR_MASK_LO_BIT_MASK 0x0000000f
 #define HSA_DBG_WATCH_ADDR_MASK_LO_BIT_SHIFT 0
 #define HSA_DBG_WATCH_ADDR_MASK_HI_BIT_MASK 0x000003f0
diff --git a/libc/kernel/uapi/linux/kvm.h b/libc/kernel/uapi/linux/kvm.h
index ffaf5e6..297a09d 100644
--- a/libc/kernel/uapi/linux/kvm.h
+++ b/libc/kernel/uapi/linux/kvm.h
@@ -149,9 +149,10 @@
 #define KVM_INTERNAL_ERROR_DELIVERY_EV 3
 #define KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON 4
 #define KVM_INTERNAL_ERROR_EMULATION_FLAG_INSTRUCTION_BYTES (1ULL << 0)
+#define HINT_UNSAFE_IN_KVM(_symbol) _symbol
 struct kvm_run {
   __u8 request_interrupt_window;
-  __u8 immediate_exit;
+  __u8 HINT_UNSAFE_IN_KVM(immediate_exit);
   __u8 padding1[6];
   __u32 exit_reason;
   __u8 ready_for_interrupt_injection;
@@ -719,6 +720,9 @@
 #define KVM_CAP_MEMORY_ATTRIBUTES 233
 #define KVM_CAP_GUEST_MEMFD 234
 #define KVM_CAP_VM_TYPES 235
+#define KVM_CAP_PRE_FAULT_MEMORY 236
+#define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
+#define KVM_CAP_X86_GUEST_MODE 238
 struct kvm_irq_routing_irqchip {
   __u32 irqchip;
   __u32 pin;
@@ -1109,4 +1113,11 @@
   __u64 flags;
   __u64 reserved[6];
 };
+#define KVM_PRE_FAULT_MEMORY _IOWR(KVMIO, 0xd5, struct kvm_pre_fault_memory)
+struct kvm_pre_fault_memory {
+  __u64 gpa;
+  __u64 size;
+  __u64 flags;
+  __u64 padding[5];
+};
 #endif
diff --git a/libc/kernel/uapi/linux/media/raspberrypi/pisp_be_config.h b/libc/kernel/uapi/linux/media/raspberrypi/pisp_be_config.h
new file mode 100644
index 0000000..2e981ad
--- /dev/null
+++ b/libc/kernel/uapi/linux/media/raspberrypi/pisp_be_config.h
@@ -0,0 +1,418 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef _UAPI_PISP_BE_CONFIG_H_
+#define _UAPI_PISP_BE_CONFIG_H_
+#include <linux/types.h>
+#include "pisp_common.h"
+#define PISP_BACK_END_INPUT_ALIGN 4u
+#define PISP_BACK_END_COMPRESSED_ALIGN 8u
+#define PISP_BACK_END_OUTPUT_MIN_ALIGN 16u
+#define PISP_BACK_END_OUTPUT_MAX_ALIGN 64u
+#define PISP_BACK_END_MIN_TILE_WIDTH 16u
+#define PISP_BACK_END_MIN_TILE_HEIGHT 16u
+#define PISP_BACK_END_NUM_OUTPUTS 2
+#define PISP_BACK_END_HOG_OUTPUT 1
+#define PISP_BACK_END_NUM_TILES 64
+enum pisp_be_bayer_enable {
+  PISP_BE_BAYER_ENABLE_INPUT = 0x000001,
+  PISP_BE_BAYER_ENABLE_DECOMPRESS = 0x000002,
+  PISP_BE_BAYER_ENABLE_DPC = 0x000004,
+  PISP_BE_BAYER_ENABLE_GEQ = 0x000008,
+  PISP_BE_BAYER_ENABLE_TDN_INPUT = 0x000010,
+  PISP_BE_BAYER_ENABLE_TDN_DECOMPRESS = 0x000020,
+  PISP_BE_BAYER_ENABLE_TDN = 0x000040,
+  PISP_BE_BAYER_ENABLE_TDN_COMPRESS = 0x000080,
+  PISP_BE_BAYER_ENABLE_TDN_OUTPUT = 0x000100,
+  PISP_BE_BAYER_ENABLE_SDN = 0x000200,
+  PISP_BE_BAYER_ENABLE_BLC = 0x000400,
+  PISP_BE_BAYER_ENABLE_STITCH_INPUT = 0x000800,
+  PISP_BE_BAYER_ENABLE_STITCH_DECOMPRESS = 0x001000,
+  PISP_BE_BAYER_ENABLE_STITCH = 0x002000,
+  PISP_BE_BAYER_ENABLE_STITCH_COMPRESS = 0x004000,
+  PISP_BE_BAYER_ENABLE_STITCH_OUTPUT = 0x008000,
+  PISP_BE_BAYER_ENABLE_WBG = 0x010000,
+  PISP_BE_BAYER_ENABLE_CDN = 0x020000,
+  PISP_BE_BAYER_ENABLE_LSC = 0x040000,
+  PISP_BE_BAYER_ENABLE_TONEMAP = 0x080000,
+  PISP_BE_BAYER_ENABLE_CAC = 0x100000,
+  PISP_BE_BAYER_ENABLE_DEBIN = 0x200000,
+  PISP_BE_BAYER_ENABLE_DEMOSAIC = 0x400000,
+};
+enum pisp_be_rgb_enable {
+  PISP_BE_RGB_ENABLE_INPUT = 0x000001,
+  PISP_BE_RGB_ENABLE_CCM = 0x000002,
+  PISP_BE_RGB_ENABLE_SAT_CONTROL = 0x000004,
+  PISP_BE_RGB_ENABLE_YCBCR = 0x000008,
+  PISP_BE_RGB_ENABLE_FALSE_COLOUR = 0x000010,
+  PISP_BE_RGB_ENABLE_SHARPEN = 0x000020,
+  PISP_BE_RGB_ENABLE_YCBCR_INVERSE = 0x000080,
+  PISP_BE_RGB_ENABLE_GAMMA = 0x000100,
+  PISP_BE_RGB_ENABLE_CSC0 = 0x000200,
+  PISP_BE_RGB_ENABLE_CSC1 = 0x000400,
+  PISP_BE_RGB_ENABLE_DOWNSCALE0 = 0x001000,
+  PISP_BE_RGB_ENABLE_DOWNSCALE1 = 0x002000,
+  PISP_BE_RGB_ENABLE_RESAMPLE0 = 0x008000,
+  PISP_BE_RGB_ENABLE_RESAMPLE1 = 0x010000,
+  PISP_BE_RGB_ENABLE_OUTPUT0 = 0x040000,
+  PISP_BE_RGB_ENABLE_OUTPUT1 = 0x080000,
+  PISP_BE_RGB_ENABLE_HOG = 0x200000
+};
+#define PISP_BE_RGB_ENABLE_CSC(i) (PISP_BE_RGB_ENABLE_CSC0 << (i))
+#define PISP_BE_RGB_ENABLE_DOWNSCALE(i) (PISP_BE_RGB_ENABLE_DOWNSCALE0 << (i))
+#define PISP_BE_RGB_ENABLE_RESAMPLE(i) (PISP_BE_RGB_ENABLE_RESAMPLE0 << (i))
+#define PISP_BE_RGB_ENABLE_OUTPUT(i) (PISP_BE_RGB_ENABLE_OUTPUT0 << (i))
+enum pisp_be_dirty {
+  PISP_BE_DIRTY_GLOBAL = 0x0001,
+  PISP_BE_DIRTY_SH_FC_COMBINE = 0x0002,
+  PISP_BE_DIRTY_CROP = 0x0004
+};
+struct pisp_be_global_config {
+  __u32 bayer_enables;
+  __u32 rgb_enables;
+  __u8 bayer_order;
+  __u8 pad[3];
+} __attribute__((packed));
+struct pisp_be_input_buffer_config {
+  __u32 addr[3][2];
+} __attribute__((packed));
+struct pisp_be_dpc_config {
+  __u8 coeff_level;
+  __u8 coeff_range;
+  __u8 pad;
+#define PISP_BE_DPC_FLAG_FOLDBACK 1
+  __u8 flags;
+} __attribute__((packed));
+struct pisp_be_geq_config {
+  __u16 offset;
+#define PISP_BE_GEQ_SHARPER (1U << 15)
+#define PISP_BE_GEQ_SLOPE ((1 << 10) - 1)
+  __u16 slope_sharper;
+  __u16 min;
+  __u16 max;
+} __attribute__((packed));
+struct pisp_be_tdn_input_buffer_config {
+  __u32 addr[2];
+} __attribute__((packed));
+struct pisp_be_tdn_config {
+  __u16 black_level;
+  __u16 ratio;
+  __u16 noise_constant;
+  __u16 noise_slope;
+  __u16 threshold;
+  __u8 reset;
+  __u8 pad;
+} __attribute__((packed));
+struct pisp_be_tdn_output_buffer_config {
+  __u32 addr[2];
+} __attribute__((packed));
+struct pisp_be_sdn_config {
+  __u16 black_level;
+  __u8 leakage;
+  __u8 pad;
+  __u16 noise_constant;
+  __u16 noise_slope;
+  __u16 noise_constant2;
+  __u16 noise_slope2;
+} __attribute__((packed));
+struct pisp_be_stitch_input_buffer_config {
+  __u32 addr[2];
+} __attribute__((packed));
+#define PISP_BE_STITCH_STREAMING_LONG 0x8000
+#define PISP_BE_STITCH_EXPOSURE_RATIO_MASK 0x7fff
+struct pisp_be_stitch_config {
+  __u16 threshold_lo;
+  __u8 threshold_diff_power;
+  __u8 pad;
+  __u16 exposure_ratio;
+  __u8 motion_threshold_256;
+  __u8 motion_threshold_recip;
+} __attribute__((packed));
+struct pisp_be_stitch_output_buffer_config {
+  __u32 addr[2];
+} __attribute__((packed));
+struct pisp_be_cdn_config {
+  __u16 thresh;
+  __u8 iir_strength;
+  __u8 g_adjust;
+} __attribute__((packed));
+#define PISP_BE_LSC_LOG_GRID_SIZE 5
+#define PISP_BE_LSC_GRID_SIZE (1 << PISP_BE_LSC_LOG_GRID_SIZE)
+#define PISP_BE_LSC_STEP_PRECISION 18
+struct pisp_be_lsc_config {
+  __u16 grid_step_x;
+  __u16 grid_step_y;
+#define PISP_BE_LSC_LUT_SIZE (PISP_BE_LSC_GRID_SIZE + 1)
+  __u32 lut_packed[PISP_BE_LSC_LUT_SIZE][PISP_BE_LSC_LUT_SIZE];
+} __attribute__((packed));
+struct pisp_be_lsc_extra {
+  __u16 offset_x;
+  __u16 offset_y;
+} __attribute__((packed));
+#define PISP_BE_CAC_LOG_GRID_SIZE 3
+#define PISP_BE_CAC_GRID_SIZE (1 << PISP_BE_CAC_LOG_GRID_SIZE)
+#define PISP_BE_CAC_STEP_PRECISION 20
+struct pisp_be_cac_config {
+  __u16 grid_step_x;
+  __u16 grid_step_y;
+#define PISP_BE_CAC_LUT_SIZE (PISP_BE_CAC_GRID_SIZE + 1)
+  __s8 lut[PISP_BE_CAC_LUT_SIZE][PISP_BE_CAC_LUT_SIZE][2][2];
+} __attribute__((packed));
+struct pisp_be_cac_extra {
+  __u16 offset_x;
+  __u16 offset_y;
+} __attribute__((packed));
+#define PISP_BE_DEBIN_NUM_COEFFS 4
+struct pisp_be_debin_config {
+  __s8 coeffs[PISP_BE_DEBIN_NUM_COEFFS];
+  __s8 h_enable;
+  __s8 v_enable;
+  __s8 pad[2];
+} __attribute__((packed));
+#define PISP_BE_TONEMAP_LUT_SIZE 64
+struct pisp_be_tonemap_config {
+  __u16 detail_constant;
+  __u16 detail_slope;
+  __u16 iir_strength;
+  __u16 strength;
+  __u32 lut[PISP_BE_TONEMAP_LUT_SIZE];
+} __attribute__((packed));
+struct pisp_be_demosaic_config {
+  __u8 sharper;
+  __u8 fc_mode;
+  __u8 pad[2];
+} __attribute__((packed));
+struct pisp_be_ccm_config {
+  __s16 coeffs[9];
+  __u8 pad[2];
+  __s32 offsets[3];
+} __attribute__((packed));
+struct pisp_be_sat_control_config {
+  __u8 shift_r;
+  __u8 shift_g;
+  __u8 shift_b;
+  __u8 pad;
+} __attribute__((packed));
+struct pisp_be_false_colour_config {
+  __u8 distance;
+  __u8 pad[3];
+} __attribute__((packed));
+#define PISP_BE_SHARPEN_SIZE 5
+#define PISP_BE_SHARPEN_FUNC_NUM_POINTS 9
+struct pisp_be_sharpen_config {
+  __s8 kernel0[PISP_BE_SHARPEN_SIZE * PISP_BE_SHARPEN_SIZE];
+  __s8 pad0[3];
+  __s8 kernel1[PISP_BE_SHARPEN_SIZE * PISP_BE_SHARPEN_SIZE];
+  __s8 pad1[3];
+  __s8 kernel2[PISP_BE_SHARPEN_SIZE * PISP_BE_SHARPEN_SIZE];
+  __s8 pad2[3];
+  __s8 kernel3[PISP_BE_SHARPEN_SIZE * PISP_BE_SHARPEN_SIZE];
+  __s8 pad3[3];
+  __s8 kernel4[PISP_BE_SHARPEN_SIZE * PISP_BE_SHARPEN_SIZE];
+  __s8 pad4[3];
+  __u16 threshold_offset0;
+  __u16 threshold_slope0;
+  __u16 scale0;
+  __u16 pad5;
+  __u16 threshold_offset1;
+  __u16 threshold_slope1;
+  __u16 scale1;
+  __u16 pad6;
+  __u16 threshold_offset2;
+  __u16 threshold_slope2;
+  __u16 scale2;
+  __u16 pad7;
+  __u16 threshold_offset3;
+  __u16 threshold_slope3;
+  __u16 scale3;
+  __u16 pad8;
+  __u16 threshold_offset4;
+  __u16 threshold_slope4;
+  __u16 scale4;
+  __u16 pad9;
+  __u16 positive_strength;
+  __u16 positive_pre_limit;
+  __u16 positive_func[PISP_BE_SHARPEN_FUNC_NUM_POINTS];
+  __u16 positive_limit;
+  __u16 negative_strength;
+  __u16 negative_pre_limit;
+  __u16 negative_func[PISP_BE_SHARPEN_FUNC_NUM_POINTS];
+  __u16 negative_limit;
+  __u8 enables;
+  __u8 white;
+  __u8 black;
+  __u8 grey;
+} __attribute__((packed));
+struct pisp_be_sh_fc_combine_config {
+  __u8 y_factor;
+  __u8 c1_factor;
+  __u8 c2_factor;
+  __u8 pad;
+} __attribute__((packed));
+#define PISP_BE_GAMMA_LUT_SIZE 64
+struct pisp_be_gamma_config {
+  __u32 lut[PISP_BE_GAMMA_LUT_SIZE];
+} __attribute__((packed));
+struct pisp_be_crop_config {
+  __u16 offset_x, offset_y;
+  __u16 width, height;
+} __attribute__((packed));
+#define PISP_BE_RESAMPLE_FILTER_SIZE 96
+struct pisp_be_resample_config {
+  __u16 scale_factor_h, scale_factor_v;
+  __s16 coef[PISP_BE_RESAMPLE_FILTER_SIZE];
+} __attribute__((packed));
+struct pisp_be_resample_extra {
+  __u16 scaled_width;
+  __u16 scaled_height;
+  __s16 initial_phase_h[3];
+  __s16 initial_phase_v[3];
+} __attribute__((packed));
+struct pisp_be_downscale_config {
+  __u16 scale_factor_h;
+  __u16 scale_factor_v;
+  __u16 scale_recip_h;
+  __u16 scale_recip_v;
+} __attribute__((packed));
+struct pisp_be_downscale_extra {
+  __u16 scaled_width;
+  __u16 scaled_height;
+} __attribute__((packed));
+struct pisp_be_hog_config {
+  __u8 compute_signed;
+  __u8 channel_mix[3];
+  __u32 stride;
+} __attribute__((packed));
+struct pisp_be_axi_config {
+  __u8 r_qos;
+  __u8 r_cache_prot;
+  __u8 w_qos;
+  __u8 w_cache_prot;
+} __attribute__((packed));
+enum pisp_be_transform {
+  PISP_BE_TRANSFORM_NONE = 0x0,
+  PISP_BE_TRANSFORM_HFLIP = 0x1,
+  PISP_BE_TRANSFORM_VFLIP = 0x2,
+  PISP_BE_TRANSFORM_ROT180 = (PISP_BE_TRANSFORM_HFLIP | PISP_BE_TRANSFORM_VFLIP)
+};
+struct pisp_be_output_format_config {
+  struct pisp_image_format_config image;
+  __u8 transform;
+  __u8 pad[3];
+  __u16 lo;
+  __u16 hi;
+  __u16 lo2;
+  __u16 hi2;
+} __attribute__((packed));
+struct pisp_be_output_buffer_config {
+  __u32 addr[3][2];
+} __attribute__((packed));
+struct pisp_be_hog_buffer_config {
+  __u32 addr[2];
+} __attribute__((packed));
+struct pisp_be_config {
+  struct pisp_be_input_buffer_config input_buffer;
+  struct pisp_be_tdn_input_buffer_config tdn_input_buffer;
+  struct pisp_be_stitch_input_buffer_config stitch_input_buffer;
+  struct pisp_be_tdn_output_buffer_config tdn_output_buffer;
+  struct pisp_be_stitch_output_buffer_config stitch_output_buffer;
+  struct pisp_be_output_buffer_config output_buffer[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_hog_buffer_config hog_buffer;
+  struct pisp_be_global_config global;
+  struct pisp_image_format_config input_format;
+  struct pisp_decompress_config decompress;
+  struct pisp_be_dpc_config dpc;
+  struct pisp_be_geq_config geq;
+  struct pisp_image_format_config tdn_input_format;
+  struct pisp_decompress_config tdn_decompress;
+  struct pisp_be_tdn_config tdn;
+  struct pisp_compress_config tdn_compress;
+  struct pisp_image_format_config tdn_output_format;
+  struct pisp_be_sdn_config sdn;
+  struct pisp_bla_config blc;
+  struct pisp_compress_config stitch_compress;
+  struct pisp_image_format_config stitch_output_format;
+  struct pisp_image_format_config stitch_input_format;
+  struct pisp_decompress_config stitch_decompress;
+  struct pisp_be_stitch_config stitch;
+  struct pisp_be_lsc_config lsc;
+  struct pisp_wbg_config wbg;
+  struct pisp_be_cdn_config cdn;
+  struct pisp_be_cac_config cac;
+  struct pisp_be_debin_config debin;
+  struct pisp_be_tonemap_config tonemap;
+  struct pisp_be_demosaic_config demosaic;
+  struct pisp_be_ccm_config ccm;
+  struct pisp_be_sat_control_config sat_control;
+  struct pisp_be_ccm_config ycbcr;
+  struct pisp_be_sharpen_config sharpen;
+  struct pisp_be_false_colour_config false_colour;
+  struct pisp_be_sh_fc_combine_config sh_fc_combine;
+  struct pisp_be_ccm_config ycbcr_inverse;
+  struct pisp_be_gamma_config gamma;
+  struct pisp_be_ccm_config csc[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_downscale_config downscale[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_resample_config resample[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_output_format_config output_format[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_hog_config hog;
+  struct pisp_be_axi_config axi;
+  struct pisp_be_lsc_extra lsc_extra;
+  struct pisp_be_cac_extra cac_extra;
+  struct pisp_be_downscale_extra downscale_extra[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_resample_extra resample_extra[PISP_BACK_END_NUM_OUTPUTS];
+  struct pisp_be_crop_config crop;
+  struct pisp_image_format_config hog_format;
+  __u32 dirty_flags_bayer;
+  __u32 dirty_flags_rgb;
+  __u32 dirty_flags_extra;
+} __attribute__((packed));
+enum pisp_tile_edge {
+  PISP_LEFT_EDGE = (1 << 0),
+  PISP_RIGHT_EDGE = (1 << 1),
+  PISP_TOP_EDGE = (1 << 2),
+  PISP_BOTTOM_EDGE = (1 << 3)
+};
+struct pisp_tile {
+  __u8 edge;
+  __u8 pad0[3];
+  __u32 input_addr_offset;
+  __u32 input_addr_offset2;
+  __u16 input_offset_x;
+  __u16 input_offset_y;
+  __u16 input_width;
+  __u16 input_height;
+  __u32 tdn_input_addr_offset;
+  __u32 tdn_output_addr_offset;
+  __u32 stitch_input_addr_offset;
+  __u32 stitch_output_addr_offset;
+  __u32 lsc_grid_offset_x;
+  __u32 lsc_grid_offset_y;
+  __u32 cac_grid_offset_x;
+  __u32 cac_grid_offset_y;
+  __u16 crop_x_start[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 crop_x_end[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 crop_y_start[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 crop_y_end[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 downscale_phase_x[3 * PISP_BACK_END_NUM_OUTPUTS];
+  __u16 downscale_phase_y[3 * PISP_BACK_END_NUM_OUTPUTS];
+  __u16 resample_in_width[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 resample_in_height[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 resample_phase_x[3 * PISP_BACK_END_NUM_OUTPUTS];
+  __u16 resample_phase_y[3 * PISP_BACK_END_NUM_OUTPUTS];
+  __u16 output_offset_x[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 output_offset_y[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 output_width[PISP_BACK_END_NUM_OUTPUTS];
+  __u16 output_height[PISP_BACK_END_NUM_OUTPUTS];
+  __u32 output_addr_offset[PISP_BACK_END_NUM_OUTPUTS];
+  __u32 output_addr_offset2[PISP_BACK_END_NUM_OUTPUTS];
+  __u32 output_hog_addr_offset;
+} __attribute__((packed));
+struct pisp_be_tiles_config {
+  struct pisp_be_config config;
+  struct pisp_tile tiles[PISP_BACK_END_NUM_TILES];
+  __u32 num_tiles;
+} __attribute__((packed));
+#endif
diff --git a/libc/kernel/uapi/linux/media/raspberrypi/pisp_common.h b/libc/kernel/uapi/linux/media/raspberrypi/pisp_common.h
new file mode 100644
index 0000000..0e0b23f
--- /dev/null
+++ b/libc/kernel/uapi/linux/media/raspberrypi/pisp_common.h
@@ -0,0 +1,119 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef _UAPI_PISP_COMMON_H_
+#define _UAPI_PISP_COMMON_H_
+#include <linux/types.h>
+struct pisp_image_format_config {
+  __u16 width;
+  __u16 height;
+  __u32 format;
+  __s32 stride;
+  __s32 stride2;
+} __attribute__((packed));
+enum pisp_bayer_order {
+  PISP_BAYER_ORDER_RGGB = 0,
+  PISP_BAYER_ORDER_GBRG = 1,
+  PISP_BAYER_ORDER_BGGR = 2,
+  PISP_BAYER_ORDER_GRBG = 3,
+  PISP_BAYER_ORDER_GREYSCALE = 128
+};
+enum pisp_image_format {
+  PISP_IMAGE_FORMAT_BPS_8 = 0x00000000,
+  PISP_IMAGE_FORMAT_BPS_10 = 0x00000001,
+  PISP_IMAGE_FORMAT_BPS_12 = 0x00000002,
+  PISP_IMAGE_FORMAT_BPS_16 = 0x00000003,
+  PISP_IMAGE_FORMAT_BPS_MASK = 0x00000003,
+  PISP_IMAGE_FORMAT_PLANARITY_INTERLEAVED = 0x00000000,
+  PISP_IMAGE_FORMAT_PLANARITY_SEMI_PLANAR = 0x00000010,
+  PISP_IMAGE_FORMAT_PLANARITY_PLANAR = 0x00000020,
+  PISP_IMAGE_FORMAT_PLANARITY_MASK = 0x00000030,
+  PISP_IMAGE_FORMAT_SAMPLING_444 = 0x00000000,
+  PISP_IMAGE_FORMAT_SAMPLING_422 = 0x00000100,
+  PISP_IMAGE_FORMAT_SAMPLING_420 = 0x00000200,
+  PISP_IMAGE_FORMAT_SAMPLING_MASK = 0x00000300,
+  PISP_IMAGE_FORMAT_ORDER_NORMAL = 0x00000000,
+  PISP_IMAGE_FORMAT_ORDER_SWAPPED = 0x00001000,
+  PISP_IMAGE_FORMAT_SHIFT_0 = 0x00000000,
+  PISP_IMAGE_FORMAT_SHIFT_1 = 0x00010000,
+  PISP_IMAGE_FORMAT_SHIFT_2 = 0x00020000,
+  PISP_IMAGE_FORMAT_SHIFT_3 = 0x00030000,
+  PISP_IMAGE_FORMAT_SHIFT_4 = 0x00040000,
+  PISP_IMAGE_FORMAT_SHIFT_5 = 0x00050000,
+  PISP_IMAGE_FORMAT_SHIFT_6 = 0x00060000,
+  PISP_IMAGE_FORMAT_SHIFT_7 = 0x00070000,
+  PISP_IMAGE_FORMAT_SHIFT_8 = 0x00080000,
+  PISP_IMAGE_FORMAT_SHIFT_MASK = 0x000f0000,
+  PISP_IMAGE_FORMAT_BPP_32 = 0x00100000,
+  PISP_IMAGE_FORMAT_UNCOMPRESSED = 0x00000000,
+  PISP_IMAGE_FORMAT_COMPRESSION_MODE_1 = 0x01000000,
+  PISP_IMAGE_FORMAT_COMPRESSION_MODE_2 = 0x02000000,
+  PISP_IMAGE_FORMAT_COMPRESSION_MODE_3 = 0x03000000,
+  PISP_IMAGE_FORMAT_COMPRESSION_MASK = 0x03000000,
+  PISP_IMAGE_FORMAT_HOG_SIGNED = 0x04000000,
+  PISP_IMAGE_FORMAT_HOG_UNSIGNED = 0x08000000,
+  PISP_IMAGE_FORMAT_INTEGRAL_IMAGE = 0x10000000,
+  PISP_IMAGE_FORMAT_WALLPAPER_ROLL = 0x20000000,
+  PISP_IMAGE_FORMAT_THREE_CHANNEL = 0x40000000,
+  PISP_IMAGE_FORMAT_SINGLE_16 = PISP_IMAGE_FORMAT_BPS_16,
+  PISP_IMAGE_FORMAT_THREE_16 = PISP_IMAGE_FORMAT_BPS_16 | PISP_IMAGE_FORMAT_THREE_CHANNEL
+};
+#define PISP_IMAGE_FORMAT_BPS_8(fmt) (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) == PISP_IMAGE_FORMAT_BPS_8)
+#define PISP_IMAGE_FORMAT_BPS_10(fmt) (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) == PISP_IMAGE_FORMAT_BPS_10)
+#define PISP_IMAGE_FORMAT_BPS_12(fmt) (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) == PISP_IMAGE_FORMAT_BPS_12)
+#define PISP_IMAGE_FORMAT_BPS_16(fmt) (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) == PISP_IMAGE_FORMAT_BPS_16)
+#define PISP_IMAGE_FORMAT_BPS(fmt) (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) ? 8 + (2 << (((fmt) & PISP_IMAGE_FORMAT_BPS_MASK) - 1)) : 8)
+#define PISP_IMAGE_FORMAT_SHIFT(fmt) (((fmt) & PISP_IMAGE_FORMAT_SHIFT_MASK) / PISP_IMAGE_FORMAT_SHIFT_1)
+#define PISP_IMAGE_FORMAT_THREE_CHANNEL(fmt) ((fmt) & PISP_IMAGE_FORMAT_THREE_CHANNEL)
+#define PISP_IMAGE_FORMAT_SINGLE_CHANNEL(fmt) (! ((fmt) & PISP_IMAGE_FORMAT_THREE_CHANNEL))
+#define PISP_IMAGE_FORMAT_COMPRESSED(fmt) (((fmt) & PISP_IMAGE_FORMAT_COMPRESSION_MASK) != PISP_IMAGE_FORMAT_UNCOMPRESSED)
+#define PISP_IMAGE_FORMAT_SAMPLING_444(fmt) (((fmt) & PISP_IMAGE_FORMAT_SAMPLING_MASK) == PISP_IMAGE_FORMAT_SAMPLING_444)
+#define PISP_IMAGE_FORMAT_SAMPLING_422(fmt) (((fmt) & PISP_IMAGE_FORMAT_SAMPLING_MASK) == PISP_IMAGE_FORMAT_SAMPLING_422)
+#define PISP_IMAGE_FORMAT_SAMPLING_420(fmt) (((fmt) & PISP_IMAGE_FORMAT_SAMPLING_MASK) == PISP_IMAGE_FORMAT_SAMPLING_420)
+#define PISP_IMAGE_FORMAT_ORDER_NORMAL(fmt) (! ((fmt) & PISP_IMAGE_FORMAT_ORDER_SWAPPED))
+#define PISP_IMAGE_FORMAT_ORDER_SWAPPED(fmt) ((fmt) & PISP_IMAGE_FORMAT_ORDER_SWAPPED)
+#define PISP_IMAGE_FORMAT_INTERLEAVED(fmt) (((fmt) & PISP_IMAGE_FORMAT_PLANARITY_MASK) == PISP_IMAGE_FORMAT_PLANARITY_INTERLEAVED)
+#define PISP_IMAGE_FORMAT_SEMIPLANAR(fmt) (((fmt) & PISP_IMAGE_FORMAT_PLANARITY_MASK) == PISP_IMAGE_FORMAT_PLANARITY_SEMI_PLANAR)
+#define PISP_IMAGE_FORMAT_PLANAR(fmt) (((fmt) & PISP_IMAGE_FORMAT_PLANARITY_MASK) == PISP_IMAGE_FORMAT_PLANARITY_PLANAR)
+#define PISP_IMAGE_FORMAT_WALLPAPER(fmt) ((fmt) & PISP_IMAGE_FORMAT_WALLPAPER_ROLL)
+#define PISP_IMAGE_FORMAT_BPP_32(fmt) ((fmt) & PISP_IMAGE_FORMAT_BPP_32)
+#define PISP_IMAGE_FORMAT_HOG(fmt) ((fmt) & (PISP_IMAGE_FORMAT_HOG_SIGNED | PISP_IMAGE_FORMAT_HOG_UNSIGNED))
+#define PISP_WALLPAPER_WIDTH 128
+struct pisp_bla_config {
+  __u16 black_level_r;
+  __u16 black_level_gr;
+  __u16 black_level_gb;
+  __u16 black_level_b;
+  __u16 output_black_level;
+  __u8 pad[2];
+} __attribute__((packed));
+struct pisp_wbg_config {
+  __u16 gain_r;
+  __u16 gain_g;
+  __u16 gain_b;
+  __u8 pad[2];
+} __attribute__((packed));
+struct pisp_compress_config {
+  __u16 offset;
+  __u8 pad;
+  __u8 mode;
+} __attribute__((packed));
+struct pisp_decompress_config {
+  __u16 offset;
+  __u8 pad;
+  __u8 mode;
+} __attribute__((packed));
+enum pisp_axi_flags {
+  PISP_AXI_FLAG_ALIGN = 128,
+  PISP_AXI_FLAG_PAD = 64,
+  PISP_AXI_FLAG_PANIC = 32,
+};
+struct pisp_axi_config {
+  __u8 maxlen_flags;
+  __u8 cache_prot;
+  __u16 qos;
+} __attribute__((packed));
+#endif
diff --git a/libc/kernel/uapi/linux/mman.h b/libc/kernel/uapi/linux/mman.h
index cf1e978..f50b51c 100644
--- a/libc/kernel/uapi/linux/mman.h
+++ b/libc/kernel/uapi/linux/mman.h
@@ -18,6 +18,7 @@
 #define MAP_SHARED 0x01
 #define MAP_PRIVATE 0x02
 #define MAP_SHARED_VALIDATE 0x03
+#define MAP_DROPPABLE 0x08
 #define MAP_HUGE_SHIFT HUGETLB_FLAG_ENCODE_SHIFT
 #define MAP_HUGE_MASK HUGETLB_FLAG_ENCODE_MASK
 #define MAP_HUGE_16KB HUGETLB_FLAG_ENCODE_16KB
diff --git a/libc/kernel/uapi/linux/mount.h b/libc/kernel/uapi/linux/mount.h
index 1166a7a..c4278b5 100644
--- a/libc/kernel/uapi/linux/mount.h
+++ b/libc/kernel/uapi/linux/mount.h
@@ -90,7 +90,7 @@
 #define MOUNT_ATTR_SIZE_VER0 32
 struct statmount {
   __u32 size;
-  __u32 __spare1;
+  __u32 mnt_opts;
   __u64 mask;
   __u32 sb_dev_major;
   __u32 sb_dev_minor;
@@ -108,7 +108,8 @@
   __u64 propagate_from;
   __u32 mnt_root;
   __u32 mnt_point;
-  __u64 __spare2[50];
+  __u64 mnt_ns_id;
+  __u64 __spare2[49];
   char str[];
 };
 struct mnt_id_req {
@@ -116,13 +117,18 @@
   __u32 spare;
   __u64 mnt_id;
   __u64 param;
+  __u64 mnt_ns_id;
 };
 #define MNT_ID_REQ_SIZE_VER0 24
+#define MNT_ID_REQ_SIZE_VER1 32
 #define STATMOUNT_SB_BASIC 0x00000001U
 #define STATMOUNT_MNT_BASIC 0x00000002U
 #define STATMOUNT_PROPAGATE_FROM 0x00000004U
 #define STATMOUNT_MNT_ROOT 0x00000008U
 #define STATMOUNT_MNT_POINT 0x00000010U
 #define STATMOUNT_FS_TYPE 0x00000020U
+#define STATMOUNT_MNT_NS_ID 0x00000040U
+#define STATMOUNT_MNT_OPTS 0x00000080U
 #define LSMT_ROOT 0xffffffffffffffff
+#define LISTMOUNT_REVERSE (1 << 0)
 #endif
diff --git a/libc/kernel/uapi/linux/netfilter/nf_tables.h b/libc/kernel/uapi/linux/netfilter/nf_tables.h
index 7922147..bfc6e25 100644
--- a/libc/kernel/uapi/linux/netfilter/nf_tables.h
+++ b/libc/kernel/uapi/linux/netfilter/nf_tables.h
@@ -695,7 +695,7 @@
   __NFTA_SECMARK_MAX,
 };
 #define NFTA_SECMARK_MAX (__NFTA_SECMARK_MAX - 1)
-#define NFT_SECMARK_CTX_MAXLEN 256
+#define NFT_SECMARK_CTX_MAXLEN 4096
 enum nft_reject_types {
   NFT_REJECT_ICMP_UNREACH,
   NFT_REJECT_TCP_RST,
diff --git a/libc/kernel/uapi/linux/nfs4.h b/libc/kernel/uapi/linux/nfs4.h
index 21f1103..6512901 100644
--- a/libc/kernel/uapi/linux/nfs4.h
+++ b/libc/kernel/uapi/linux/nfs4.h
@@ -34,6 +34,7 @@
 #define NFS4_OPEN_RESULT_CONFIRM 0x0002
 #define NFS4_OPEN_RESULT_LOCKTYPE_POSIX 0x0004
 #define NFS4_OPEN_RESULT_PRESERVE_UNLINKED 0x0008
+#define NFS4_OPEN_RESULT_NO_OPEN_STATEID 0x0010
 #define NFS4_OPEN_RESULT_MAY_NOTIFY_LOCK 0x0020
 #define NFS4_SHARE_ACCESS_MASK 0x000F
 #define NFS4_SHARE_ACCESS_READ 0x0001
@@ -52,6 +53,8 @@
 #define NFS4_SHARE_WHEN_MASK 0xF0000
 #define NFS4_SHARE_SIGNAL_DELEG_WHEN_RESRC_AVAIL 0x10000
 #define NFS4_SHARE_PUSH_DELEG_WHEN_UNCONTENDED 0x20000
+#define NFS4_SHARE_WANT_DELEG_TIMESTAMPS 0x100000
+#define NFS4_SHARE_WANT_OPEN_XOR_DELEGATION 0x200000
 #define NFS4_CDFC4_FORE 0x1
 #define NFS4_CDFC4_BACK 0x2
 #define NFS4_CDFC4_BOTH 0x3
diff --git a/libc/kernel/uapi/linux/nfsd_netlink.h b/libc/kernel/uapi/linux/nfsd_netlink.h
index 45cb504..bd3f02c 100644
--- a/libc/kernel/uapi/linux/nfsd_netlink.h
+++ b/libc/kernel/uapi/linux/nfsd_netlink.h
@@ -58,6 +58,12 @@
   NFSD_A_SERVER_SOCK_MAX = (__NFSD_A_SERVER_SOCK_MAX - 1)
 };
 enum {
+  NFSD_A_POOL_MODE_MODE = 1,
+  NFSD_A_POOL_MODE_NPOOLS,
+  __NFSD_A_POOL_MODE_MAX,
+  NFSD_A_POOL_MODE_MAX = (__NFSD_A_POOL_MODE_MAX - 1)
+};
+enum {
   NFSD_CMD_RPC_STATUS_GET = 1,
   NFSD_CMD_THREADS_SET,
   NFSD_CMD_THREADS_GET,
@@ -65,6 +71,8 @@
   NFSD_CMD_VERSION_GET,
   NFSD_CMD_LISTENER_SET,
   NFSD_CMD_LISTENER_GET,
+  NFSD_CMD_POOL_MODE_SET,
+  NFSD_CMD_POOL_MODE_GET,
   __NFSD_CMD_MAX,
   NFSD_CMD_MAX = (__NFSD_CMD_MAX - 1)
 };
diff --git a/libc/kernel/uapi/linux/nl80211.h b/libc/kernel/uapi/linux/nl80211.h
index 98180c2..1bad2f2 100644
--- a/libc/kernel/uapi/linux/nl80211.h
+++ b/libc/kernel/uapi/linux/nl80211.h
@@ -528,6 +528,8 @@
   NL80211_ATTR_MLO_TTLM_DLINK,
   NL80211_ATTR_MLO_TTLM_ULINK,
   NL80211_ATTR_ASSOC_SPP_AMSDU,
+  NL80211_ATTR_WIPHY_RADIOS,
+  NL80211_ATTR_WIPHY_INTERFACE_COMBINATIONS,
   __NL80211_ATTR_AFTER_LAST,
   NUM_NL80211_ATTR = __NL80211_ATTR_AFTER_LAST,
   NL80211_ATTR_MAX = __NL80211_ATTR_AFTER_LAST - 1
@@ -878,6 +880,7 @@
   NL80211_FREQUENCY_ATTR_NO_6GHZ_VLP_CLIENT,
   NL80211_FREQUENCY_ATTR_NO_6GHZ_AFC_CLIENT,
   NL80211_FREQUENCY_ATTR_CAN_MONITOR,
+  NL80211_FREQUENCY_ATTR_ALLOW_6GHZ_VLP_AP,
   __NL80211_FREQUENCY_ATTR_AFTER_LAST,
   NL80211_FREQUENCY_ATTR_MAX = __NL80211_FREQUENCY_ATTR_AFTER_LAST - 1
 };
@@ -955,6 +958,7 @@
   NL80211_RRF_DFS_CONCURRENT = 1 << 21,
   NL80211_RRF_NO_6GHZ_VLP_CLIENT = 1 << 22,
   NL80211_RRF_NO_6GHZ_AFC_CLIENT = 1 << 23,
+  NL80211_RRF_ALLOW_6GHZ_VLP_AP = 1 << 24,
 };
 #define NL80211_RRF_PASSIVE_SCAN NL80211_RRF_NO_IR
 #define NL80211_RRF_NO_IBSS NL80211_RRF_NO_IR
@@ -1973,4 +1977,19 @@
   NL80211_AP_SETTINGS_EXTERNAL_AUTH_SUPPORT = 1 << 0,
   NL80211_AP_SETTINGS_SA_QUERY_OFFLOAD_SUPPORT = 1 << 1,
 };
+enum nl80211_wiphy_radio_attrs {
+  __NL80211_WIPHY_RADIO_ATTR_INVALID,
+  NL80211_WIPHY_RADIO_ATTR_INDEX,
+  NL80211_WIPHY_RADIO_ATTR_FREQ_RANGE,
+  NL80211_WIPHY_RADIO_ATTR_INTERFACE_COMBINATION,
+  __NL80211_WIPHY_RADIO_ATTR_LAST,
+  NL80211_WIPHY_RADIO_ATTR_MAX = __NL80211_WIPHY_RADIO_ATTR_LAST - 1,
+};
+enum nl80211_wiphy_radio_freq_range {
+  __NL80211_WIPHY_RADIO_FREQ_ATTR_INVALID,
+  NL80211_WIPHY_RADIO_FREQ_ATTR_START,
+  NL80211_WIPHY_RADIO_FREQ_ATTR_END,
+  __NL80211_WIPHY_RADIO_FREQ_ATTR_LAST,
+  NL80211_WIPHY_RADIO_FREQ_ATTR_MAX = __NL80211_WIPHY_RADIO_FREQ_ATTR_LAST - 1,
+};
 #endif
diff --git a/libc/kernel/uapi/linux/nsfs.h b/libc/kernel/uapi/linux/nsfs.h
index 61a5797..c8f2208 100644
--- a/libc/kernel/uapi/linux/nsfs.h
+++ b/libc/kernel/uapi/linux/nsfs.h
@@ -7,9 +7,15 @@
 #ifndef __LINUX_NSFS_H
 #define __LINUX_NSFS_H
 #include <linux/ioctl.h>
+#include <linux/types.h>
 #define NSIO 0xb7
 #define NS_GET_USERNS _IO(NSIO, 0x1)
 #define NS_GET_PARENT _IO(NSIO, 0x2)
 #define NS_GET_NSTYPE _IO(NSIO, 0x3)
 #define NS_GET_OWNER_UID _IO(NSIO, 0x4)
+#define NS_GET_MNTNS_ID _IOR(NSIO, 0x5, __u64)
+#define NS_GET_PID_FROM_PIDNS _IOR(NSIO, 0x6, int)
+#define NS_GET_TGID_FROM_PIDNS _IOR(NSIO, 0x7, int)
+#define NS_GET_PID_IN_PIDNS _IOR(NSIO, 0x8, int)
+#define NS_GET_TGID_IN_PIDNS _IOR(NSIO, 0x9, int)
 #endif
diff --git a/libc/kernel/uapi/linux/openvswitch.h b/libc/kernel/uapi/linux/openvswitch.h
index d45f4fa..98c8037 100644
--- a/libc/kernel/uapi/linux/openvswitch.h
+++ b/libc/kernel/uapi/linux/openvswitch.h
@@ -429,6 +429,13 @@
   __OVS_CHECK_PKT_LEN_ATTR_MAX,
 };
 #define OVS_CHECK_PKT_LEN_ATTR_MAX (__OVS_CHECK_PKT_LEN_ATTR_MAX - 1)
+#define OVS_PSAMPLE_COOKIE_MAX_SIZE 16
+enum ovs_psample_attr {
+  OVS_PSAMPLE_ATTR_GROUP = 1,
+  OVS_PSAMPLE_ATTR_COOKIE,
+  __OVS_PSAMPLE_ATTR_MAX
+};
+#define OVS_PSAMPLE_ATTR_MAX (__OVS_PSAMPLE_ATTR_MAX - 1)
 enum ovs_action_attr {
   OVS_ACTION_ATTR_UNSPEC,
   OVS_ACTION_ATTR_OUTPUT,
@@ -455,6 +462,7 @@
   OVS_ACTION_ATTR_ADD_MPLS,
   OVS_ACTION_ATTR_DEC_TTL,
   OVS_ACTION_ATTR_DROP,
+  OVS_ACTION_ATTR_PSAMPLE,
   __OVS_ACTION_ATTR_MAX,
 };
 #define OVS_ACTION_ATTR_MAX (__OVS_ACTION_ATTR_MAX - 1)
diff --git a/libc/kernel/uapi/linux/perf_event.h b/libc/kernel/uapi/linux/perf_event.h
index 16a1a2e..ec9b856 100644
--- a/libc/kernel/uapi/linux/perf_event.h
+++ b/libc/kernel/uapi/linux/perf_event.h
@@ -460,6 +460,8 @@
 #define PERF_MEM_LVLNUM_L2 0x02
 #define PERF_MEM_LVLNUM_L3 0x03
 #define PERF_MEM_LVLNUM_L4 0x04
+#define PERF_MEM_LVLNUM_L2_MHB 0x05
+#define PERF_MEM_LVLNUM_MSC 0x06
 #define PERF_MEM_LVLNUM_UNC 0x08
 #define PERF_MEM_LVLNUM_CXL 0x09
 #define PERF_MEM_LVLNUM_IO 0x0a
diff --git a/libc/kernel/uapi/linux/pidfd.h b/libc/kernel/uapi/linux/pidfd.h
index 082b4a0..9068727 100644
--- a/libc/kernel/uapi/linux/pidfd.h
+++ b/libc/kernel/uapi/linux/pidfd.h
@@ -8,9 +8,21 @@
 #define _UAPI_LINUX_PIDFD_H
 #include <linux/types.h>
 #include <linux/fcntl.h>
+#include <linux/ioctl.h>
 #define PIDFD_NONBLOCK O_NONBLOCK
 #define PIDFD_THREAD O_EXCL
 #define PIDFD_SIGNAL_THREAD (1UL << 0)
 #define PIDFD_SIGNAL_THREAD_GROUP (1UL << 1)
 #define PIDFD_SIGNAL_PROCESS_GROUP (1UL << 2)
+#define PIDFS_IOCTL_MAGIC 0xFF
+#define PIDFD_GET_CGROUP_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 1)
+#define PIDFD_GET_IPC_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 2)
+#define PIDFD_GET_MNT_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 3)
+#define PIDFD_GET_NET_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 4)
+#define PIDFD_GET_PID_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 5)
+#define PIDFD_GET_PID_FOR_CHILDREN_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 6)
+#define PIDFD_GET_TIME_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 7)
+#define PIDFD_GET_TIME_FOR_CHILDREN_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 8)
+#define PIDFD_GET_USER_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 9)
+#define PIDFD_GET_UTS_NAMESPACE _IO(PIDFS_IOCTL_MAGIC, 10)
 #endif
diff --git a/libc/kernel/uapi/linux/pkt_cls.h b/libc/kernel/uapi/linux/pkt_cls.h
index 6b5143c..bdca553 100644
--- a/libc/kernel/uapi/linux/pkt_cls.h
+++ b/libc/kernel/uapi/linux/pkt_cls.h
@@ -419,6 +419,8 @@
   TCA_FLOWER_KEY_CFM,
   TCA_FLOWER_KEY_SPI,
   TCA_FLOWER_KEY_SPI_MASK,
+  TCA_FLOWER_KEY_ENC_FLAGS,
+  TCA_FLOWER_KEY_ENC_FLAGS_MASK,
   __TCA_FLOWER_MAX,
 };
 #define TCA_FLOWER_MAX (__TCA_FLOWER_MAX - 1)
@@ -497,7 +499,13 @@
 enum {
   TCA_FLOWER_KEY_FLAGS_IS_FRAGMENT = (1 << 0),
   TCA_FLOWER_KEY_FLAGS_FRAG_IS_FIRST = (1 << 1),
+  TCA_FLOWER_KEY_FLAGS_TUNNEL_CSUM = (1 << 2),
+  TCA_FLOWER_KEY_FLAGS_TUNNEL_DONT_FRAGMENT = (1 << 3),
+  TCA_FLOWER_KEY_FLAGS_TUNNEL_OAM = (1 << 4),
+  TCA_FLOWER_KEY_FLAGS_TUNNEL_CRIT_OPT = (1 << 5),
+  __TCA_FLOWER_KEY_FLAGS_MAX,
 };
+#define TCA_FLOWER_KEY_FLAGS_MAX (__TCA_FLOWER_KEY_FLAGS_MAX - 1)
 enum {
   TCA_FLOWER_KEY_CFM_OPT_UNSPEC,
   TCA_FLOWER_KEY_CFM_MD_LEVEL,
diff --git a/libc/kernel/uapi/linux/psample.h b/libc/kernel/uapi/linux/psample.h
index c82e76e..f9f979c 100644
--- a/libc/kernel/uapi/linux/psample.h
+++ b/libc/kernel/uapi/linux/psample.h
@@ -22,6 +22,8 @@
   PSAMPLE_ATTR_LATENCY,
   PSAMPLE_ATTR_TIMESTAMP,
   PSAMPLE_ATTR_PROTO,
+  PSAMPLE_ATTR_USER_COOKIE,
+  PSAMPLE_ATTR_SAMPLE_PROBABILITY,
   __PSAMPLE_ATTR_MAX
 };
 enum psample_command {
diff --git a/libc/kernel/uapi/linux/psp-sev.h b/libc/kernel/uapi/linux/psp-sev.h
index 82fcbf1..7274081 100644
--- a/libc/kernel/uapi/linux/psp-sev.h
+++ b/libc/kernel/uapi/linux/psp-sev.h
@@ -20,6 +20,7 @@
   SNP_PLATFORM_STATUS,
   SNP_COMMIT,
   SNP_SET_CONFIG,
+  SNP_VLEK_LOAD,
   SEV_MAX,
 };
 typedef enum {
@@ -28,6 +29,7 @@
   SEV_RET_INVALID_PLATFORM_STATE,
   SEV_RET_INVALID_GUEST_STATE,
   SEV_RET_INAVLID_CONFIG,
+  SEV_RET_INVALID_CONFIG = SEV_RET_INAVLID_CONFIG,
   SEV_RET_INVALID_LEN,
   SEV_RET_ALREADY_OWNED,
   SEV_RET_INVALID_CERTIFICATE,
@@ -113,6 +115,15 @@
   __u32 rsvd : 30;
   __u8 rsvd1[52];
 } __attribute__((__packed__));
+struct sev_user_data_snp_vlek_load {
+  __u32 len;
+  __u8 vlek_wrapped_version;
+  __u8 rsvd[3];
+  __u64 vlek_wrapped_address;
+} __attribute__((__packed__));
+struct sev_user_data_snp_wrapped_vlek_hashstick {
+  __u8 data[432];
+} __attribute__((__packed__));
 struct sev_issue_cmd {
   __u32 cmd;
   __u64 data;
diff --git a/libc/kernel/uapi/linux/random.h b/libc/kernel/uapi/linux/random.h
index d1fd998..64f62d9 100644
--- a/libc/kernel/uapi/linux/random.h
+++ b/libc/kernel/uapi/linux/random.h
@@ -24,4 +24,10 @@
 #define GRND_NONBLOCK 0x0001
 #define GRND_RANDOM 0x0002
 #define GRND_INSECURE 0x0004
+struct vgetrandom_opaque_params {
+  __u32 size_of_opaque_state;
+  __u32 mmap_prot;
+  __u32 mmap_flags;
+  __u32 reserved[13];
+};
 #endif
diff --git a/libc/kernel/uapi/linux/sev-guest.h b/libc/kernel/uapi/linux/sev-guest.h
index a822bed..a722641 100644
--- a/libc/kernel/uapi/linux/sev-guest.h
+++ b/libc/kernel/uapi/linux/sev-guest.h
@@ -51,6 +51,8 @@
 #define SNP_GUEST_FW_ERR_MASK GENMASK_ULL(31, 0)
 #define SNP_GUEST_VMM_ERR_SHIFT 32
 #define SNP_GUEST_VMM_ERR(x) (((u64) x) << SNP_GUEST_VMM_ERR_SHIFT)
+#define SNP_GUEST_FW_ERR(x) ((x) & SNP_GUEST_FW_ERR_MASK)
+#define SNP_GUEST_ERR(vmm_err,fw_err) (SNP_GUEST_VMM_ERR(vmm_err) | SNP_GUEST_FW_ERR(fw_err))
 #define SNP_GUEST_VMM_ERR_INVALID_LEN 1
 #define SNP_GUEST_VMM_ERR_BUSY 2
 #endif
diff --git a/libc/kernel/uapi/linux/stat.h b/libc/kernel/uapi/linux/stat.h
index ff98fd2..aae9ed4 100644
--- a/libc/kernel/uapi/linux/stat.h
+++ b/libc/kernel/uapi/linux/stat.h
@@ -69,7 +69,11 @@
   __u32 stx_dio_mem_align;
   __u32 stx_dio_offset_align;
   __u64 stx_subvol;
-  __u64 __spare3[11];
+  __u32 stx_atomic_write_unit_min;
+  __u32 stx_atomic_write_unit_max;
+  __u32 stx_atomic_write_segments_max;
+  __u32 __spare1[1];
+  __u64 __spare3[9];
 };
 #define STATX_TYPE 0x00000001U
 #define STATX_MODE 0x00000002U
@@ -88,6 +92,7 @@
 #define STATX_DIOALIGN 0x00002000U
 #define STATX_MNT_ID_UNIQUE 0x00004000U
 #define STATX_SUBVOL 0x00008000U
+#define STATX_WRITE_ATOMIC 0x00010000U
 #define STATX__RESERVED 0x80000000U
 #define STATX_ALL 0x00000fffU
 #define STATX_ATTR_COMPRESSED 0x00000004
@@ -99,4 +104,5 @@
 #define STATX_ATTR_MOUNT_ROOT 0x00002000
 #define STATX_ATTR_VERITY 0x00100000
 #define STATX_ATTR_DAX 0x00200000
+#define STATX_ATTR_WRITE_ATOMIC 0x00400000
 #endif
diff --git a/libc/kernel/uapi/linux/swab.h b/libc/kernel/uapi/linux/swab.h
index 6225a76..5d240e1 100644
--- a/libc/kernel/uapi/linux/swab.h
+++ b/libc/kernel/uapi/linux/swab.h
@@ -15,29 +15,14 @@
 #define ___constant_swab64(x) ((__u64) ((((__u64) (x) & (__u64) 0x00000000000000ffULL) << 56) | (((__u64) (x) & (__u64) 0x000000000000ff00ULL) << 40) | (((__u64) (x) & (__u64) 0x0000000000ff0000ULL) << 24) | (((__u64) (x) & (__u64) 0x00000000ff000000ULL) << 8) | (((__u64) (x) & (__u64) 0x000000ff00000000ULL) >> 8) | (((__u64) (x) & (__u64) 0x0000ff0000000000ULL) >> 24) | (((__u64) (x) & (__u64) 0x00ff000000000000ULL) >> 40) | (((__u64) (x) & (__u64) 0xff00000000000000ULL) >> 56)))
 #define ___constant_swahw32(x) ((__u32) ((((__u32) (x) & (__u32) 0x0000ffffUL) << 16) | (((__u32) (x) & (__u32) 0xffff0000UL) >> 16)))
 #define ___constant_swahb32(x) ((__u32) ((((__u32) (x) & (__u32) 0x00ff00ffUL) << 8) | (((__u32) (x) & (__u32) 0xff00ff00UL) >> 8)))
-#ifdef __arch_swab16
-#else
-#endif
-#ifdef __arch_swab32
-#else
-#endif
-#ifdef __arch_swab64
-#elif defined(__SWAB_64_THRU_32__)
+#ifdef __SWAB_64_THRU_32__
 #else
 #endif
 static inline __attribute__((__const__)) __u32 __fswahw32(__u32 val) {
-#ifdef __arch_swahw32
-  return __arch_swahw32(val);
-#else
   return ___constant_swahw32(val);
-#endif
 }
 static inline __attribute__((__const__)) __u32 __fswahb32(__u32 val) {
-#ifdef __arch_swahb32
-  return __arch_swahb32(val);
-#else
   return ___constant_swahb32(val);
-#endif
 }
 #define __swab16(x) (__u16) __builtin_bswap16((__u16) (x))
 #define __swab32(x) (__u32) __builtin_bswap32((__u32) (x))
@@ -48,73 +33,33 @@
 #define __swahw32(x) (__builtin_constant_p((__u32) (x)) ? ___constant_swahw32(x) : __fswahw32(x))
 #define __swahb32(x) (__builtin_constant_p((__u32) (x)) ? ___constant_swahb32(x) : __fswahb32(x))
 static __always_inline __u16 __swab16p(const __u16 * p) {
-#ifdef __arch_swab16p
-  return __arch_swab16p(p);
-#else
   return __swab16(* p);
-#endif
 }
 static __always_inline __u32 __swab32p(const __u32 * p) {
-#ifdef __arch_swab32p
-  return __arch_swab32p(p);
-#else
   return __swab32(* p);
-#endif
 }
 static __always_inline __u64 __swab64p(const __u64 * p) {
-#ifdef __arch_swab64p
-  return __arch_swab64p(p);
-#else
   return __swab64(* p);
-#endif
 }
 static inline __u32 __swahw32p(const __u32 * p) {
-#ifdef __arch_swahw32p
-  return __arch_swahw32p(p);
-#else
   return __swahw32(* p);
-#endif
 }
 static inline __u32 __swahb32p(const __u32 * p) {
-#ifdef __arch_swahb32p
-  return __arch_swahb32p(p);
-#else
   return __swahb32(* p);
-#endif
 }
 static inline void __swab16s(__u16 * p) {
-#ifdef __arch_swab16s
-  __arch_swab16s(p);
-#else
   * p = __swab16p(p);
-#endif
 }
 static __always_inline void __swab32s(__u32 * p) {
-#ifdef __arch_swab32s
-  __arch_swab32s(p);
-#else
   * p = __swab32p(p);
-#endif
 }
 static __always_inline void __swab64s(__u64 * p) {
-#ifdef __arch_swab64s
-  __arch_swab64s(p);
-#else
   * p = __swab64p(p);
-#endif
 }
 static inline void __swahw32s(__u32 * p) {
-#ifdef __arch_swahw32s
-  __arch_swahw32s(p);
-#else
   * p = __swahw32p(p);
-#endif
 }
 static inline void __swahb32s(__u32 * p) {
-#ifdef __arch_swahb32s
-  __arch_swahb32s(p);
-#else
   * p = __swahb32p(p);
-#endif
 }
 #endif
diff --git a/libc/kernel/uapi/linux/tcp_metrics.h b/libc/kernel/uapi/linux/tcp_metrics.h
index 931f50c..46ca141 100644
--- a/libc/kernel/uapi/linux/tcp_metrics.h
+++ b/libc/kernel/uapi/linux/tcp_metrics.h
@@ -4,8 +4,8 @@
  * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
  * for more information.
  */
-#ifndef _LINUX_TCP_METRICS_H
-#define _LINUX_TCP_METRICS_H
+#ifndef _UAPI_LINUX_TCP_METRICS_H
+#define _UAPI_LINUX_TCP_METRICS_H
 #include <linux/types.h>
 #define TCP_METRICS_GENL_NAME "tcp_metrics"
 #define TCP_METRICS_GENL_VERSION 0x1
@@ -21,6 +21,17 @@
 };
 #define TCP_METRIC_MAX (__TCP_METRIC_MAX - 1)
 enum {
+  TCP_METRICS_A_METRICS_RTT = 1,
+  TCP_METRICS_A_METRICS_RTTVAR,
+  TCP_METRICS_A_METRICS_SSTHRESH,
+  TCP_METRICS_A_METRICS_CWND,
+  TCP_METRICS_A_METRICS_REODERING,
+  TCP_METRICS_A_METRICS_RTT_US,
+  TCP_METRICS_A_METRICS_RTTVAR_US,
+  __TCP_METRICS_A_METRICS_MAX
+};
+#define TCP_METRICS_A_METRICS_MAX (__TCP_METRICS_A_METRICS_MAX - 1)
+enum {
   TCP_METRICS_ATTR_UNSPEC,
   TCP_METRICS_ATTR_ADDR_IPV4,
   TCP_METRICS_ATTR_ADDR_IPV6,
diff --git a/libc/kernel/uapi/linux/um_timetravel.h b/libc/kernel/uapi/linux/um_timetravel.h
index 8706017..49eeb28 100644
--- a/libc/kernel/uapi/linux/um_timetravel.h
+++ b/libc/kernel/uapi/linux/um_timetravel.h
@@ -12,6 +12,15 @@
   __u32 seq;
   __u64 time;
 };
+#define UM_TIMETRAVEL_MAX_FDS 2
+enum um_timetravel_shared_mem_fds {
+  UM_TIMETRAVEL_SHARED_MEMFD,
+  UM_TIMETRAVEL_SHARED_LOGFD,
+  UM_TIMETRAVEL_SHARED_MAX_FDS,
+};
+enum um_timetravel_start_ack {
+  UM_TIMETRAVEL_START_ACK_ID = 0xffff,
+};
 enum um_timetravel_ops {
   UM_TIMETRAVEL_ACK = 0,
   UM_TIMETRAVEL_START = 1,
@@ -22,5 +31,36 @@
   UM_TIMETRAVEL_RUN = 6,
   UM_TIMETRAVEL_FREE_UNTIL = 7,
   UM_TIMETRAVEL_GET_TOD = 8,
+  UM_TIMETRAVEL_BROADCAST = 9,
+};
+#define UM_TIMETRAVEL_SCHEDSHM_VERSION 2
+enum um_timetravel_schedshm_cap {
+  UM_TIMETRAVEL_SCHEDSHM_CAP_TIME_SHARE = 0x1,
+};
+enum um_timetravel_schedshm_flags {
+  UM_TIMETRAVEL_SCHEDSHM_FLAGS_REQ_RUN = 0x1,
+};
+union um_timetravel_schedshm_client {
+  struct {
+    __u32 capa;
+    __u32 flags;
+    __u64 req_time;
+    __u64 name;
+  };
+  char reserve[128];
+};
+struct um_timetravel_schedshm {
+  union {
+    struct {
+      __u32 version;
+      __u32 len;
+      __u64 free_until;
+      __u64 current_time;
+      __u16 running_id;
+      __u16 max_clients;
+    };
+    char hdr[4096];
+  };
+  union um_timetravel_schedshm_client clients[];
 };
 #endif
diff --git a/libc/kernel/uapi/linux/v4l2-controls.h b/libc/kernel/uapi/linux/v4l2-controls.h
index 23158dc..f6ef26c 100644
--- a/libc/kernel/uapi/linux/v4l2-controls.h
+++ b/libc/kernel/uapi/linux/v4l2-controls.h
@@ -714,6 +714,7 @@
   V4L2_MPEG_VIDEO_AV1_LEVEL_7_2 = 22,
   V4L2_MPEG_VIDEO_AV1_LEVEL_7_3 = 23
 };
+#define V4L2_CID_MPEG_VIDEO_AVERAGE_QP (V4L2_CID_CODEC_BASE + 657)
 #define V4L2_CID_CODEC_CX2341X_BASE (V4L2_CTRL_CLASS_CODEC | 0x1000)
 #define V4L2_CID_MPEG_CX2341X_VIDEO_SPATIAL_FILTER_MODE (V4L2_CID_CODEC_CX2341X_BASE + 0)
 enum v4l2_mpeg_cx2341x_video_spatial_filter_mode {
diff --git a/libc/kernel/uapi/linux/version.h b/libc/kernel/uapi/linux/version.h
index 7faa30f..0cc45cf 100644
--- a/libc/kernel/uapi/linux/version.h
+++ b/libc/kernel/uapi/linux/version.h
@@ -4,8 +4,8 @@
  * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
  * for more information.
  */
-#define LINUX_VERSION_CODE 395776
+#define LINUX_VERSION_CODE 396032
 #define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + ((c) > 255 ? 255 : (c)))
 #define LINUX_VERSION_MAJOR 6
-#define LINUX_VERSION_PATCHLEVEL 10
+#define LINUX_VERSION_PATCHLEVEL 11
 #define LINUX_VERSION_SUBLEVEL 0
diff --git a/libc/kernel/uapi/linux/videodev2.h b/libc/kernel/uapi/linux/videodev2.h
index 3905949..fb69a8a 100644
--- a/libc/kernel/uapi/linux/videodev2.h
+++ b/libc/kernel/uapi/linux/videodev2.h
@@ -236,6 +236,8 @@
 #define V4L2_PIX_FMT_RGBA1010102 v4l2_fourcc('R', 'A', '3', '0')
 #define V4L2_PIX_FMT_ARGB2101010 v4l2_fourcc('A', 'R', '3', '0')
 #define V4L2_PIX_FMT_BGR48_12 v4l2_fourcc('B', '3', '1', '2')
+#define V4L2_PIX_FMT_BGR48 v4l2_fourcc('B', 'G', 'R', '6')
+#define V4L2_PIX_FMT_RGB48 v4l2_fourcc('R', 'G', 'B', '6')
 #define V4L2_PIX_FMT_ABGR64_12 v4l2_fourcc('B', '4', '1', '2')
 #define V4L2_PIX_FMT_GREY v4l2_fourcc('G', 'R', 'E', 'Y')
 #define V4L2_PIX_FMT_Y4 v4l2_fourcc('Y', '0', '4', ' ')
@@ -425,6 +427,16 @@
 #define V4L2_PIX_FMT_IPU3_SGBRG10 v4l2_fourcc('i', 'p', '3', 'g')
 #define V4L2_PIX_FMT_IPU3_SGRBG10 v4l2_fourcc('i', 'p', '3', 'G')
 #define V4L2_PIX_FMT_IPU3_SRGGB10 v4l2_fourcc('i', 'p', '3', 'r')
+#define V4L2_PIX_FMT_PISP_COMP1_RGGB v4l2_fourcc('P', 'C', '1', 'R')
+#define V4L2_PIX_FMT_PISP_COMP1_GRBG v4l2_fourcc('P', 'C', '1', 'G')
+#define V4L2_PIX_FMT_PISP_COMP1_GBRG v4l2_fourcc('P', 'C', '1', 'g')
+#define V4L2_PIX_FMT_PISP_COMP1_BGGR v4l2_fourcc('P', 'C', '1', 'B')
+#define V4L2_PIX_FMT_PISP_COMP1_MONO v4l2_fourcc('P', 'C', '1', 'M')
+#define V4L2_PIX_FMT_PISP_COMP2_RGGB v4l2_fourcc('P', 'C', '2', 'R')
+#define V4L2_PIX_FMT_PISP_COMP2_GRBG v4l2_fourcc('P', 'C', '2', 'G')
+#define V4L2_PIX_FMT_PISP_COMP2_GBRG v4l2_fourcc('P', 'C', '2', 'g')
+#define V4L2_PIX_FMT_PISP_COMP2_BGGR v4l2_fourcc('P', 'C', '2', 'B')
+#define V4L2_PIX_FMT_PISP_COMP2_MONO v4l2_fourcc('P', 'C', '2', 'M')
 #define V4L2_SDR_FMT_CU8 v4l2_fourcc('C', 'U', '0', '8')
 #define V4L2_SDR_FMT_CU16LE v4l2_fourcc('C', 'U', '1', '6')
 #define V4L2_SDR_FMT_CS8 v4l2_fourcc('C', 'S', '0', '8')
@@ -444,6 +456,7 @@
 #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D')
 #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P')
 #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S')
+#define V4L2_META_FMT_RPI_BE_CFG v4l2_fourcc('R', 'P', 'B', 'C')
 #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
 #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA 0x00000001
 #define V4L2_PIX_FMT_FLAG_SET_CSC 0x00000002
diff --git a/libc/kernel/uapi/linux/xfrm.h b/libc/kernel/uapi/linux/xfrm.h
index b8e79e8..9509efe 100644
--- a/libc/kernel/uapi/linux/xfrm.h
+++ b/libc/kernel/uapi/linux/xfrm.h
@@ -260,6 +260,7 @@
   XFRMA_IF_ID,
   XFRMA_MTIMER_THRESH,
   XFRMA_SA_DIR,
+  XFRMA_NAT_KEEPALIVE_INTERVAL,
   __XFRMA_MAX
 #define XFRMA_OUTPUT_MARK XFRMA_SET_MARK
 #define XFRMA_MAX (__XFRMA_MAX - 1)
diff --git a/libc/kernel/uapi/linux/zorro_ids.h b/libc/kernel/uapi/linux/zorro_ids.h
index f47c899..05239cd 100644
--- a/libc/kernel/uapi/linux/zorro_ids.h
+++ b/libc/kernel/uapi/linux/zorro_ids.h
@@ -354,6 +354,8 @@
 #define ZORRO_MANUF_VMC 0x1389
 #define ZORRO_PROD_VMC_ISDN_BLASTER_Z2 ZORRO_ID(VMC, 0x01, 0)
 #define ZORRO_PROD_VMC_HYPERCOM_4 ZORRO_ID(VMC, 0x02, 0)
+#define ZORRO_MANUF_CSLAB 0x1400
+#define ZORRO_PROD_CSLAB_WARP_1260 ZORRO_ID(CSLAB, 0x65, 0)
 #define ZORRO_MANUF_INFORMATION 0x157C
 #define ZORRO_PROD_INFORMATION_ISDN_ENGINE_I ZORRO_ID(INFORMATION, 0x64, 0)
 #define ZORRO_MANUF_VORTEX 0x2017
diff --git a/libc/kernel/uapi/misc/mrvl_cn10k_dpi.h b/libc/kernel/uapi/misc/mrvl_cn10k_dpi.h
new file mode 100644
index 0000000..7d8671f
--- /dev/null
+++ b/libc/kernel/uapi/misc/mrvl_cn10k_dpi.h
@@ -0,0 +1,26 @@
+/*
+ * This file is auto-generated. Modifications will be lost.
+ *
+ * See https://android.googlesource.com/platform/bionic/+/master/libc/kernel/
+ * for more information.
+ */
+#ifndef __MRVL_CN10K_DPI_H__
+#define __MRVL_CN10K_DPI_H__
+#include <linux/types.h>
+#define DPI_MAX_ENGINES 6
+struct dpi_mps_mrrs_cfg {
+  __u16 max_read_req_sz;
+  __u16 max_payload_sz;
+  __u16 port;
+  __u16 reserved;
+};
+struct dpi_engine_cfg {
+  __u64 fifo_mask;
+  __u16 molr[DPI_MAX_ENGINES];
+  __u16 update_molr;
+  __u16 reserved;
+};
+#define DPI_MAGIC_NUM 0xB8
+#define DPI_MPS_MRRS_CFG _IOW(DPI_MAGIC_NUM, 1, struct dpi_mps_mrrs_cfg)
+#define DPI_ENGINE_CFG _IOW(DPI_MAGIC_NUM, 2, struct dpi_engine_cfg)
+#endif
diff --git a/libc/kernel/uapi/rdma/bnxt_re-abi.h b/libc/kernel/uapi/rdma/bnxt_re-abi.h
index 3dceafd..50f8b8a 100644
--- a/libc/kernel/uapi/rdma/bnxt_re-abi.h
+++ b/libc/kernel/uapi/rdma/bnxt_re-abi.h
@@ -18,7 +18,7 @@
   BNXT_RE_UCNTX_CMASK_WC_DPI_ENABLED = 0x04ULL,
   BNXT_RE_UCNTX_CMASK_DBR_PACING_ENABLED = 0x08ULL,
   BNXT_RE_UCNTX_CMASK_POW2_DISABLED = 0x10ULL,
-  BNXT_RE_COMP_MASK_UCNTX_HW_RETX_ENABLED = 0x40,
+  BNXT_RE_UCNTX_CMASK_MSN_TABLE_ENABLED = 0x40,
 };
 enum bnxt_re_wqe_mode {
   BNXT_QPLIB_WQE_MODE_STATIC = 0x00,
diff --git a/libc/kernel/uapi/rdma/ib_user_ioctl_cmds.h b/libc/kernel/uapi/rdma/ib_user_ioctl_cmds.h
index d774fba..5f5c426 100644
--- a/libc/kernel/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/libc/kernel/uapi/rdma/ib_user_ioctl_cmds.h
@@ -8,8 +8,6 @@
 #define IB_USER_IOCTL_CMDS_H
 #define UVERBS_ID_NS_MASK 0xF000
 #define UVERBS_ID_NS_SHIFT 12
-#define UVERBS_UDATA_DRIVER_DATA_NS 1
-#define UVERBS_UDATA_DRIVER_DATA_FLAG (1UL << UVERBS_ID_NS_SHIFT)
 enum uverbs_default_objects {
   UVERBS_OBJECT_DEVICE,
   UVERBS_OBJECT_PD,
@@ -30,8 +28,10 @@
   UVERBS_OBJECT_ASYNC_EVENT,
 };
 enum {
-  UVERBS_ATTR_UHW_IN = UVERBS_UDATA_DRIVER_DATA_FLAG,
+  UVERBS_ID_DRIVER_NS = 1UL << UVERBS_ID_NS_SHIFT,
+  UVERBS_ATTR_UHW_IN = UVERBS_ID_DRIVER_NS,
   UVERBS_ATTR_UHW_OUT,
+  UVERBS_ID_DRIVER_NS_WITH_UHW,
 };
 enum uverbs_methods_device {
   UVERBS_METHOD_INVOKE_WRITE,
diff --git a/libc/kernel/uapi/rdma/mana-abi.h b/libc/kernel/uapi/rdma/mana-abi.h
index 7347175..02bb061 100644
--- a/libc/kernel/uapi/rdma/mana-abi.h
+++ b/libc/kernel/uapi/rdma/mana-abi.h
@@ -33,6 +33,13 @@
   __u32 tx_vp_offset;
   __u32 reserved;
 };
+struct mana_ib_create_rc_qp {
+  __aligned_u64 queue_buf[4];
+  __u32 queue_size[4];
+};
+struct mana_ib_create_rc_qp_resp {
+  __u32 queue_id[4];
+};
 struct mana_ib_create_wq {
   __aligned_u64 wq_buf_addr;
   __u32 wq_buf_size;
diff --git a/libc/kernel/uapi/rdma/mlx5_user_ioctl_cmds.h b/libc/kernel/uapi/rdma/mlx5_user_ioctl_cmds.h
index 69d5250..ebafb00 100644
--- a/libc/kernel/uapi/rdma/mlx5_user_ioctl_cmds.h
+++ b/libc/kernel/uapi/rdma/mlx5_user_ioctl_cmds.h
@@ -201,6 +201,9 @@
 enum mlx5_ib_device_query_context_attrs {
   MLX5_IB_ATTR_QUERY_CONTEXT_RESP_UCTX = (1U << UVERBS_ID_NS_SHIFT),
 };
+enum mlx5_ib_create_cq_attrs {
+  MLX5_IB_ATTR_CREATE_CQ_UAR_INDEX = UVERBS_ID_DRIVER_NS_WITH_UHW,
+};
 #define MLX5_IB_DW_MATCH_PARAM 0xA0
 struct mlx5_ib_match_params {
   __u32 match_params[MLX5_IB_DW_MATCH_PARAM];
diff --git a/libc/kernel/uapi/rdma/rdma_netlink.h b/libc/kernel/uapi/rdma/rdma_netlink.h
index 912a3c0..ac027ac 100644
--- a/libc/kernel/uapi/rdma/rdma_netlink.h
+++ b/libc/kernel/uapi/rdma/rdma_netlink.h
@@ -199,6 +199,8 @@
   RDMA_NLDEV_CMD_RES_SRQ_GET,
   RDMA_NLDEV_CMD_STAT_GET_STATUS,
   RDMA_NLDEV_CMD_RES_SRQ_GET_RAW,
+  RDMA_NLDEV_CMD_NEWDEV,
+  RDMA_NLDEV_CMD_DELDEV,
   RDMA_NLDEV_NUM_OPS
 };
 enum rdma_nldev_print_type {
@@ -306,6 +308,9 @@
   RDMA_NLDEV_SYS_ATTR_PRIVILEGED_QKEY_MODE,
   RDMA_NLDEV_ATTR_DRIVER_DETAILS,
   RDMA_NLDEV_ATTR_RES_SUBTYPE,
+  RDMA_NLDEV_ATTR_DEV_TYPE,
+  RDMA_NLDEV_ATTR_PARENT_NAME,
+  RDMA_NLDEV_ATTR_NAME_ASSIGN_TYPE,
   RDMA_NLDEV_ATTR_MAX
 };
 enum rdma_nl_counter_mode {
@@ -318,4 +323,11 @@
   RDMA_COUNTER_MASK_QP_TYPE = 1,
   RDMA_COUNTER_MASK_PID = 1 << 1,
 };
+enum rdma_nl_dev_type {
+  RDMA_DEVICE_TYPE_SMI = 1,
+};
+enum rdma_nl_name_assign_type {
+  RDMA_NAME_ASSIGN_TYPE_UNKNOWN = 0,
+  RDMA_NAME_ASSIGN_TYPE_USER = 1,
+};
 #endif
diff --git a/libc/kernel/uapi/scsi/scsi_bsg_mpi3mr.h b/libc/kernel/uapi/scsi/scsi_bsg_mpi3mr.h
index d98e8fa..fe3094e 100644
--- a/libc/kernel/uapi/scsi/scsi_bsg_mpi3mr.h
+++ b/libc/kernel/uapi/scsi/scsi_bsg_mpi3mr.h
@@ -144,7 +144,7 @@
 };
 struct mpi3mr_bsg_in_hdb_status {
   __u8 num_hdb_types;
-  __u8 rsvd1;
+  __u8 element_trigger_format;
   __u16 rsvd2;
   __u32 rsvd3;
   struct mpi3mr_hdb_entry entry[1];
diff --git a/libc/kernel/uapi/sound/asequencer.h b/libc/kernel/uapi/sound/asequencer.h
index 7918528..a3826a5 100644
--- a/libc/kernel/uapi/sound/asequencer.h
+++ b/libc/kernel/uapi/sound/asequencer.h
@@ -7,7 +7,7 @@
 #ifndef _UAPI__SOUND_ASEQUENCER_H
 #define _UAPI__SOUND_ASEQUENCER_H
 #include <sound/asound.h>
-#define SNDRV_SEQ_VERSION SNDRV_PROTOCOL_VERSION(1, 0, 3)
+#define SNDRV_SEQ_VERSION SNDRV_PROTOCOL_VERSION(1, 0, 4)
 #define SNDRV_SEQ_EVENT_SYSTEM 0
 #define SNDRV_SEQ_EVENT_RESULT 1
 #define SNDRV_SEQ_EVENT_NOTE 5
@@ -343,7 +343,8 @@
   int ppq;
   unsigned int skew_value;
   unsigned int skew_base;
-  char reserved[24];
+  unsigned short tempo_base;
+  char reserved[22];
 };
 #define SNDRV_SEQ_TIMER_ALSA 0
 #define SNDRV_SEQ_TIMER_MIDI_CLOCK 1
diff --git a/libc/kernel/uapi/sound/asound.h b/libc/kernel/uapi/sound/asound.h
index b608ed5..cfe9f66 100644
--- a/libc/kernel/uapi/sound/asound.h
+++ b/libc/kernel/uapi/sound/asound.h
@@ -92,7 +92,7 @@
 #define SNDRV_HWDEP_IOCTL_INFO _IOR('H', 0x01, struct snd_hwdep_info)
 #define SNDRV_HWDEP_IOCTL_DSP_STATUS _IOR('H', 0x02, struct snd_hwdep_dsp_status)
 #define SNDRV_HWDEP_IOCTL_DSP_LOAD _IOW('H', 0x03, struct snd_hwdep_dsp_image)
-#define SNDRV_PCM_VERSION SNDRV_PROTOCOL_VERSION(2, 0, 17)
+#define SNDRV_PCM_VERSION SNDRV_PROTOCOL_VERSION(2, 0, 18)
 typedef unsigned long snd_pcm_uframes_t;
 typedef signed long snd_pcm_sframes_t;
 enum {
@@ -263,7 +263,7 @@
   unsigned char id[16];
   unsigned short id16[8];
   unsigned int id32[4];
-};
+} __attribute__((deprecated));
 struct snd_pcm_info {
   unsigned int device;
   unsigned int subdevice;
@@ -276,7 +276,7 @@
   int dev_subclass;
   unsigned int subdevices_count;
   unsigned int subdevices_avail;
-  union snd_pcm_sync_id sync;
+  unsigned char pad1[16];
   unsigned char reserved[64];
 };
 typedef int snd_pcm_hw_param_t;
@@ -324,7 +324,8 @@
   unsigned int rate_num;
   unsigned int rate_den;
   snd_pcm_uframes_t fifo_size;
-  unsigned char reserved[64];
+  unsigned char sync[16];
+  unsigned char reserved[48];
 };
 enum {
   SNDRV_PCM_TSTAMP_NONE = 0,
diff --git a/libc/kernel/uapi/sound/sof/abi.h b/libc/kernel/uapi/sound/sof/abi.h
index fe1fe47..ed66131 100644
--- a/libc/kernel/uapi/sound/sof/abi.h
+++ b/libc/kernel/uapi/sound/sof/abi.h
@@ -9,7 +9,7 @@
 #include <linux/types.h>
 #define SOF_ABI_MAJOR 3
 #define SOF_ABI_MINOR 23
-#define SOF_ABI_PATCH 0
+#define SOF_ABI_PATCH 1
 #define SOF_ABI_MAJOR_SHIFT 24
 #define SOF_ABI_MAJOR_MASK 0xff
 #define SOF_ABI_MINOR_SHIFT 12
diff --git a/libc/libc.map.txt b/libc/libc.map.txt
index 0d2e42f..86dcc39 100644
--- a/libc/libc.map.txt
+++ b/libc/libc.map.txt
@@ -591,7 +591,6 @@
     killpg;
     klogctl;
     labs;
-    lchmod; # introduced=36
     lchown;
     lcong48; # introduced=23
     ldexp;
@@ -671,7 +670,6 @@
     mprotect;
     mrand48;
     mremap;
-    mseal; # introduced=36
     msync;
     munlock;
     munlockall;
@@ -846,7 +844,6 @@
     pwrite;
     pwrite64;
     qsort;
-    qsort_r; # introduced=36
     quick_exit;
     raise;
     rand;
@@ -955,7 +952,6 @@
     setvbuf;
     setxattr;
     shutdown;
-    sig2str; # introduced=36
     sigaction;
     sigaddset;
     sigaltstack;
@@ -998,7 +994,6 @@
     stdout; # var introduced=23
     stpcpy;
     stpncpy;
-    str2sig; # introduced=36
     strcasecmp;
     strcasecmp_l; # introduced=23
     strcasestr;
@@ -1613,6 +1608,17 @@
     __system_properties_zygote_reload; # apex
 } LIBC_U;
 
+LIBC_36 { # introduced=36
+  global:
+    lchmod;
+    mseal;
+    pthread_getaffinity_np;
+    pthread_setaffinity_np;
+    qsort_r;
+    sig2str;
+    str2sig;
+} LIBC_V;
+
 LIBC_PRIVATE {
   global:
     __accept4; # arm x86
diff --git a/libc/malloc_debug/Android.bp b/libc/malloc_debug/Android.bp
index 3828c28..5d61801 100644
--- a/libc/malloc_debug/Android.bp
+++ b/libc/malloc_debug/Android.bp
@@ -79,6 +79,10 @@
         "libmemunreachable",
     ],
 
+    whole_static_libs: [
+        "libmemory_trace",
+    ],
+
     shared_libs: [
         "libunwindstack",
     ],
diff --git a/libc/malloc_debug/Nanotime.h b/libc/malloc_debug/Nanotime.h
new file mode 100644
index 0000000..d7c3f60
--- /dev/null
+++ b/libc/malloc_debug/Nanotime.h
@@ -0,0 +1,38 @@
+/*
+ * Copyright (C) 2012 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <stdint.h>
+#include <time.h>
+
+static inline __always_inline uint64_t Nanotime() {
+  struct timespec t = {};
+  clock_gettime(CLOCK_MONOTONIC, &t);
+  return static_cast<uint64_t>(t.tv_sec) * 1000000000LL + t.tv_nsec;
+}
diff --git a/libc/malloc_debug/RecordData.cpp b/libc/malloc_debug/RecordData.cpp
index 79e051b..1641732 100644
--- a/libc/malloc_debug/RecordData.cpp
+++ b/libc/malloc_debug/RecordData.cpp
@@ -39,72 +39,19 @@
 #include <mutex>
 
 #include <android-base/stringprintf.h>
+#include <memory_trace/MemoryTrace.h>
 
 #include "Config.h"
 #include "DebugData.h"
+#include "Nanotime.h"
 #include "RecordData.h"
 #include "debug_disable.h"
 #include "debug_log.h"
 
-RecordEntry::RecordEntry() : tid_(gettid()) {
-}
-
-bool ThreadCompleteEntry::Write(int fd) const {
-  return dprintf(fd, "%d: thread_done 0x0\n", tid_) > 0;
-}
-
-AllocEntry::AllocEntry(void* pointer, uint64_t start_ns, uint64_t end_ns)
-    : pointer_(pointer), start_ns_(start_ns), end_ns_(end_ns) {}
-
-MallocEntry::MallocEntry(void* pointer, size_t size, uint64_t start_ns, uint64_t end_ns)
-    : AllocEntry(pointer, start_ns, end_ns), size_(size) {}
-
-bool MallocEntry::Write(int fd) const {
-  return dprintf(fd, "%d: malloc %p %zu %" PRIu64 " %" PRIu64 "\n", tid_, pointer_, size_,
-                 start_ns_, end_ns_) > 0;
-}
-
-FreeEntry::FreeEntry(void* pointer, uint64_t start_ns, uint64_t end_ns)
-    : AllocEntry(pointer, start_ns, end_ns) {}
-
-bool FreeEntry::Write(int fd) const {
-  return dprintf(fd, "%d: free %p %" PRIu64 " %" PRIu64 "\n", tid_, pointer_, start_ns_, end_ns_) >
-         0;
-}
-
-CallocEntry::CallocEntry(void* pointer, size_t nmemb, size_t size, uint64_t start_ns,
-                         uint64_t end_ns)
-    : MallocEntry(pointer, size, start_ns, end_ns), nmemb_(nmemb) {}
-
-bool CallocEntry::Write(int fd) const {
-  return dprintf(fd, "%d: calloc %p %zu %zu %" PRIu64 " %" PRIu64 "\n", tid_, pointer_, nmemb_,
-                 size_, start_ns_, end_ns_) > 0;
-}
-
-ReallocEntry::ReallocEntry(void* pointer, size_t size, void* old_pointer, uint64_t start_ns,
-                           uint64_t end_ns)
-    : MallocEntry(pointer, size, start_ns, end_ns), old_pointer_(old_pointer) {}
-
-bool ReallocEntry::Write(int fd) const {
-  return dprintf(fd, "%d: realloc %p %p %zu %" PRIu64 " %" PRIu64 "\n", tid_, pointer_,
-                 old_pointer_, size_, start_ns_, end_ns_) > 0;
-}
-
-// aligned_alloc, posix_memalign, memalign, pvalloc, valloc all recorded with this class.
-MemalignEntry::MemalignEntry(void* pointer, size_t size, size_t alignment, uint64_t start_ns,
-                             uint64_t end_ns)
-    : MallocEntry(pointer, size, start_ns, end_ns), alignment_(alignment) {}
-
-bool MemalignEntry::Write(int fd) const {
-  return dprintf(fd, "%d: memalign %p %zu %zu %" PRIu64 " %" PRIu64 "\n", tid_, pointer_,
-                 alignment_, size_, start_ns_, end_ns_) > 0;
-}
-
 struct ThreadData {
-  ThreadData(RecordData* record_data, ThreadCompleteEntry* entry)
-      : record_data(record_data), entry(entry) {}
-  RecordData* record_data;
-  ThreadCompleteEntry* entry;
+  ThreadData(RecordData* record_data) : record_data(record_data) {}
+
+  RecordData* record_data = nullptr;
   size_t count = 0;
 };
 
@@ -117,7 +64,8 @@
   if (thread_data->count == 4) {
     ScopedDisableDebugCalls disable;
 
-    thread_data->record_data->AddEntryOnly(thread_data->entry);
+    thread_data->record_data->AddEntryOnly(memory_trace::Entry{
+        .tid = gettid(), .type = memory_trace::THREAD_DONE, .end_ns = Nanotime()});
     delete thread_data;
   } else {
     pthread_setspecific(thread_data->record_data->key(), data);
@@ -159,7 +107,7 @@
   }
 
   for (size_t i = 0; i < cur_index_; i++) {
-    if (!entries_[i]->Write(dump_fd)) {
+    if (!memory_trace::WriteEntryToFd(dump_fd, entries_[i])) {
       error_log("Failed to write record alloc information: %s", strerror(errno));
       break;
     }
@@ -201,23 +149,23 @@
   pthread_key_delete(key_);
 }
 
-void RecordData::AddEntryOnly(const RecordEntry* entry) {
+void RecordData::AddEntryOnly(const memory_trace::Entry& entry) {
   std::lock_guard<std::mutex> entries_lock(entries_lock_);
   if (cur_index_ == entries_.size()) {
     // Maxed out, throw the entry away.
     return;
   }
 
-  entries_[cur_index_++].reset(entry);
+  entries_[cur_index_++] = entry;
   if (cur_index_ == entries_.size()) {
     info_log("Maximum number of records added, all new operations will be dropped.");
   }
 }
 
-void RecordData::AddEntry(const RecordEntry* entry) {
+void RecordData::AddEntry(const memory_trace::Entry& entry) {
   void* data = pthread_getspecific(key_);
   if (data == nullptr) {
-    ThreadData* thread_data = new ThreadData(this, new ThreadCompleteEntry());
+    ThreadData* thread_data = new ThreadData(this);
     pthread_setspecific(key_, thread_data);
   }
 
diff --git a/libc/malloc_debug/RecordData.h b/libc/malloc_debug/RecordData.h
index 7efa1f7..f4b0d82 100644
--- a/libc/malloc_debug/RecordData.h
+++ b/libc/malloc_debug/RecordData.h
@@ -39,117 +39,9 @@
 #include <string>
 #include <vector>
 
+#include <memory_trace/MemoryTrace.h>
 #include <platform/bionic/macros.h>
 
-class RecordEntry {
- public:
-  RecordEntry();
-  virtual ~RecordEntry() = default;
-
-  virtual bool Write(int fd) const = 0;
-
- protected:
-  pid_t tid_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(RecordEntry);
-};
-
-class ThreadCompleteEntry : public RecordEntry {
- public:
-  ThreadCompleteEntry() = default;
-  virtual ~ThreadCompleteEntry() = default;
-
-  bool Write(int fd) const override;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(ThreadCompleteEntry);
-};
-
-class AllocEntry : public RecordEntry {
- public:
-  explicit AllocEntry(void* pointer, uint64_t st, uint64_t et);
-  virtual ~AllocEntry() = default;
-
- protected:
-  void* pointer_;
-
-  // The start/end time of this operation.
-  uint64_t start_ns_;
-  uint64_t end_ns_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(AllocEntry);
-};
-
-class MallocEntry : public AllocEntry {
- public:
-  MallocEntry(void* pointer, size_t size, uint64_t st, uint64_t et);
-  virtual ~MallocEntry() = default;
-
-  bool Write(int fd) const override;
-
- protected:
-  size_t size_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(MallocEntry);
-};
-
-class FreeEntry : public AllocEntry {
- public:
-  explicit FreeEntry(void* pointer, uint64_t st, uint64_t et);
-  virtual ~FreeEntry() = default;
-
-  bool Write(int fd) const override;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(FreeEntry);
-};
-
-class CallocEntry : public MallocEntry {
- public:
-  CallocEntry(void* pointer, size_t nmemb, size_t size, uint64_t st, uint64_t et);
-  virtual ~CallocEntry() = default;
-
-  bool Write(int fd) const override;
-
- protected:
-  size_t nmemb_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(CallocEntry);
-};
-
-class ReallocEntry : public MallocEntry {
- public:
-  ReallocEntry(void* pointer, size_t size, void* old_pointer, uint64_t st, uint64_t et);
-  virtual ~ReallocEntry() = default;
-
-  bool Write(int fd) const override;
-
- protected:
-  void* old_pointer_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(ReallocEntry);
-};
-
-// aligned_alloc, posix_memalign, memalign, pvalloc, valloc all recorded with this class.
-class MemalignEntry : public MallocEntry {
- public:
-  MemalignEntry(void* pointer, size_t size, size_t alignment, uint64_t st, uint64_t et);
-  virtual ~MemalignEntry() = default;
-
-  bool Write(int fd) const override;
-
- protected:
-  size_t alignment_;
-
- private:
-  BIONIC_DISALLOW_COPY_AND_ASSIGN(MemalignEntry);
-};
-
 class Config;
 
 class RecordData {
@@ -159,8 +51,8 @@
 
   bool Initialize(const Config& config);
 
-  void AddEntry(const RecordEntry* entry);
-  void AddEntryOnly(const RecordEntry* entry);
+  void AddEntry(const memory_trace::Entry& entry);
+  void AddEntryOnly(const memory_trace::Entry& entry);
 
   const std::string& file() { return file_; }
   pthread_key_t key() { return key_; }
@@ -176,7 +68,7 @@
 
   std::mutex entries_lock_;
   pthread_key_t key_;
-  std::vector<std::unique_ptr<const RecordEntry>> entries_;
+  std::vector<memory_trace::Entry> entries_;
   size_t cur_index_;
   std::string file_;
 
diff --git a/libc/malloc_debug/malloc_debug.cpp b/libc/malloc_debug/malloc_debug.cpp
index 3743852..c183897 100644
--- a/libc/malloc_debug/malloc_debug.cpp
+++ b/libc/malloc_debug/malloc_debug.cpp
@@ -54,6 +54,7 @@
 #include "Config.h"
 #include "DebugData.h"
 #include "LogAllocatorStats.h"
+#include "Nanotime.h"
 #include "Unreachable.h"
 #include "UnwindBacktrace.h"
 #include "backtrace.h"
@@ -70,12 +71,6 @@
 
 const MallocDispatch* g_dispatch;
 
-static inline __always_inline uint64_t Nanotime() {
-  struct timespec t = {};
-  clock_gettime(CLOCK_MONOTONIC, &t);
-  return static_cast<uint64_t>(t.tv_sec) * 1000000000LL + t.tv_nsec;
-}
-
 namespace {
 // A TimedResult contains the result of from malloc end_ns al. functions and the
 // start/end timestamps.
@@ -598,8 +593,13 @@
   TimedResult result = InternalMalloc(size);
 
   if (g_debug->config().options() & RECORD_ALLOCS) {
-    g_debug->record->AddEntry(new MallocEntry(result.getValue<void*>(), size,
-                                              result.GetStartTimeNS(), result.GetEndTimeNS()));
+    g_debug->record->AddEntry(
+        memory_trace::Entry{.tid = gettid(),
+                            .type = memory_trace::MALLOC,
+                            .ptr = reinterpret_cast<uint64_t>(result.getValue<void*>()),
+                            .size = size,
+                            .start_ns = result.GetStartTimeNS(),
+                            .end_ns = result.GetEndTimeNS()});
   }
 
   return result.getValue<void*>();
@@ -687,8 +687,11 @@
   TimedResult result = InternalFree(pointer);
 
   if (g_debug->config().options() & RECORD_ALLOCS) {
-    g_debug->record->AddEntry(
-        new FreeEntry(pointer, result.GetStartTimeNS(), result.GetEndTimeNS()));
+    g_debug->record->AddEntry(memory_trace::Entry{.tid = gettid(),
+                                                  .type = memory_trace::FREE,
+                                                  .ptr = reinterpret_cast<uint64_t>(pointer),
+                                                  .start_ns = result.GetStartTimeNS(),
+                                                  .end_ns = result.GetEndTimeNS()});
   }
 }
 
@@ -771,8 +774,13 @@
     }
 
     if (g_debug->config().options() & RECORD_ALLOCS) {
-      g_debug->record->AddEntry(new MemalignEntry(pointer, bytes, alignment,
-                                                  result.GetStartTimeNS(), result.GetEndTimeNS()));
+      g_debug->record->AddEntry(memory_trace::Entry{.tid = gettid(),
+                                                    .type = memory_trace::MEMALIGN,
+                                                    .ptr = reinterpret_cast<uint64_t>(pointer),
+                                                    .size = bytes,
+                                                    .u.align = alignment,
+                                                    .start_ns = result.GetStartTimeNS(),
+                                                    .end_ns = result.GetEndTimeNS()});
     }
   }
 
@@ -791,11 +799,16 @@
 
   if (pointer == nullptr) {
     TimedResult result = InternalMalloc(bytes);
-    if (g_debug->config().options() & RECORD_ALLOCS) {
-      g_debug->record->AddEntry(new ReallocEntry(result.getValue<void*>(), bytes, nullptr,
-                                                 result.GetStartTimeNS(), result.GetEndTimeNS()));
-    }
     pointer = result.getValue<void*>();
+    if (g_debug->config().options() & RECORD_ALLOCS) {
+      g_debug->record->AddEntry(memory_trace::Entry{.tid = gettid(),
+                                                    .type = memory_trace::REALLOC,
+                                                    .ptr = reinterpret_cast<uint64_t>(pointer),
+                                                    .size = bytes,
+                                                    .u.old_ptr = 0,
+                                                    .start_ns = result.GetStartTimeNS(),
+                                                    .end_ns = result.GetEndTimeNS()});
+    }
     return pointer;
   }
 
@@ -807,8 +820,14 @@
     TimedResult result = InternalFree(pointer);
 
     if (g_debug->config().options() & RECORD_ALLOCS) {
-      g_debug->record->AddEntry(new ReallocEntry(nullptr, bytes, pointer, result.GetStartTimeNS(),
-                                                 result.GetEndTimeNS()));
+      g_debug->record->AddEntry(
+          memory_trace::Entry{.tid = gettid(),
+                              .type = memory_trace::REALLOC,
+                              .ptr = 0,
+                              .size = 0,
+                              .u.old_ptr = reinterpret_cast<uint64_t>(pointer),
+                              .start_ns = result.GetStartTimeNS(),
+                              .end_ns = result.GetEndTimeNS()});
     }
 
     return nullptr;
@@ -905,8 +924,13 @@
   }
 
   if (g_debug->config().options() & RECORD_ALLOCS) {
-    g_debug->record->AddEntry(new ReallocEntry(new_pointer, bytes, pointer, result.GetStartTimeNS(),
-                                               result.GetEndTimeNS()));
+    g_debug->record->AddEntry(memory_trace::Entry{.tid = gettid(),
+                                                  .type = memory_trace::REALLOC,
+                                                  .ptr = reinterpret_cast<uint64_t>(new_pointer),
+                                                  .size = bytes,
+                                                  .u.old_ptr = reinterpret_cast<uint64_t>(pointer),
+                                                  .start_ns = result.GetStartTimeNS(),
+                                                  .end_ns = result.GetEndTimeNS()});
   }
 
   return new_pointer;
@@ -962,8 +986,13 @@
   }
 
   if (g_debug->config().options() & RECORD_ALLOCS) {
-    g_debug->record->AddEntry(
-        new CallocEntry(pointer, nmemb, bytes, result.GetStartTimeNS(), result.GetEndTimeNS()));
+    g_debug->record->AddEntry(memory_trace::Entry{.tid = gettid(),
+                                                  .type = memory_trace::CALLOC,
+                                                  .ptr = reinterpret_cast<uint64_t>(pointer),
+                                                  .size = bytes,
+                                                  .u.n_elements = nmemb,
+                                                  .start_ns = result.GetStartTimeNS(),
+                                                  .end_ns = result.GetEndTimeNS()});
   }
 
   if (pointer != nullptr && g_debug->TrackPointers()) {
diff --git a/libc/platform/bionic/macros.h b/libc/platform/bionic/macros.h
index 93268c1..b2d6f96 100644
--- a/libc/platform/bionic/macros.h
+++ b/libc/platform/bionic/macros.h
@@ -97,3 +97,26 @@
 static inline T* _Nonnull untag_address(T* _Nonnull p) {
   return reinterpret_cast<T*>(untag_address(reinterpret_cast<uintptr_t>(p)));
 }
+
+// MTE globals protects internal and external global variables. One of the main
+// things that MTE globals does is force all global variable accesses to go
+// through the GOT. In the linker though, some global variables are accessed (or
+// address-taken) prior to relocations being processed. Because relocations
+// haven't run yet, the GOT entry hasn't been populated, and this leads to
+// crashes. Thus, any globals used by the linker prior to relocation should be
+// annotated with this attribute, which suppresses tagging of this global
+// variable, restoring the pc-relative address computation.
+//
+// A way to find global variables that need this attribute is to build the
+// linker/libc with `SANITIZE_TARGET=memtag_globals`, push them onto a device
+// (it doesn't have to be MTE capable), and then run an executable using
+// LD_LIBRARY_PATH and using the linker in interpreter mode (e.g.
+// `LD_LIBRARY_PATH=/data/tmp/ /data/tmp/linker64 /data/tmp/my_binary`). A
+// good heuristic is that the global variable is in a file that should be
+// compiled with `-ffreestanding` (but there are global variables there that
+// don't need this attribute).
+#if __has_feature(memtag_globals)
+#define BIONIC_USED_BEFORE_LINKER_RELOCATES __attribute__((no_sanitize("memtag")))
+#else  // __has_feature(memtag_globals)
+#define BIONIC_USED_BEFORE_LINKER_RELOCATES
+#endif  // __has_feature(memtag_globals)
diff --git a/libc/platform/bionic/mte.h b/libc/platform/bionic/mte.h
index 98b3d27..610cb45 100644
--- a/libc/platform/bionic/mte.h
+++ b/libc/platform/bionic/mte.h
@@ -28,6 +28,7 @@
 
 #pragma once
 
+#include <stddef.h>
 #include <sys/auxv.h>
 #include <sys/mman.h>
 #include <sys/prctl.h>
@@ -49,6 +50,36 @@
   return supported;
 }
 
+inline void* get_tagged_address(const void* ptr) {
+#if defined(__aarch64__)
+  if (mte_supported()) {
+    __asm__ __volatile__(".arch_extension mte; ldg %0, [%0]" : "+r"(ptr));
+  }
+#endif  // aarch64
+  return const_cast<void*>(ptr);
+}
+
+// Inserts a random tag tag to `ptr`, using any of the set lower 16 bits in
+// `mask` to exclude the corresponding tag from being generated. Note: This does
+// not tag memory. This generates a pointer to be used with set_memory_tag.
+inline void* insert_random_tag(const void* ptr, __attribute__((unused)) uint64_t mask = 0) {
+#if defined(__aarch64__)
+  if (mte_supported() && ptr) {
+    __asm__ __volatile__(".arch_extension mte; irg %0, %0, %1" : "+r"(ptr) : "r"(mask));
+  }
+#endif  // aarch64
+  return const_cast<void*>(ptr);
+}
+
+// Stores the address tag in `ptr` to memory, at `ptr`.
+inline void set_memory_tag(__attribute__((unused)) void* ptr) {
+#if defined(__aarch64__)
+  if (mte_supported()) {
+    __asm__ __volatile__(".arch_extension mte; stg %0, [%0]" : "+r"(ptr));
+  }
+#endif  // aarch64
+}
+
 #ifdef __aarch64__
 class ScopedDisableMTE {
   size_t prev_tco_;
@@ -86,6 +117,12 @@
   return ptr | ((1ULL << size_cls) << 56ULL);
 }
 
+inline void stack_mte_free_ringbuffer(uintptr_t stack_mte_tls) {
+  size_t size = stack_mte_ringbuffer_size_from_pointer(stack_mte_tls);
+  void* ptr = reinterpret_cast<void*>(stack_mte_tls & ((1ULL << 56ULL) - 1ULL));
+  munmap(ptr, size);
+}
+
 inline void* stack_mte_ringbuffer_allocate(size_t n, const char* name) {
   if (n > 7) return nullptr;
   // Allocation needs to be aligned to 2*size to make the fancy code-gen work.
diff --git a/libc/private/bionic_asm_arm.h b/libc/private/bionic_asm_arm.h
index d8381d3..9ca5f38 100644
--- a/libc/private/bionic_asm_arm.h
+++ b/libc/private/bionic_asm_arm.h
@@ -37,7 +37,7 @@
 
 #pragma once
 
-#define __bionic_asm_align 0
+#define __bionic_asm_align 64
 
 #undef __bionic_asm_custom_entry
 #undef __bionic_asm_custom_end
diff --git a/libc/private/bionic_asm_arm64.h b/libc/private/bionic_asm_arm64.h
index ffc7181..1e907a1 100644
--- a/libc/private/bionic_asm_arm64.h
+++ b/libc/private/bionic_asm_arm64.h
@@ -37,7 +37,7 @@
 
 #pragma once
 
-#define __bionic_asm_align 16
+#define __bionic_asm_align 64
 
 #undef __bionic_asm_function_type
 #define __bionic_asm_function_type %function
diff --git a/libc/private/bionic_constants.h b/libc/private/bionic_constants.h
index 6274fe2..ce484d8 100644
--- a/libc/private/bionic_constants.h
+++ b/libc/private/bionic_constants.h
@@ -16,6 +16,7 @@
 
 #pragma once
 
+#define US_PER_S 1'000'000LL
 #define NS_PER_S 1'000'000'000LL
 
 // Size of the shadow call stack. This can be small because these stacks only
diff --git a/libc/private/bionic_globals.h b/libc/private/bionic_globals.h
index a1bebda..cd6dca9 100644
--- a/libc/private/bionic_globals.h
+++ b/libc/private/bionic_globals.h
@@ -157,6 +157,10 @@
 };
 
 __LIBC_HIDDEN__ libc_shared_globals* __libc_shared_globals();
+__LIBC_HIDDEN__ bool __libc_mte_enabled();
+__LIBC_HIDDEN__ void __libc_init_mte(const memtag_dynamic_entries_t*, const void*, size_t,
+                                     uintptr_t);
+__LIBC_HIDDEN__ void __libc_init_mte_stack(void*);
 __LIBC_HIDDEN__ void __libc_init_fdsan();
 __LIBC_HIDDEN__ void __libc_init_fdtrack();
 __LIBC_HIDDEN__ void __libc_init_profiling_handlers();
diff --git a/libc/private/bionic_lock.h b/libc/private/bionic_lock.h
index 8ed4939..d0c6d5e 100644
--- a/libc/private/bionic_lock.h
+++ b/libc/private/bionic_lock.h
@@ -46,7 +46,7 @@
 
  public:
   void init(bool process_shared) {
-    atomic_init(&state, Unlocked);
+    atomic_store_explicit(&state, Unlocked, memory_order_relaxed);
     this->process_shared = process_shared;
   }
 
diff --git a/libc/private/bionic_time_conversions.h b/libc/private/bionic_time_conversions.h
index c6b3c78..ce7de0d 100644
--- a/libc/private/bionic_time_conversions.h
+++ b/libc/private/bionic_time_conversions.h
@@ -26,8 +26,7 @@
  * SUCH DAMAGE.
  */
 
-#ifndef _BIONIC_TIME_CONVERSIONS_H
-#define _BIONIC_TIME_CONVERSIONS_H
+#pragma once
 
 #include <errno.h>
 #include <time.h>
@@ -35,20 +34,21 @@
 
 #include "private/bionic_constants.h"
 
-__BEGIN_DECLS
+bool timespec_from_timeval(timespec& ts, const timeval& tv);
+void timespec_from_ms(timespec& ts, const int ms);
 
-__LIBC_HIDDEN__ bool timespec_from_timeval(timespec& ts, const timeval& tv);
-__LIBC_HIDDEN__ void timespec_from_ms(timespec& ts, const int ms);
+void timeval_from_timespec(timeval& tv, const timespec& ts);
 
-__LIBC_HIDDEN__ void timeval_from_timespec(timeval& tv, const timespec& ts);
+void monotonic_time_from_realtime_time(timespec& monotonic_time, const timespec& realtime_time);
+void realtime_time_from_monotonic_time(timespec& realtime_time, const timespec& monotonic_time);
 
-__LIBC_HIDDEN__ void monotonic_time_from_realtime_time(timespec& monotonic_time,
-                                                       const timespec& realtime_time);
+static inline int64_t to_ns(const timespec& ts) {
+  return ts.tv_sec * NS_PER_S + ts.tv_nsec;
+}
 
-__LIBC_HIDDEN__ void realtime_time_from_monotonic_time(timespec& realtime_time,
-                                                       const timespec& monotonic_time);
-
-__END_DECLS
+static inline int64_t to_us(const timeval& tv) {
+  return tv.tv_sec * US_PER_S + tv.tv_usec;
+}
 
 static inline int check_timespec(const timespec* ts, bool null_allowed) {
   if (null_allowed && ts == nullptr) {
@@ -76,5 +76,3 @@
   }
 }
 #endif
-
-#endif
diff --git a/libc/stdio/stdio.cpp b/libc/stdio/stdio.cpp
index 37b9665..a5f2f81 100644
--- a/libc/stdio/stdio.cpp
+++ b/libc/stdio/stdio.cpp
@@ -58,8 +58,6 @@
 #include "private/bionic_fortify.h"
 #include "private/thread_private.h"
 
-#include "private/bsd_sys_param.h" // For ALIGN/ALIGNBYTES.
-
 #define	NDYNAMIC 10		/* add ten more whenever necessary */
 
 #define PRINTF_IMPL(expr) \
@@ -135,12 +133,14 @@
 };
 
 static glue* moreglue(int n) {
-  char* data = new char[sizeof(glue) + ALIGNBYTES + n * sizeof(FILE) + n * sizeof(__sfileext)];
+  char* data = new char[sizeof(glue) +
+                        alignof(FILE) + n * sizeof(FILE) +
+                        alignof(__sfileext) + n * sizeof(__sfileext)];
   if (data == nullptr) return nullptr;
 
   glue* g = reinterpret_cast<glue*>(data);
-  FILE* p = reinterpret_cast<FILE*>(ALIGN(data + sizeof(*g)));
-  __sfileext* pext = reinterpret_cast<__sfileext*>(ALIGN(data + sizeof(*g)) + n * sizeof(FILE));
+  FILE* p = reinterpret_cast<FILE*>(__builtin_align_up(g + 1, alignof(FILE)));
+  __sfileext* pext = reinterpret_cast<__sfileext*>(__builtin_align_up(p + n, alignof(__sfileext)));
   g->next = nullptr;
   g->niobs = n;
   g->iobs = p;
diff --git a/libc/system_properties/include/system_properties/prop_area.h b/libc/system_properties/include/system_properties/prop_area.h
index 187ff75..089cf52 100644
--- a/libc/system_properties/include/system_properties/prop_area.h
+++ b/libc/system_properties/include/system_properties/prop_area.h
@@ -102,7 +102,7 @@
   }
 
   prop_area(const uint32_t magic, const uint32_t version) : magic_(magic), version_(version) {
-    atomic_init(&serial_, 0u);
+    atomic_store_explicit(&serial_, 0u, memory_order_relaxed);
     memset(reserved_, 0, sizeof(reserved_));
     // Allocate enough space for the root node.
     bytes_used_ = sizeof(prop_trie_node);
diff --git a/libc/system_properties/prop_info.cpp b/libc/system_properties/prop_info.cpp
index c3bf177..499b36a 100644
--- a/libc/system_properties/prop_info.cpp
+++ b/libc/system_properties/prop_info.cpp
@@ -38,7 +38,7 @@
 prop_info::prop_info(const char* name, uint32_t namelen, const char* value, uint32_t valuelen) {
   memcpy(this->name, name, namelen);
   this->name[namelen] = '\0';
-  atomic_init(&this->serial, valuelen << 24);
+  atomic_store_explicit(&this->serial, valuelen << 24, memory_order_relaxed);
   memcpy(this->value, value, valuelen);
   this->value[valuelen] = '\0';
 }
@@ -48,7 +48,7 @@
   this->name[namelen] = '\0';
 
   auto error_value_len = sizeof(kLongLegacyError) - 1;
-  atomic_init(&this->serial, error_value_len << 24 | kLongFlag);
+  atomic_store_explicit(&this->serial, error_value_len << 24 | kLongFlag, memory_order_relaxed);
   memcpy(this->long_property.error_message, kLongLegacyError, sizeof(kLongLegacyError));
 
   this->long_property.offset = long_offset;
diff --git a/libc/upstream-netbsd/lib/libc/stdlib/bsearch.c b/libc/upstream-netbsd/lib/libc/stdlib/bsearch.c
deleted file mode 100644
index e48fe85..0000000
--- a/libc/upstream-netbsd/lib/libc/stdlib/bsearch.c
+++ /dev/null
@@ -1,85 +0,0 @@
-/*	$NetBSD: bsearch.c,v 1.16 2022/05/31 08:43:14 andvar Exp $	*/
-
-/*
- * Copyright (c) 1990, 1993
- *	The Regents of the University of California.  All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the University nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include <sys/cdefs.h>
-#if defined(LIBC_SCCS) && !defined(lint)
-#if 0
-static char sccsid[] = "@(#)bsearch.c	8.1 (Berkeley) 6/4/93";
-#else
-__RCSID("$NetBSD: bsearch.c,v 1.16 2022/05/31 08:43:14 andvar Exp $");
-#endif
-#endif /* LIBC_SCCS and not lint */
-
-#include <assert.h>
-#include <errno.h>
-#include <stdlib.h>
-
-/*
- * Perform a binary search.
- *
- * The code below is a bit sneaky.  After a comparison fails, we
- * divide the work in half by moving either left or right. If lim
- * is odd, moving left simply involves halving lim: e.g., when lim
- * is 5 we look at item 2, so we change lim to 2 so that we will
- * look at items 0 & 1.  If lim is even, the same applies.  If lim
- * is odd, moving right again involves halving lim, this time moving
- * the base up one item past p: e.g., when lim is 5 we change base
- * to item 3 and make lim 2 so that we will look at items 3 and 4.
- * If lim is even, however, we have to shrink it by one before
- * halving: e.g., when lim is 4, we still looked at item 2, so we
- * have to make lim 3, then halve, obtaining 1, so that we will only
- * look at item 3.
- */
-void *
-bsearch(const void *key, const void *base0, size_t nmemb, size_t size,
-    int (*compar)(const void *, const void *))
-{
-	const char *base = base0;
-	size_t lim;
-	int cmp;
-	const void *p;
-
-	_DIAGASSERT(key != NULL);
-	_DIAGASSERT(base0 != NULL || nmemb == 0);
-	_DIAGASSERT(compar != NULL);
-
-	for (lim = nmemb; lim != 0; lim >>= 1) {
-		p = base + (lim >> 1) * size;
-		cmp = (*compar)(key, p);
-		if (cmp == 0)
-			return __UNCONST(p);
-		if (cmp > 0) {	/* key > p: move right */
-			base = (const char *)p + size;
-			lim--;
-		}		/* else move left */
-	}
-	return (NULL);
-}
diff --git a/libc/upstream-openbsd/android/include/openbsd-compat.h b/libc/upstream-openbsd/android/include/openbsd-compat.h
index cbc52b5..ac6840a 100644
--- a/libc/upstream-openbsd/android/include/openbsd-compat.h
+++ b/libc/upstream-openbsd/android/include/openbsd-compat.h
@@ -25,8 +25,6 @@
 
 #include <sys/random.h> // For getentropy.
 
-#include "private/bsd_sys_param.h"
-
 #define __BEGIN_HIDDEN_DECLS _Pragma("GCC visibility push(hidden)")
 #define __END_HIDDEN_DECLS _Pragma("GCC visibility pop")
 
diff --git a/libc/upstream-openbsd/lib/libc/string/memrchr.c b/libc/upstream-openbsd/lib/libc/string/memrchr.c
deleted file mode 100644
index e123bc1..0000000
--- a/libc/upstream-openbsd/lib/libc/string/memrchr.c
+++ /dev/null
@@ -1,39 +0,0 @@
-/*	$OpenBSD: memrchr.c,v 1.4 2019/01/25 00:19:25 millert Exp $	*/
-
-/*
- * Copyright (c) 2007 Todd C. Miller <millert@openbsd.org>
- *
- * Permission to use, copy, modify, and distribute this software for any
- * purpose with or without fee is hereby granted, provided that the above
- * copyright notice and this permission notice appear in all copies.
- *
- * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
- * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
- * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
- * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
- * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
- * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
- */
-
-#include <string.h>
-
-/*
- * Reverse memchr()
- * Find the last occurrence of 'c' in the buffer 's' of size 'n'.
- */
-void *
-memrchr(const void *s, int c, size_t n)
-{
-	const unsigned char *cp;
-
-	if (n != 0) {
-		cp = (unsigned char *)s + n;
-		do {
-			if (*(--cp) == (unsigned char)c)
-				return((void *)cp);
-		} while (--n != 0);
-	}
-	return(NULL);
-}
-DEF_WEAK(memrchr);
diff --git a/libc/upstream-openbsd/lib/libc/string/strlcat.c b/libc/upstream-openbsd/lib/libc/string/strlcat.c
deleted file mode 100644
index aa3db7ab..0000000
--- a/libc/upstream-openbsd/lib/libc/string/strlcat.c
+++ /dev/null
@@ -1,56 +0,0 @@
-/*	$OpenBSD: strlcat.c,v 1.19 2019/01/25 00:19:25 millert Exp $	*/
-
-/*
- * Copyright (c) 1998, 2015 Todd C. Miller <millert@openbsd.org>
- *
- * Permission to use, copy, modify, and distribute this software for any
- * purpose with or without fee is hereby granted, provided that the above
- * copyright notice and this permission notice appear in all copies.
- *
- * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
- * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
- * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
- * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
- * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
- * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
- */
-
-#include <sys/types.h>
-#include <string.h>
-
-/*
- * Appends src to string dst of size dsize (unlike strncat, dsize is the
- * full size of dst, not space left).  At most dsize-1 characters
- * will be copied.  Always NUL terminates (unless dsize <= strlen(dst)).
- * Returns strlen(src) + MIN(dsize, strlen(initial dst)).
- * If retval >= dsize, truncation occurred.
- */
-size_t
-strlcat(char *dst, const char *src, size_t dsize)
-{
-	const char *odst = dst;
-	const char *osrc = src;
-	size_t n = dsize;
-	size_t dlen;
-
-	/* Find the end of dst and adjust bytes left but don't go past end. */
-	while (n-- != 0 && *dst != '\0')
-		dst++;
-	dlen = dst - odst;
-	n = dsize - dlen;
-
-	if (n-- == 0)
-		return(dlen + strlen(src));
-	while (*src != '\0') {
-		if (n != 0) {
-			*dst++ = *src;
-			n--;
-		}
-		src++;
-	}
-	*dst = '\0';
-
-	return(dlen + (src - osrc));	/* count does not include NUL */
-}
-DEF_WEAK(strlcat);
diff --git a/libc/upstream-openbsd/lib/libc/string/strlcpy.c b/libc/upstream-openbsd/lib/libc/string/strlcpy.c
deleted file mode 100644
index 7e3b9ae..0000000
--- a/libc/upstream-openbsd/lib/libc/string/strlcpy.c
+++ /dev/null
@@ -1,51 +0,0 @@
-/*	$OpenBSD: strlcpy.c,v 1.16 2019/01/25 00:19:25 millert Exp $	*/
-
-/*
- * Copyright (c) 1998, 2015 Todd C. Miller <millert@openbsd.org>
- *
- * Permission to use, copy, modify, and distribute this software for any
- * purpose with or without fee is hereby granted, provided that the above
- * copyright notice and this permission notice appear in all copies.
- *
- * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
- * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
- * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
- * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
- * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
- * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
- */
-
-#include <sys/types.h>
-#include <string.h>
-
-/*
- * Copy string src to buffer dst of size dsize.  At most dsize-1
- * chars will be copied.  Always NUL terminates (unless dsize == 0).
- * Returns strlen(src); if retval >= dsize, truncation occurred.
- */
-size_t
-strlcpy(char *dst, const char *src, size_t dsize)
-{
-	const char *osrc = src;
-	size_t nleft = dsize;
-
-	/* Copy as many bytes as will fit. */
-	if (nleft != 0) {
-		while (--nleft != 0) {
-			if ((*dst++ = *src++) == '\0')
-				break;
-		}
-	}
-
-	/* Not enough room in dst, add NUL and traverse rest of src. */
-	if (nleft == 0) {
-		if (dsize != 0)
-			*dst = '\0';		/* NUL-terminate dst */
-		while (*src++)
-			;
-	}
-
-	return(src - osrc - 1);	/* count does not include NUL */
-}
-DEF_WEAK(strlcpy);
diff --git a/libdl/libdl_android.cpp b/libdl/libdl_android.cpp
index 47a164a..f0959eb 100644
--- a/libdl/libdl_android.cpp
+++ b/libdl/libdl_android.cpp
@@ -59,6 +59,9 @@
 __attribute__((__weak__, visibility("default")))
 struct android_namespace_t* __loader_android_get_exported_namespace(const char* name);
 
+__attribute__((__weak__, visibility("default"))) void __loader_android_set_16kb_appcompat_mode(
+    bool enable_app_compat);
+
 // Proxy calls to bionic loader
 __attribute__((__weak__))
 void android_get_LD_LIBRARY_PATH(char* buffer, size_t buffer_size) {
@@ -115,4 +118,8 @@
   return __loader_android_get_exported_namespace(name);
 }
 
+__attribute__((__weak__)) void android_set_16kb_appcompat_mode(bool enable_app_compat) {
+  __loader_android_set_16kb_appcompat_mode(enable_app_compat);
+}
+
 } // extern "C"
diff --git a/libdl/libdl_android.map.txt b/libdl/libdl_android.map.txt
index 7afcd9c..efbc841 100644
--- a/libdl/libdl_android.map.txt
+++ b/libdl/libdl_android.map.txt
@@ -24,6 +24,7 @@
     android_init_anonymous_namespace; # apex
     android_link_namespaces; # apex
     android_set_application_target_sdk_version; # apex
+    android_set_16kb_appcompat_mode; #apex
   local:
     *;
 };
diff --git a/linker/Android.bp b/linker/Android.bp
index 563cf3d..4863b92 100644
--- a/linker/Android.bp
+++ b/linker/Android.bp
@@ -108,6 +108,12 @@
 
     // We need to access Bionic private headers in the linker.
     include_dirs: ["bionic/libc"],
+
+    sanitize: {
+        // Supporting memtag_globals in the linker would be tricky,
+        // because it relocates itself very early.
+        memtag_globals: false,
+    },
 }
 
 // ========================================================
@@ -184,6 +190,7 @@
         "linker_mapped_file_fragment.cpp",
         "linker_note_gnu_property.cpp",
         "linker_phdr.cpp",
+        "linker_phdr_16kib_compat.cpp",
         "linker_relocate.cpp",
         "linker_sdk_versions.cpp",
         "linker_soinfo.cpp",
@@ -493,6 +500,7 @@
         "linker_mapped_file_fragment.cpp",
         "linker_sdk_versions.cpp",
         "linker_dlwarning.cpp",
+        "linker_phdr_16kib_compat.cpp"
     ],
 
     static_libs: [
diff --git a/linker/dlfcn.cpp b/linker/dlfcn.cpp
index fee19f4..f811d6d 100644
--- a/linker/dlfcn.cpp
+++ b/linker/dlfcn.cpp
@@ -89,6 +89,7 @@
                       const void* caller_addr) __LINKER_PUBLIC__;
 void __loader_add_thread_local_dtor(void* dso_handle) __LINKER_PUBLIC__;
 void __loader_remove_thread_local_dtor(void* dso_handle) __LINKER_PUBLIC__;
+void __loader_android_set_16kb_appcompat_mode(bool enable_app_compat) __LINKER_PUBLIC__;
 libc_shared_globals* __loader_shared_globals() __LINKER_PUBLIC__;
 #if defined(__arm__)
 _Unwind_Ptr __loader_dl_unwind_find_exidx(_Unwind_Ptr pc, int* pcount) __LINKER_PUBLIC__;
@@ -301,6 +302,11 @@
   decrement_dso_handle_reference_counter(dso_handle);
 }
 
+void __loader_android_set_16kb_appcompat_mode(bool enable_app_compat) {
+  ScopedPthreadMutexLocker locker(&g_dl_mutex);
+  set_16kb_appcompat_mode(enable_app_compat);
+}
+
 libc_shared_globals* __loader_shared_globals() {
   return __libc_shared_globals();
 }
@@ -331,6 +337,7 @@
     __libdl_info->gnu_bloom_filter_ = linker_si.gnu_bloom_filter_;
     __libdl_info->gnu_bucket_ = linker_si.gnu_bucket_;
     __libdl_info->gnu_chain_ = linker_si.gnu_chain_;
+    __libdl_info->memtag_dynamic_entries_ = linker_si.memtag_dynamic_entries_;
 
     __libdl_info->ref_count_ = 1;
     __libdl_info->strtab_size_ = linker_si.strtab_size_;
diff --git a/linker/ld_android.cpp b/linker/ld_android.cpp
index 1c03106..c938a16 100644
--- a/linker/ld_android.cpp
+++ b/linker/ld_android.cpp
@@ -55,6 +55,7 @@
 __strong_alias(__loader_add_thread_local_dtor, __internal_linker_error);
 __strong_alias(__loader_remove_thread_local_dtor, __internal_linker_error);
 __strong_alias(__loader_shared_globals, __internal_linker_error);
+__strong_alias(__loader_android_set_16kb_appcompat_mode, __internal_linker_error);
 #if defined(__arm__)
 __strong_alias(__loader_dl_unwind_find_exidx, __internal_linker_error);
 #endif
diff --git a/linker/linker.arm.map b/linker/linker.arm.map
index b805cd6..edfa249 100644
--- a/linker/linker.arm.map
+++ b/linker/linker.arm.map
@@ -25,6 +25,7 @@
     __loader_shared_globals;
     rtld_db_dlactivity;
     __loader_android_handle_signal;
+    __loader_android_set_16kb_appcompat_mode;
   local:
     *;
 };
diff --git a/linker/linker.cpp b/linker/linker.cpp
index bcc2500..8f78915 100644
--- a/linker/linker.cpp
+++ b/linker/linker.cpp
@@ -51,6 +51,7 @@
 #include <android-base/scopeguard.h>
 #include <async_safe/log.h>
 #include <bionic/pthread_internal.h>
+#include <platform/bionic/mte.h>
 
 // Private C library headers.
 
@@ -640,6 +641,11 @@
     si_->set_gap_start(elf_reader.gap_start());
     si_->set_gap_size(elf_reader.gap_size());
     si_->set_should_pad_segments(elf_reader.should_pad_segments());
+    si_->set_should_use_16kib_app_compat(elf_reader.should_use_16kib_app_compat());
+    if (si_->should_use_16kib_app_compat()) {
+      si_->set_compat_relro_start(elf_reader.compat_relro_start());
+      si_->set_compat_relro_size(elf_reader.compat_relro_size());
+    }
 
     return true;
   }
@@ -1692,16 +1698,24 @@
     }
   }
 
+  // The WebView loader uses RELRO sharing in order to promote page sharing of the large RELRO
+  // segment, as it's full of C++ vtables. Because MTE globals, by default, applies random tags to
+  // each global variable, the RELRO segment is polluted and unique for each process. In order to
+  // allow sharing, but still provide some protection, we use deterministic global tagging schemes
+  // for DSOs that are loaded through android_dlopen_ext, such as those loaded by WebView.
+  bool dlext_use_relro =
+      extinfo && extinfo->flags & (ANDROID_DLEXT_WRITE_RELRO | ANDROID_DLEXT_USE_RELRO);
+
   // Step 3: pre-link all DT_NEEDED libraries in breadth first order.
   bool any_memtag_stack = false;
   for (auto&& task : load_tasks) {
     soinfo* si = task->get_soinfo();
-    if (!si->is_linked() && !si->prelink_image()) {
+    if (!si->is_linked() && !si->prelink_image(dlext_use_relro)) {
       return false;
     }
     // si->memtag_stack() needs to be called after si->prelink_image() which populates
     // the dynamic section.
-    if (si->has_min_version(7) && si->memtag_stack()) {
+    if (si->memtag_stack()) {
       any_memtag_stack = true;
       LD_LOG(kLogDlopen,
              "... load_library requesting stack MTE for: realpath=\"%s\", soname=\"%s\"",
@@ -1715,7 +1729,7 @@
     } else {
       // find_library is used by the initial linking step, so we communicate that we
       // want memtag_stack enabled to __libc_init_mte.
-      __libc_shared_globals()->initial_memtag_stack = true;
+      __libc_shared_globals()->initial_memtag_stack_abi = true;
     }
   }
 
@@ -2356,7 +2370,7 @@
         void* tls_block = get_tls_block_for_this_thread(tls_module, /*should_alloc=*/true);
         *symbol = static_cast<char*>(tls_block) + sym->st_value;
       } else {
-        *symbol = reinterpret_cast<void*>(found->resolve_symbol_address(sym));
+        *symbol = get_tagged_address(reinterpret_cast<void*>(found->resolve_symbol_address(sym)));
       }
       failure_guard.Disable();
       LD_LOG(kLogDlsym,
@@ -2786,15 +2800,25 @@
   return true;
 }
 
-static void apply_relr_reloc(ElfW(Addr) offset, ElfW(Addr) load_bias) {
-  ElfW(Addr) address = offset + load_bias;
-  *reinterpret_cast<ElfW(Addr)*>(address) += load_bias;
+static void apply_relr_reloc(ElfW(Addr) offset, ElfW(Addr) load_bias, bool has_memtag_globals) {
+  ElfW(Addr) destination = offset + load_bias;
+  if (!has_memtag_globals) {
+    *reinterpret_cast<ElfW(Addr)*>(destination) += load_bias;
+    return;
+  }
+
+  ElfW(Addr)* tagged_destination =
+      reinterpret_cast<ElfW(Addr)*>(get_tagged_address(reinterpret_cast<void*>(destination)));
+  ElfW(Addr) tagged_value = reinterpret_cast<ElfW(Addr)>(
+      get_tagged_address(reinterpret_cast<void*>(*tagged_destination + load_bias)));
+  *tagged_destination = tagged_value;
 }
 
 // Process relocations in SHT_RELR section (experimental).
 // Details of the encoding are described in this post:
 //   https://groups.google.com/d/msg/generic-abi/bX460iggiKg/Pi9aSwwABgAJ
-bool relocate_relr(const ElfW(Relr)* begin, const ElfW(Relr)* end, ElfW(Addr) load_bias) {
+bool relocate_relr(const ElfW(Relr) * begin, const ElfW(Relr) * end, ElfW(Addr) load_bias,
+                   bool has_memtag_globals) {
   constexpr size_t wordsize = sizeof(ElfW(Addr));
 
   ElfW(Addr) base = 0;
@@ -2805,7 +2829,7 @@
     if ((entry&1) == 0) {
       // Even entry: encodes the offset for next relocation.
       offset = static_cast<ElfW(Addr)>(entry);
-      apply_relr_reloc(offset, load_bias);
+      apply_relr_reloc(offset, load_bias, has_memtag_globals);
       // Set base offset for subsequent bitmap entries.
       base = offset + wordsize;
       continue;
@@ -2816,7 +2840,7 @@
     while (entry != 0) {
       entry >>= 1;
       if ((entry&1) != 0) {
-        apply_relr_reloc(offset, load_bias);
+        apply_relr_reloc(offset, load_bias, has_memtag_globals);
       }
       offset += wordsize;
     }
@@ -2831,7 +2855,7 @@
 // An empty list of soinfos
 static soinfo_list_t g_empty_list;
 
-bool soinfo::prelink_image() {
+bool soinfo::prelink_image(bool dlext_use_relro) {
   if (flags_ & FLAG_PRELINKED) return true;
   /* Extract dynamic section */
   ElfW(Word) dynamic_flags = 0;
@@ -3320,6 +3344,18 @@
   // it each time we look up a symbol with a version.
   if (!validate_verdef_section(this)) return false;
 
+  // MTE globals requires remapping data segments with PROT_MTE as anonymous mappings, because file
+  // based mappings may not be backed by tag-capable memory (see "MAP_ANONYMOUS" on
+  // https://www.kernel.org/doc/html/latest/arch/arm64/memory-tagging-extension.html). This is only
+  // done if the binary has MTE globals (evidenced by the dynamic table entries), as it destroys
+  // page sharing. It's also only done on devices that support MTE, because the act of remapping
+  // pages is unnecessary on non-MTE devices (where we might still run MTE-globals enabled code).
+  if (should_tag_memtag_globals() &&
+      remap_memtag_globals_segments(phdr, phnum, base) == 0) {
+    tag_globals(dlext_use_relro);
+    protect_memtag_globals_ro_segments(phdr, phnum, base);
+  }
+
   flags_ |= FLAG_PRELINKED;
   return true;
 }
@@ -3361,7 +3397,8 @@
                               "\"%s\" has text relocations",
                               get_realpath());
     add_dlwarning(get_realpath(), "text relocations");
-    if (phdr_table_unprotect_segments(phdr, phnum, load_bias, should_pad_segments_) < 0) {
+    if (phdr_table_unprotect_segments(phdr, phnum, load_bias, should_pad_segments_,
+                                      should_use_16kib_app_compat_) < 0) {
       DL_ERR("can't unprotect loadable segments for \"%s\": %m", get_realpath());
       return false;
     }
@@ -3377,7 +3414,8 @@
 #if !defined(__LP64__)
   if (has_text_relocations) {
     // All relocations are done, we can protect our segments back to read-only.
-    if (phdr_table_protect_segments(phdr, phnum, load_bias, should_pad_segments_) < 0) {
+    if (phdr_table_protect_segments(phdr, phnum, load_bias, should_pad_segments_,
+                                    should_use_16kib_app_compat_) < 0) {
       DL_ERR("can't protect segments for \"%s\": %m", get_realpath());
       return false;
     }
@@ -3390,6 +3428,13 @@
     return false;
   }
 
+  if (should_tag_memtag_globals()) {
+    std::list<std::string>* vma_names_ptr = vma_names();
+    // should_tag_memtag_globals -> __aarch64__ -> vma_names() != nullptr
+    CHECK(vma_names_ptr);
+    name_memtag_globals_segments(phdr, phnum, base, get_realpath(), vma_names_ptr);
+  }
+
   /* Handle serializing/sharing the RELRO segment */
   if (extinfo && (extinfo->flags & ANDROID_DLEXT_WRITE_RELRO)) {
     if (phdr_table_serialize_gnu_relro(phdr, phnum, load_bias,
@@ -3412,13 +3457,70 @@
 }
 
 bool soinfo::protect_relro() {
-  if (phdr_table_protect_gnu_relro(phdr, phnum, load_bias, should_pad_segments_) < 0) {
-    DL_ERR("can't enable GNU RELRO protection for \"%s\": %m", get_realpath());
-    return false;
+  if (should_use_16kib_app_compat_) {
+    if (phdr_table_protect_gnu_relro_16kib_compat(compat_relro_start_, compat_relro_size_) < 0) {
+      DL_ERR("can't enable COMPAT GNU RELRO protection for \"%s\": %s", get_realpath(),
+             strerror(errno));
+      return false;
+    }
+  } else {
+    if (phdr_table_protect_gnu_relro(phdr, phnum, load_bias, should_pad_segments_,
+                                     should_use_16kib_app_compat_) < 0) {
+      DL_ERR("can't enable GNU RELRO protection for \"%s\": %m", get_realpath());
+      return false;
+    }
   }
   return true;
 }
 
+// https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst#global-variable-tagging
+void soinfo::tag_globals(bool dlext_use_relro) {
+  if (is_linked()) return;
+  if (flags_ & FLAG_GLOBALS_TAGGED) return;
+  flags_ |= FLAG_GLOBALS_TAGGED;
+
+  constexpr size_t kTagGranuleSize = 16;
+  const uint8_t* descriptor_stream = reinterpret_cast<const uint8_t*>(memtag_globals());
+
+  if (memtag_globalssz() == 0) {
+    DL_ERR("Invalid memtag descriptor pool size: %zu", memtag_globalssz());
+  }
+
+  uint64_t addr = load_bias;
+  uleb128_decoder decoder(descriptor_stream, memtag_globalssz());
+  // Don't ever generate tag zero, to easily distinguish between tagged and
+  // untagged globals in register/tag dumps.
+  uint64_t last_tag_mask = 1;
+  uint64_t last_tag = 1;
+  constexpr uint64_t kDistanceReservedBits = 3;
+
+  while (decoder.has_bytes()) {
+    uint64_t value = decoder.pop_front();
+    uint64_t distance = (value >> kDistanceReservedBits) * kTagGranuleSize;
+    uint64_t ngranules = value & ((1 << kDistanceReservedBits) - 1);
+    if (ngranules == 0) {
+      ngranules = decoder.pop_front() + 1;
+    }
+
+    addr += distance;
+    void* tagged_addr;
+    if (dlext_use_relro) {
+      tagged_addr = reinterpret_cast<void*>(addr | (last_tag++ << 56));
+      if (last_tag > (1 << kTagGranuleSize)) last_tag = 1;
+    } else {
+      tagged_addr = insert_random_tag(reinterpret_cast<void*>(addr), last_tag_mask);
+      uint64_t tag = (reinterpret_cast<uint64_t>(tagged_addr) >> 56) & 0x0f;
+      last_tag_mask = 1 | (1 << tag);
+    }
+
+    for (size_t k = 0; k < ngranules; k++) {
+      auto* granule = static_cast<uint8_t*>(tagged_addr) + k * kTagGranuleSize;
+      set_memory_tag(static_cast<void*>(granule));
+    }
+    addr += ngranules * kTagGranuleSize;
+  }
+}
+
 static std::vector<android_namespace_t*> init_default_namespace_no_config(bool is_asan, bool is_hwasan) {
   g_default_namespace.set_isolated(false);
   auto default_ld_paths = is_asan ? kAsanDefaultLdPaths : (
diff --git a/linker/linker.generic.map b/linker/linker.generic.map
index 4d7f236..2beae65 100644
--- a/linker/linker.generic.map
+++ b/linker/linker.generic.map
@@ -24,6 +24,7 @@
     __loader_shared_globals;
     rtld_db_dlactivity;
     __loader_android_handle_signal;
+    __loader_android_set_16kb_appcompat_mode;
   local:
     *;
 };
diff --git a/linker/linker.h b/linker/linker.h
index ac2222d..7afa0d7 100644
--- a/linker/linker.h
+++ b/linker/linker.h
@@ -108,6 +108,9 @@
 
 bool get_transparent_hugepages_supported();
 
+void set_16kb_appcompat_mode(bool enable_app_compat);
+bool get_16kb_appcompat_mode();
+
 enum {
   /* A regular namespace is the namespace with a custom search path that does
    * not impose any restrictions on the location of native libraries.
@@ -179,7 +182,8 @@
 int get_application_target_sdk_version();
 ElfW(Versym) find_verdef_version_index(const soinfo* si, const version_info* vi);
 bool validate_verdef_section(const soinfo* si);
-bool relocate_relr(const ElfW(Relr)* begin, const ElfW(Relr)* end, ElfW(Addr) load_bias);
+bool relocate_relr(const ElfW(Relr) * begin, const ElfW(Relr) * end, ElfW(Addr) load_bias,
+                   bool has_memtag_globals);
 
 struct platform_properties {
 #if defined(__aarch64__)
diff --git a/linker/linker_main.cpp b/linker/linker_main.cpp
index 6ccd75b..f65f82d 100644
--- a/linker/linker_main.cpp
+++ b/linker/linker_main.cpp
@@ -46,6 +46,7 @@
 #include "linker_tls.h"
 #include "linker_utils.h"
 
+#include "platform/bionic/macros.h"
 #include "private/KernelArgumentBlock.h"
 #include "private/bionic_call_ifunc_resolver.h"
 #include "private/bionic_globals.h"
@@ -71,7 +72,9 @@
 static void set_bss_vma_name(soinfo* si);
 
 void __libc_init_mte(const memtag_dynamic_entries_t* memtag_dynamic_entries, const void* phdr_start,
-                     size_t phdr_count, uintptr_t load_bias, void* stack_top);
+                     size_t phdr_count, uintptr_t load_bias);
+
+void __libc_init_mte_stack(void* stack_top);
 
 static void __linker_cannot_link(const char* argv0) {
   __linker_error("CANNOT LINK EXECUTABLE \"%s\": %s", argv0, linker_get_error_buffer());
@@ -365,13 +368,16 @@
   init_link_map_head(*solinker);
 
 #if defined(__aarch64__)
+  __libc_init_mte(somain->memtag_dynamic_entries(), somain->phdr, somain->phnum, somain->load_bias);
+
   if (exe_to_load == nullptr) {
     // Kernel does not add PROT_BTI to executable pages of the loaded ELF.
     // Apply appropriate protections here if it is needed.
     auto note_gnu_property = GnuPropertySection(somain);
     if (note_gnu_property.IsBTICompatible() &&
-        (phdr_table_protect_segments(somain->phdr, somain->phnum, somain->load_bias,
-                                     somain->should_pad_segments(), &note_gnu_property) < 0)) {
+        (phdr_table_protect_segments(
+             somain->phdr, somain->phnum, somain->load_bias, somain->should_pad_segments(),
+             somain->should_use_16kib_app_compat(), &note_gnu_property) < 0)) {
       __linker_error("error: can't protect segments for \"%s\": %m", exe_info.path.c_str());
     }
   }
@@ -464,8 +470,7 @@
 #if defined(__aarch64__)
   // This has to happen after the find_libraries, which will have collected any possible
   // libraries that request memtag_stack in the dynamic section.
-  __libc_init_mte(somain->memtag_dynamic_entries(), somain->phdr, somain->phnum, somain->load_bias,
-                  args.argv);
+  __libc_init_mte_stack(args.argv);
 #endif
 
   linker_finalize_static_tls();
@@ -624,8 +629,13 @@
     // Apply RELR relocations first so that the GOT is initialized for ifunc
     // resolvers.
     if (relr && relrsz) {
+      // Nothing has tagged the memtag globals here, so it is pointless either
+      // way to handle them, the tags will be zero anyway.
+      // That is moot though, because the linker does not use memtag_globals
+      // in the first place.
       relocate_relr(reinterpret_cast<ElfW(Relr*)>(ehdr + relr),
-                    reinterpret_cast<ElfW(Relr*)>(ehdr + relr + relrsz), ehdr);
+                    reinterpret_cast<ElfW(Relr*)>(ehdr + relr + relrsz), ehdr,
+                    /*has_memtag_globals=*/ false);
     }
     if (pltrel && pltrelsz) {
       call_ifunc_resolvers_for_section(reinterpret_cast<RelType*>(ehdr + pltrel),
@@ -645,6 +655,16 @@
   }
 }
 
+// Remapping MTE globals segments happens before the linker relocates itself, and so can't use
+// memcpy() from string.h. This function is compiled with -ffreestanding.
+void linker_memcpy(void* dst, const void* src, size_t n) {
+  char* dst_bytes = reinterpret_cast<char*>(dst);
+  const char* src_bytes = reinterpret_cast<const char*>(src);
+  for (size_t i = 0; i < n; ++i) {
+    dst_bytes[i] = src_bytes[i];
+  }
+}
+
 // Detect an attempt to run the linker on itself. e.g.:
 //   /system/bin/linker64 /system/bin/linker64
 // Use priority-1 to run this constructor before other constructors.
@@ -722,6 +742,8 @@
   tmp_linker_so.set_linker_flag();
 
   if (!tmp_linker_so.prelink_image()) __linker_cannot_link(args.argv[0]);
+  // There is special logic in soinfo::relocate to avoid duplicating the
+  // relocations we did in relocate_linker().
   if (!tmp_linker_so.link_image(SymbolLookupList(&tmp_linker_so), &tmp_linker_so, nullptr, nullptr)) __linker_cannot_link(args.argv[0]);
 
   return __linker_init_post_relocation(args, tmp_linker_so);
diff --git a/linker/linker_main.h b/linker/linker_main.h
index 724f43c..ffbcf0f 100644
--- a/linker/linker_main.h
+++ b/linker/linker_main.h
@@ -70,3 +70,5 @@
 soinfo* solist_get_head();
 soinfo* solist_get_somain();
 soinfo* solist_get_vdso();
+
+void linker_memcpy(void* dst, const void* src, size_t n);
diff --git a/linker/linker_phdr.cpp b/linker/linker_phdr.cpp
index b7db4cd..e5369ac 100644
--- a/linker/linker_phdr.cpp
+++ b/linker/linker_phdr.cpp
@@ -37,9 +37,12 @@
 #include <unistd.h>
 
 #include "linker.h"
+#include "linker_debug.h"
 #include "linker_dlwarning.h"
 #include "linker_globals.h"
-#include "linker_debug.h"
+#include "linker_logger.h"
+#include "linker_main.h"
+#include "linker_soinfo.h"
 #include "linker_utils.h"
 
 #include "private/bionic_asm_note.h"
@@ -47,6 +50,7 @@
 #include "private/elf_note.h"
 
 #include <android-base/file.h>
+#include <android-base/properties.h>
 
 static int GetTargetElfMachine() {
 #if defined(__arm__)
@@ -139,11 +143,6 @@
 
  **/
 
-#define MAYBE_MAP_FLAG(x, from, to)  (((x) & (from)) ? (to) : 0)
-#define PFLAGS_TO_PROT(x)            (MAYBE_MAP_FLAG((x), PF_X, PROT_EXEC) | \
-                                      MAYBE_MAP_FLAG((x), PF_R, PROT_READ) | \
-                                      MAYBE_MAP_FLAG((x), PF_W, PROT_WRITE))
-
 static const size_t kPageSize = page_size();
 
 /*
@@ -182,6 +181,15 @@
     did_read_ = true;
   }
 
+  if (kPageSize == 0x4000 && phdr_table_get_minimum_alignment(phdr_table_, phdr_num_) == 0x1000) {
+    // This prop needs to be read on 16KiB devices for each ELF where min_palign is 4KiB.
+    // It cannot be cached since the developer may toggle app compat on/off.
+    // This check will be removed once app compat is made the default on 16KiB devices.
+    should_use_16kib_app_compat_ =
+        ::android::base::GetBoolProperty("bionic.linker.16kb.app_compat.enabled", false) ||
+        get_16kb_appcompat_mode();
+  }
+
   return did_read_;
 }
 
@@ -197,8 +205,9 @@
 #if defined(__aarch64__)
     // For Armv8.5-A loaded executable segments may require PROT_BTI.
     if (note_gnu_property_.IsBTICompatible()) {
-      did_load_ = (phdr_table_protect_segments(phdr_table_, phdr_num_, load_bias_,
-                                               should_pad_segments_, &note_gnu_property_) == 0);
+      did_load_ =
+          (phdr_table_protect_segments(phdr_table_, phdr_num_, load_bias_, should_pad_segments_,
+                                       should_use_16kib_app_compat_, &note_gnu_property_) == 0);
     }
 #endif
   }
@@ -690,6 +699,13 @@
     return false;
   }
 
+  if (should_use_16kib_app_compat_) {
+    // Reserve additional space for aligning the permission boundary in compat loading
+    // Up to kPageSize-kCompatPageSize additional space is needed, but reservation
+    // is done with mmap which gives kPageSize multiple-sized reservations.
+    load_size_ += kPageSize;
+  }
+
   uint8_t* addr = reinterpret_cast<uint8_t*>(min_vaddr);
   void* start;
 
@@ -725,13 +741,21 @@
 
   load_start_ = start;
   load_bias_ = reinterpret_cast<uint8_t*>(start) - addr;
+
+  if (should_use_16kib_app_compat_) {
+    // In compat mode make the initial mapping RW since the ELF contents will be read
+    // into it; instead of mapped over it.
+    mprotect(reinterpret_cast<void*>(start), load_size_, PROT_READ | PROT_WRITE);
+  }
+
   return true;
 }
 
 /*
- * Returns true if the kernel supports page size migration, else false.
+ * Returns true if the kernel supports page size migration for this process.
  */
 bool page_size_migration_supported() {
+#if defined(__LP64__)
   static bool pgsize_migration_enabled = []() {
     std::string enabled;
     if (!android::base::ReadFileToString("/sys/kernel/mm/pgsize_migration/enabled", &enabled)) {
@@ -740,6 +764,9 @@
     return enabled.find("1") != std::string::npos;
   }();
   return pgsize_migration_enabled;
+#else
+  return false;
+#endif
 }
 
 // Find the ELF note of type NT_ANDROID_TYPE_PAD_SEGMENT and check that the desc value is 1.
@@ -808,8 +835,15 @@
 }
 
 static inline void _extend_load_segment_vma(const ElfW(Phdr)* phdr_table, size_t phdr_count,
-                                             size_t phdr_idx, ElfW(Addr)* p_memsz,
-                                             ElfW(Addr)* p_filesz, bool should_pad_segments) {
+                                            size_t phdr_idx, ElfW(Addr)* p_memsz,
+                                            ElfW(Addr)* p_filesz, bool should_pad_segments,
+                                            bool should_use_16kib_app_compat) {
+  // NOTE: Segment extension is only applicable where the ELF's max-page-size > runtime page size;
+  // to save kernel VMA slab memory. 16KiB compat mode is the exact opposite scenario.
+  if (should_use_16kib_app_compat) {
+    return;
+  }
+
   const ElfW(Phdr)* phdr = &phdr_table[phdr_idx];
   const ElfW(Phdr)* next = nullptr;
   size_t next_idx = phdr_idx + 1;
@@ -879,6 +913,13 @@
 }
 
 void ElfReader::ZeroFillSegment(const ElfW(Phdr)* phdr) {
+  // NOTE: In 16KiB app compat mode, the ELF mapping is anonymous, meaning that
+  // RW segments are COW-ed from the kernel's zero page. So there is no need to
+  // explicitly zero-fill until the last page's limit.
+  if (should_use_16kib_app_compat_) {
+    return;
+  }
+
   ElfW(Addr) seg_start = phdr->p_vaddr + load_bias_;
   uint64_t unextended_seg_file_end = seg_start + phdr->p_filesz;
 
@@ -898,6 +939,12 @@
 }
 
 void ElfReader::DropPaddingPages(const ElfW(Phdr)* phdr, uint64_t seg_file_end) {
+  // NOTE: Padding pages are only applicable where the ELF's max-page-size > runtime page size;
+  // 16KiB compat mode is the exact opposite scenario.
+  if (should_use_16kib_app_compat_) {
+    return;
+  }
+
   ElfW(Addr) seg_start = phdr->p_vaddr + load_bias_;
   uint64_t unextended_seg_file_end = seg_start + phdr->p_filesz;
 
@@ -926,6 +973,12 @@
 
 bool ElfReader::MapBssSection(const ElfW(Phdr)* phdr, ElfW(Addr) seg_page_end,
                               ElfW(Addr) seg_file_end) {
+  // NOTE: We do not need to handle .bss in 16KiB compat mode since the mapping
+  // reservation is anonymous and RW to begin with.
+  if (should_use_16kib_app_compat_) {
+    return true;
+  }
+
   // seg_file_end is now the first page address after the file content.
   seg_file_end = page_end(seg_file_end);
 
@@ -952,15 +1005,27 @@
 }
 
 bool ElfReader::LoadSegments() {
+  // NOTE: The compat(legacy) page size (4096) must be used when aligning
+  // the 4KiB segments for loading in compat mode. The larger 16KiB page size
+  // will lead to overwriting adjacent segments since the ELF's segment(s)
+  // are not 16KiB aligned.
+  size_t seg_align = should_use_16kib_app_compat_ ? kCompatPageSize : kPageSize;
+
   size_t min_palign = phdr_table_get_minimum_alignment(phdr_table_, phdr_num_);
-  // Only enforce this on 16 KB systems. Apps may rely on undefined behavior
-  // here on 4 KB systems, which is the norm before this change is introduced.
-  if (kPageSize >= 16384 && min_palign < kPageSize) {
+  // Only enforce this on 16 KB systems with app compat disabled.
+  // Apps may rely on undefined behavior here on 4 KB systems,
+  // which is the norm before this change is introduced
+  if (kPageSize >= 16384 && min_palign < kPageSize && !should_use_16kib_app_compat_) {
     DL_ERR("\"%s\" program alignment (%zu) cannot be smaller than system page size (%zu)",
            name_.c_str(), min_palign, kPageSize);
     return false;
   }
 
+  if (!Setup16KiBAppCompat()) {
+    DL_ERR("\"%s\" failed to setup 16KiB App Compat", name_.c_str());
+    return false;
+  }
+
   for (size_t i = 0; i < phdr_num_; ++i) {
     const ElfW(Phdr)* phdr = &phdr_table_[i];
 
@@ -970,13 +1035,14 @@
 
     ElfW(Addr) p_memsz = phdr->p_memsz;
     ElfW(Addr) p_filesz = phdr->p_filesz;
-    _extend_load_segment_vma(phdr_table_, phdr_num_, i, &p_memsz, &p_filesz, should_pad_segments_);
+    _extend_load_segment_vma(phdr_table_, phdr_num_, i, &p_memsz, &p_filesz, should_pad_segments_,
+                             should_use_16kib_app_compat_);
 
     // Segment addresses in memory.
     ElfW(Addr) seg_start = phdr->p_vaddr + load_bias_;
     ElfW(Addr) seg_end = seg_start + p_memsz;
 
-    ElfW(Addr) seg_page_end = page_end(seg_end);
+    ElfW(Addr) seg_page_end = align_up(seg_end, seg_align);
 
     ElfW(Addr) seg_file_end = seg_start + p_filesz;
 
@@ -984,7 +1050,7 @@
     ElfW(Addr) file_start = phdr->p_offset;
     ElfW(Addr) file_end = file_start + p_filesz;
 
-    ElfW(Addr) file_page_start = page_start(file_start);
+    ElfW(Addr) file_page_start = align_down(file_start, seg_align);
     ElfW(Addr) file_length = file_end - file_page_start;
 
     if (file_size_ <= 0) {
@@ -1017,8 +1083,14 @@
       }
 
       // Pass the file_length, since it may have been extended by _extend_load_segment_vma().
-      if (!MapSegment(i, file_length)) {
-        return false;
+      if (should_use_16kib_app_compat_) {
+        if (!CompatMapSegment(i, file_length)) {
+          return false;
+        }
+      } else {
+        if (!MapSegment(i, file_length)) {
+          return false;
+        }
       }
     }
 
@@ -1039,7 +1111,7 @@
  */
 static int _phdr_table_set_load_prot(const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                      ElfW(Addr) load_bias, int extra_prot_flags,
-                                     bool should_pad_segments) {
+                                     bool should_pad_segments, bool should_use_16kib_app_compat) {
   for (size_t i = 0; i < phdr_count; ++i) {
     const ElfW(Phdr)* phdr = &phdr_table[i];
 
@@ -1049,7 +1121,8 @@
 
     ElfW(Addr) p_memsz = phdr->p_memsz;
     ElfW(Addr) p_filesz = phdr->p_filesz;
-    _extend_load_segment_vma(phdr_table, phdr_count, i, &p_memsz, &p_filesz, should_pad_segments);
+    _extend_load_segment_vma(phdr_table, phdr_count, i, &p_memsz, &p_filesz, should_pad_segments,
+                             should_use_16kib_app_compat);
 
     ElfW(Addr) seg_page_start = page_start(phdr->p_vaddr + load_bias);
     ElfW(Addr) seg_page_end = page_end(phdr->p_vaddr + p_memsz + load_bias);
@@ -1088,12 +1161,14 @@
  *   phdr_count  -> number of entries in tables
  *   load_bias   -> load bias
  *   should_pad_segments -> Are segments extended to avoid gaps in the memory map
+ *   should_use_16kib_app_compat -> Is the ELF being loaded in 16KiB app compat mode.
  *   prop        -> GnuPropertySection or nullptr
  * Return:
  *   0 on success, -1 on failure (error code in errno).
  */
 int phdr_table_protect_segments(const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                 ElfW(Addr) load_bias, bool should_pad_segments,
+                                bool should_use_16kib_app_compat,
                                 const GnuPropertySection* prop __unused) {
   int prot = 0;
 #if defined(__aarch64__)
@@ -1101,7 +1176,127 @@
     prot |= PROT_BTI;
   }
 #endif
-  return _phdr_table_set_load_prot(phdr_table, phdr_count, load_bias, prot, should_pad_segments);
+  return _phdr_table_set_load_prot(phdr_table, phdr_count, load_bias, prot, should_pad_segments,
+                                   should_use_16kib_app_compat);
+}
+
+static bool segment_needs_memtag_globals_remapping(const ElfW(Phdr) * phdr) {
+  // For now, MTE globals is only supported on writeable data segments.
+  return phdr->p_type == PT_LOAD && !(phdr->p_flags & PF_X) && (phdr->p_flags & PF_W);
+}
+
+/* When MTE globals are requested by the binary, and when the hardware supports
+ * it, remap the executable's PT_LOAD data pages to have PROT_MTE.
+ *
+ * Returns 0 on success, -1 on failure (error code in errno).
+ */
+int remap_memtag_globals_segments(const ElfW(Phdr) * phdr_table __unused,
+                                  size_t phdr_count __unused, ElfW(Addr) load_bias __unused) {
+#if defined(__aarch64__)
+  for (const ElfW(Phdr)* phdr = phdr_table; phdr < phdr_table + phdr_count; phdr++) {
+    if (!segment_needs_memtag_globals_remapping(phdr)) {
+      continue;
+    }
+
+    uintptr_t seg_page_start = page_start(phdr->p_vaddr) + load_bias;
+    uintptr_t seg_page_end = page_end(phdr->p_vaddr + phdr->p_memsz) + load_bias;
+    size_t seg_page_aligned_size = seg_page_end - seg_page_start;
+
+    int prot = PFLAGS_TO_PROT(phdr->p_flags);
+    // For anonymous private mappings, it may be possible to simply mprotect()
+    // the PROT_MTE flag over the top. For file-based mappings, this will fail,
+    // and we'll need to fall back. We also allow PROT_WRITE here to allow
+    // writing memory tags (in `soinfo::tag_globals()`), and set these sections
+    // back to read-only after tags are applied (similar to RELRO).
+    prot |= PROT_MTE;
+    if (mprotect(reinterpret_cast<void*>(seg_page_start), seg_page_aligned_size,
+                 prot | PROT_WRITE) == 0) {
+      continue;
+    }
+
+    void* mapping_copy = mmap(nullptr, seg_page_aligned_size, PROT_READ | PROT_WRITE,
+                              MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+    linker_memcpy(mapping_copy, reinterpret_cast<void*>(seg_page_start), seg_page_aligned_size);
+
+    void* seg_addr = mmap(reinterpret_cast<void*>(seg_page_start), seg_page_aligned_size,
+                          prot | PROT_WRITE, MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+    if (seg_addr == MAP_FAILED) return -1;
+
+    linker_memcpy(seg_addr, mapping_copy, seg_page_aligned_size);
+    munmap(mapping_copy, seg_page_aligned_size);
+  }
+#endif  // defined(__aarch64__)
+  return 0;
+}
+
+void protect_memtag_globals_ro_segments(const ElfW(Phdr) * phdr_table __unused,
+                                        size_t phdr_count __unused, ElfW(Addr) load_bias __unused) {
+#if defined(__aarch64__)
+  for (const ElfW(Phdr)* phdr = phdr_table; phdr < phdr_table + phdr_count; phdr++) {
+    int prot = PFLAGS_TO_PROT(phdr->p_flags);
+    if (!segment_needs_memtag_globals_remapping(phdr) || (prot & PROT_WRITE)) {
+      continue;
+    }
+
+    prot |= PROT_MTE;
+
+    uintptr_t seg_page_start = page_start(phdr->p_vaddr) + load_bias;
+    uintptr_t seg_page_end = page_end(phdr->p_vaddr + phdr->p_memsz) + load_bias;
+    size_t seg_page_aligned_size = seg_page_end - seg_page_start;
+    mprotect(reinterpret_cast<void*>(seg_page_start), seg_page_aligned_size, prot);
+  }
+#endif  // defined(__aarch64__)
+}
+
+void name_memtag_globals_segments(const ElfW(Phdr) * phdr_table, size_t phdr_count,
+                                  ElfW(Addr) load_bias, const char* soname,
+                                  std::list<std::string>* vma_names) {
+  for (const ElfW(Phdr)* phdr = phdr_table; phdr < phdr_table + phdr_count; phdr++) {
+    if (!segment_needs_memtag_globals_remapping(phdr)) {
+      continue;
+    }
+
+    uintptr_t seg_page_start = page_start(phdr->p_vaddr) + load_bias;
+    uintptr_t seg_page_end = page_end(phdr->p_vaddr + phdr->p_memsz) + load_bias;
+    size_t seg_page_aligned_size = seg_page_end - seg_page_start;
+
+    // For file-based mappings that we're now forcing to be anonymous mappings, set the VMA name to
+    // make debugging easier.
+    // Once we are targeting only devices that run kernel 5.10 or newer (and thus include
+    // https://android-review.git.corp.google.com/c/kernel/common/+/1934723 which causes the
+    // VMA_ANON_NAME to be copied into the kernel), we can get rid of the storage here.
+    // For now, that is not the case:
+    // https://source.android.com/docs/core/architecture/kernel/android-common#compatibility-matrix
+    constexpr int kVmaNameLimit = 80;
+    std::string& vma_name = vma_names->emplace_back(kVmaNameLimit, '\0');
+    int full_vma_length =
+        async_safe_format_buffer(vma_name.data(), kVmaNameLimit, "mt:%s+%" PRIxPTR, soname,
+                                 page_start(phdr->p_vaddr)) +
+        /* include the null terminator */ 1;
+    // There's an upper limit of 80 characters, including the null terminator, in the anonymous VMA
+    // name. If we run over that limit, we end up truncating the segment offset and parts of the
+    // DSO's name, starting on the right hand side of the basename. Because the basename is the most
+    // important thing, chop off the soname from the left hand side first.
+    //
+    // Example (with '#' as the null terminator):
+    //   - "mt:/data/nativetest64/bionic-unit-tests/bionic-loader-test-libs/libdlext_test.so+e000#"
+    //     is a `full_vma_length` == 86.
+    //
+    // We need to left-truncate (86 - 80) 6 characters from the soname, plus the
+    // `vma_truncation_prefix`, so 9 characters total.
+    if (full_vma_length > kVmaNameLimit) {
+      const char vma_truncation_prefix[] = "...";
+      int soname_truncated_bytes =
+          full_vma_length - kVmaNameLimit + sizeof(vma_truncation_prefix) - 1;
+      async_safe_format_buffer(vma_name.data(), kVmaNameLimit, "mt:%s%s+%" PRIxPTR,
+                               vma_truncation_prefix, soname + soname_truncated_bytes,
+                               page_start(phdr->p_vaddr));
+    }
+    if (prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, reinterpret_cast<void*>(seg_page_start),
+              seg_page_aligned_size, vma_name.data()) != 0) {
+      DL_WARN("Failed to rename memtag global segment: %m");
+    }
+  }
 }
 
 /* Change the protection of all loaded segments in memory to writable.
@@ -1118,20 +1313,22 @@
  *   phdr_count  -> number of entries in tables
  *   load_bias   -> load bias
  *   should_pad_segments -> Are segments extended to avoid gaps in the memory map
+ *   should_use_16kib_app_compat -> Is the ELF being loaded in 16KiB app compat mode.
  * Return:
  *   0 on success, -1 on failure (error code in errno).
  */
-int phdr_table_unprotect_segments(const ElfW(Phdr)* phdr_table,
-                                  size_t phdr_count, ElfW(Addr) load_bias,
-                                  bool should_pad_segments) {
+int phdr_table_unprotect_segments(const ElfW(Phdr)* phdr_table, size_t phdr_count,
+                                  ElfW(Addr) load_bias, bool should_pad_segments,
+                                  bool should_use_16kib_app_compat) {
   return _phdr_table_set_load_prot(phdr_table, phdr_count, load_bias, PROT_WRITE,
-                                   should_pad_segments);
+                                   should_pad_segments, should_use_16kib_app_compat);
 }
 
 static inline void _extend_gnu_relro_prot_end(const ElfW(Phdr)* relro_phdr,
                                               const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                               ElfW(Addr) load_bias, ElfW(Addr)* seg_page_end,
-                                              bool should_pad_segments) {
+                                              bool should_pad_segments,
+                                              bool should_use_16kib_app_compat) {
   // Find the index and phdr of the LOAD containing the GNU_RELRO segment
   for (size_t index = 0; index < phdr_count; ++index) {
     const ElfW(Phdr)* phdr = &phdr_table[index];
@@ -1179,7 +1376,7 @@
       // mprotect will only RO protect a part of the extended RW LOAD segment, which
       // will leave an extra split RW VMA (the gap).
       _extend_load_segment_vma(phdr_table, phdr_count, index, &p_memsz, &p_filesz,
-                               should_pad_segments);
+                               should_pad_segments, should_use_16kib_app_compat);
 
       *seg_page_end = page_end(phdr->p_vaddr + p_memsz + load_bias);
       return;
@@ -1192,7 +1389,8 @@
  */
 static int _phdr_table_set_gnu_relro_prot(const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                           ElfW(Addr) load_bias, int prot_flags,
-                                          bool should_pad_segments) {
+                                          bool should_pad_segments,
+                                          bool should_use_16kib_app_compat) {
   const ElfW(Phdr)* phdr = phdr_table;
   const ElfW(Phdr)* phdr_limit = phdr + phdr_count;
 
@@ -1220,7 +1418,7 @@
     ElfW(Addr) seg_page_start = page_start(phdr->p_vaddr) + load_bias;
     ElfW(Addr) seg_page_end = page_end(phdr->p_vaddr + phdr->p_memsz) + load_bias;
     _extend_gnu_relro_prot_end(phdr, phdr_table, phdr_count, load_bias, &seg_page_end,
-                               should_pad_segments);
+                               should_pad_segments, should_use_16kib_app_compat);
 
     int ret = mprotect(reinterpret_cast<void*>(seg_page_start),
                        seg_page_end - seg_page_start,
@@ -1246,13 +1444,29 @@
  *   phdr_count  -> number of entries in tables
  *   load_bias   -> load bias
  *   should_pad_segments -> Were segments extended to avoid gaps in the memory map
+ *   should_use_16kib_app_compat -> Is the ELF being loaded in 16KiB app compat mode.
  * Return:
  *   0 on success, -1 on failure (error code in errno).
  */
 int phdr_table_protect_gnu_relro(const ElfW(Phdr)* phdr_table, size_t phdr_count,
-                                 ElfW(Addr) load_bias, bool should_pad_segments) {
+                                 ElfW(Addr) load_bias, bool should_pad_segments,
+                                 bool should_use_16kib_app_compat) {
   return _phdr_table_set_gnu_relro_prot(phdr_table, phdr_count, load_bias, PROT_READ,
-                                        should_pad_segments);
+                                        should_pad_segments, should_use_16kib_app_compat);
+}
+
+/*
+ * Apply RX protection to the compat relro region of the ELF being loaded in
+ * 16KiB compat mode.
+ *
+ * Input:
+ *   start  -> start address of the compat relro region.
+ *   size   -> size of the compat relro region in bytes.
+ * Return:
+ *   0 on success, -1 on failure (error code in errno).
+ */
+int phdr_table_protect_gnu_relro_16kib_compat(ElfW(Addr) start, ElfW(Addr) size) {
+  return mprotect(reinterpret_cast<void*>(start), size, PROT_READ | PROT_EXEC);
 }
 
 /* Serialize the GNU relro segments to the given file descriptor. This can be
diff --git a/linker/linker_phdr.h b/linker/linker_phdr.h
index 1d6bbe3..e15ece4 100644
--- a/linker/linker_phdr.h
+++ b/linker/linker_phdr.h
@@ -39,6 +39,15 @@
 #include "linker_mapped_file_fragment.h"
 #include "linker_note_gnu_property.h"
 
+#include <list>
+
+#define MAYBE_MAP_FLAG(x, from, to)  (((x) & (from)) ? (to) : 0)
+#define PFLAGS_TO_PROT(x)            (MAYBE_MAP_FLAG((x), PF_X, PROT_EXEC) | \
+                                      MAYBE_MAP_FLAG((x), PF_R, PROT_READ) | \
+                                      MAYBE_MAP_FLAG((x), PF_W, PROT_WRITE))
+
+static constexpr size_t kCompatPageSize = 0x1000;
+
 class ElfReader {
  public:
   ElfReader();
@@ -59,6 +68,9 @@
   bool is_mapped_by_caller() const { return mapped_by_caller_; }
   ElfW(Addr) entry_point() const { return header_.e_entry + load_bias_; }
   bool should_pad_segments() const { return should_pad_segments_; }
+  bool should_use_16kib_app_compat() const { return should_use_16kib_app_compat_; }
+  ElfW(Addr) compat_relro_start() const { return compat_relro_start_; }
+  ElfW(Addr) compat_relro_size() const { return compat_relro_size_; }
 
  private:
   [[nodiscard]] bool ReadElfHeader();
@@ -69,10 +81,14 @@
   [[nodiscard]] bool ReadPadSegmentNote();
   [[nodiscard]] bool ReserveAddressSpace(address_space_params* address_space);
   [[nodiscard]] bool MapSegment(size_t seg_idx, size_t len);
+  [[nodiscard]] bool CompatMapSegment(size_t seg_idx, size_t len);
   void ZeroFillSegment(const ElfW(Phdr)* phdr);
   void DropPaddingPages(const ElfW(Phdr)* phdr, uint64_t seg_file_end);
   [[nodiscard]] bool MapBssSection(const ElfW(Phdr)* phdr, ElfW(Addr) seg_page_end,
                                    ElfW(Addr) seg_file_end);
+  [[nodiscard]] bool IsEligibleFor16KiBAppCompat(ElfW(Addr)* vaddr);
+  [[nodiscard]] bool HasAtMostOneRelroSegment(const ElfW(Phdr)** relro_phdr);
+  [[nodiscard]] bool Setup16KiBAppCompat();
   [[nodiscard]] bool LoadSegments();
   [[nodiscard]] bool FindPhdr();
   [[nodiscard]] bool FindGnuPropertySection();
@@ -123,6 +139,13 @@
   // Pad gaps between segments when memory mapping?
   bool should_pad_segments_ = false;
 
+  // Use app compat mode when loading 4KiB max-page-size ELFs on 16KiB page-size devices?
+  bool should_use_16kib_app_compat_ = false;
+
+  // RELRO region for 16KiB compat loading
+  ElfW(Addr) compat_relro_start_ = 0;
+  ElfW(Addr) compat_relro_size_ = 0;
+
   // Only used by AArch64 at the moment.
   GnuPropertySection note_gnu_property_ __unused;
 };
@@ -135,13 +158,18 @@
 
 int phdr_table_protect_segments(const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                 ElfW(Addr) load_bias, bool should_pad_segments,
+                                bool should_use_16kib_app_compat,
                                 const GnuPropertySection* prop = nullptr);
 
 int phdr_table_unprotect_segments(const ElfW(Phdr)* phdr_table, size_t phdr_count,
-                                  ElfW(Addr) load_bias, bool should_pad_segments);
+                                  ElfW(Addr) load_bias, bool should_pad_segments,
+                                  bool should_use_16kib_app_compat);
 
 int phdr_table_protect_gnu_relro(const ElfW(Phdr)* phdr_table, size_t phdr_count,
-                                 ElfW(Addr) load_bias, bool should_pad_segments);
+                                 ElfW(Addr) load_bias, bool should_pad_segments,
+                                 bool should_use_16kib_app_compat);
+
+int phdr_table_protect_gnu_relro_16kib_compat(ElfW(Addr) start, ElfW(Addr) size);
 
 int phdr_table_serialize_gnu_relro(const ElfW(Phdr)* phdr_table, size_t phdr_count,
                                    ElfW(Addr) load_bias, int fd, size_t* file_offset);
@@ -162,3 +190,13 @@
                                             ElfW(Addr) load_bias);
 
 bool page_size_migration_supported();
+
+int remap_memtag_globals_segments(const ElfW(Phdr) * phdr_table, size_t phdr_count,
+                                  ElfW(Addr) load_bias);
+
+void protect_memtag_globals_ro_segments(const ElfW(Phdr) * phdr_table, size_t phdr_count,
+                                        ElfW(Addr) load_bias);
+
+void name_memtag_globals_segments(const ElfW(Phdr) * phdr_table, size_t phdr_count,
+                                  ElfW(Addr) load_bias, const char* soname,
+                                  std::list<std::string>* vma_names);
diff --git a/linker/linker_phdr_16kib_compat.cpp b/linker/linker_phdr_16kib_compat.cpp
new file mode 100644
index 0000000..bad20ba
--- /dev/null
+++ b/linker/linker_phdr_16kib_compat.cpp
@@ -0,0 +1,247 @@
+/*
+ * Copyright (C) 2012 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "linker_phdr.h"
+
+#include <linux/prctl.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <unistd.h>
+
+#include "linker_debug.h"
+#include "linker_dlwarning.h"
+#include "linker_globals.h"
+
+#include "platform/bionic/macros.h"
+#include "platform/bionic/page.h"
+
+#include <string>
+
+static bool g_enable_16kb_app_compat;
+
+static inline bool segment_contains_prefix(const ElfW(Phdr)* segment, const ElfW(Phdr)* prefix) {
+  return segment && prefix && segment->p_vaddr == prefix->p_vaddr;
+}
+
+void set_16kb_appcompat_mode(bool enable_app_compat) {
+  g_enable_16kb_app_compat = enable_app_compat;
+}
+
+bool get_16kb_appcompat_mode() {
+  return g_enable_16kb_app_compat;
+}
+
+/*
+ * Returns true if the ELF contains at most 1 RELRO segment; and populates @relro_phdr
+ * with the relro phdr or nullptr if none.
+ *
+ * Returns false if more than 1 RELRO segments are found.
+ */
+bool ElfReader::HasAtMostOneRelroSegment(const ElfW(Phdr)** relro_phdr) {
+  const ElfW(Phdr)* relro = nullptr;
+  for (size_t i = 0; i < phdr_num_; ++i) {
+    const ElfW(Phdr)* phdr = &phdr_table_[i];
+
+    if (phdr->p_type != PT_GNU_RELRO) {
+      continue;
+    }
+
+    if (relro == nullptr) {
+      relro = phdr;
+    } else {
+      return false;
+    }
+  }
+
+  *relro_phdr = relro;
+
+  return true;
+}
+
+/*
+ * In 16KiB compatibility mode ELFs with the following segment layout
+ * can be loaded successfully:
+ *
+ *         ┌────────────┬─────────────────────────┬────────────┐
+ *         │            │                         │            │
+ *         │  (RO|RX)*  │   (RW - RELRO prefix)?  │    (RW)*   │
+ *         │            │                         │            │
+ *         └────────────┴─────────────────────────┴────────────┘
+ *
+ * In other words, compatible layouts have:
+ *         - zero or more RO or RX segments;
+ *         - followed by zero or one RELRO prefix;
+ *         - followed by zero or more RW segments (this can include the RW
+ *           suffix from the segment containing the RELRO prefix, if any)
+ *
+ * In 16KiB compat mode, after relocation, the ELF is layout in virtual
+ * memory is as shown below:
+ *         ┌──────────────────────────────────────┬────────────┐
+ *         │                                      │            │
+ *         │                (RX)?                 │    (RW)?   │
+ *         │                                      │            │
+ *         └──────────────────────────────────────┴────────────┘
+ *
+ * In compat mode:
+ *         - the RO and RX segments along with the RELRO prefix are protected
+ *           as RX;
+ *         - and the RW segments along with RW suffix from the relro segment,
+ *           if any; are RW protected.
+ *
+ * This allows for the single RX|RW permission boundary to be aligned with
+ * a 16KiB page boundary; since a single page cannot share multiple
+ * permissions.
+ *
+ * IsEligibleFor16KiBAppCompat() identifies compatible ELFs and populates @vaddr
+ * with the boundary between RX|RW portions.
+ *
+ * Returns true if the ELF can be loaded in compat mode, else false.
+ */
+bool ElfReader::IsEligibleFor16KiBAppCompat(ElfW(Addr)* vaddr) {
+  const ElfW(Phdr)* relro_phdr = nullptr;
+  if (!HasAtMostOneRelroSegment(&relro_phdr)) {
+    DL_WARN("\"%s\": Compat loading failed: Multiple RELRO segments found", name_.c_str());
+    return false;
+  }
+
+  const ElfW(Phdr)* last_rw = nullptr;
+  const ElfW(Phdr)* first_rw = nullptr;
+
+  for (size_t i = 0; i < phdr_num_; ++i) {
+    const ElfW(Phdr)* curr = &phdr_table_[i];
+    const ElfW(Phdr)* prev = (i > 0) ? &phdr_table_[i - 1] : nullptr;
+
+    if (curr->p_type != PT_LOAD) {
+      continue;
+    }
+
+    int prot = PFLAGS_TO_PROT(curr->p_flags);
+
+    if ((prot & PROT_WRITE) && (prot & PROT_READ)) {
+      if (!first_rw) {
+        first_rw = curr;
+      }
+
+      if (last_rw && last_rw != prev) {
+        DL_WARN("\"%s\": Compat loading failed: ELF contains multiple non-adjacent RW segments",
+                name_.c_str());
+        return false;
+      }
+
+      last_rw = curr;
+    }
+  }
+
+  if (!relro_phdr) {
+    *vaddr = align_down(first_rw->p_vaddr, kCompatPageSize);
+    return true;
+  }
+
+  // The RELRO segment is present, it must be the prefix of the first RW segment.
+  if (!segment_contains_prefix(first_rw, relro_phdr)) {
+    DL_WARN("\"%s\": Compat loading failed: RELRO is not in the first RW segment",
+            name_.c_str());
+    return false;
+  }
+
+  uint64_t end;
+  if (__builtin_add_overflow(relro_phdr->p_vaddr, relro_phdr->p_memsz, &end)) {
+    DL_WARN("\"%s\": Compat loading failed: relro vaddr + memsz overflowed", name_.c_str());
+    return false;
+  }
+
+  *vaddr = align_up(end, kCompatPageSize);
+  return true;
+}
+
+/*
+ * Returns the offset/shift needed to align @vaddr to a page boundary.
+ */
+static inline ElfW(Addr) perm_boundary_offset(const ElfW(Addr) addr) {
+  ElfW(Addr) offset = page_offset(addr);
+
+  return offset ? page_size() - offset : 0;
+}
+
+bool ElfReader::Setup16KiBAppCompat() {
+  if (!should_use_16kib_app_compat_) {
+    return true;
+  }
+
+  ElfW(Addr) rx_rw_boundary;  // Permission bounadry for compat mode
+  if (!IsEligibleFor16KiBAppCompat(&rx_rw_boundary)) {
+    return false;
+  }
+
+  // Adjust the load_bias to position the RX|RW boundary on a page boundary
+  load_bias_ += perm_boundary_offset(rx_rw_boundary);
+
+  // RW region (.data, .bss ...)
+  ElfW(Addr) rw_start = load_bias_ + rx_rw_boundary;
+  ElfW(Addr) rw_size = load_size_ - (rw_start - reinterpret_cast<ElfW(Addr)>(load_start_));
+
+  CHECK(rw_start % getpagesize() == 0);
+  CHECK(rw_size % getpagesize() == 0);
+
+  // Compat RELRO (RX) region (.text, .data.relro, ...)
+  compat_relro_start_ = reinterpret_cast<ElfW(Addr)>(load_start_);
+  compat_relro_size_ = load_size_ - rw_size;
+
+  // Label the ELF VMA, since compat mode uses anonymous mappings.
+  std::string compat_name = name_ + " (compat loaded)";
+  prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, load_start_, load_size_, compat_name.c_str());
+
+  return true;
+}
+
+bool ElfReader::CompatMapSegment(size_t seg_idx, size_t len) {
+  const ElfW(Phdr)* phdr = &phdr_table_[seg_idx];
+
+  // NOTE: The compat(legacy) page size (4096) must be used when aligning
+  // the 4KiB segments for loading (reading). The larger 16KiB page size
+  // will lead to overwriting adjacent segments since the ELF's segment(s)
+  // are not 16KiB aligned.
+
+  void* start = reinterpret_cast<void*>(align_down(phdr->p_vaddr + load_bias_, kCompatPageSize));
+
+  // The ELF could be being loaded directly from a zipped APK,
+  // the zip offset must be added to find the segment offset.
+  const ElfW(Addr) offset = file_offset_ + align_down(phdr->p_offset, kCompatPageSize);
+
+  CHECK(should_use_16kib_app_compat_);
+
+  // Since the 4KiB max-page-size ELF is not properly aligned, loading it by
+  // directly mmapping the ELF file is not feasible.
+  // Instead, read the ELF contents into the anonymous RW mapping.
+  if (TEMP_FAILURE_RETRY(pread64(fd_, start, len, offset)) == -1) {
+    DL_ERR("Compat loading: \"%s\" failed to read LOAD segment %zu: %m", name_.c_str(), seg_idx);
+    return false;
+  }
+
+  return true;
+}
diff --git a/linker/linker_relocate.cpp b/linker/linker_relocate.cpp
index bcb1efc..bbf8359 100644
--- a/linker/linker_relocate.cpp
+++ b/linker/linker_relocate.cpp
@@ -44,6 +44,8 @@
 #include "linker_soinfo.h"
 #include "private/bionic_globals.h"
 
+#include <platform/bionic/mte.h>
+
 static bool is_tls_reloc(ElfW(Word) type) {
   switch (type) {
     case R_GENERIC_TLS_DTPMOD:
@@ -163,7 +165,8 @@
 static bool process_relocation_impl(Relocator& relocator, const rel_t& reloc) {
   constexpr bool IsGeneral = Mode == RelocMode::General;
 
-  void* const rel_target = reinterpret_cast<void*>(reloc.r_offset + relocator.si->load_bias);
+  void* const rel_target = reinterpret_cast<void*>(
+      relocator.si->apply_memtag_if_mte_globals(reloc.r_offset + relocator.si->load_bias));
   const uint32_t r_type = ELFW(R_TYPE)(reloc.r_info);
   const uint32_t r_sym = ELFW(R_SYM)(reloc.r_info);
 
@@ -188,8 +191,8 @@
   auto protect_segments = [&]() {
     // Make .text executable.
     if (phdr_table_protect_segments(relocator.si->phdr, relocator.si->phnum,
-                                    relocator.si->load_bias,
-                                    relocator.si->should_pad_segments()) < 0) {
+                                    relocator.si->load_bias, relocator.si->should_pad_segments(),
+                                    relocator.si->should_use_16kib_app_compat()) < 0) {
       DL_ERR("can't protect segments for \"%s\": %m", relocator.si->get_realpath());
       return false;
     }
@@ -198,8 +201,8 @@
   auto unprotect_segments = [&]() {
     // Make .text writable.
     if (phdr_table_unprotect_segments(relocator.si->phdr, relocator.si->phnum,
-                                      relocator.si->load_bias,
-                                      relocator.si->should_pad_segments()) < 0) {
+                                      relocator.si->load_bias, relocator.si->should_pad_segments(),
+                                      relocator.si->should_use_16kib_app_compat()) < 0) {
       DL_ERR("can't unprotect loadable segments for \"%s\": %m",
              relocator.si->get_realpath());
       return false;
@@ -316,6 +319,7 @@
     // common in non-platform binaries.
     if (r_type == R_GENERIC_ABSOLUTE) {
       count_relocation_if<IsGeneral>(kRelocAbsolute);
+      if (found_in) sym_addr = found_in->apply_memtag_if_mte_globals(sym_addr);
       const ElfW(Addr) result = sym_addr + get_addend_rel();
       LD_DEBUG(reloc && IsGeneral, "RELO ABSOLUTE %16p <- %16p %s",
                rel_target, reinterpret_cast<void*>(result), sym_name);
@@ -326,6 +330,7 @@
       // document (IHI0044F) specifies that R_ARM_GLOB_DAT has an addend, but Bionic isn't adding
       // it.
       count_relocation_if<IsGeneral>(kRelocAbsolute);
+      if (found_in) sym_addr = found_in->apply_memtag_if_mte_globals(sym_addr);
       const ElfW(Addr) result = sym_addr + get_addend_norel();
       LD_DEBUG(reloc && IsGeneral, "RELO GLOB_DAT %16p <- %16p %s",
                rel_target, reinterpret_cast<void*>(result), sym_name);
@@ -335,7 +340,18 @@
       // In practice, r_sym is always zero, but if it weren't, the linker would still look up the
       // referenced symbol (and abort if the symbol isn't found), even though it isn't used.
       count_relocation_if<IsGeneral>(kRelocRelative);
-      const ElfW(Addr) result = relocator.si->load_bias + get_addend_rel();
+      ElfW(Addr) result = relocator.si->load_bias + get_addend_rel();
+      // MTE globals reuses the place bits for additional tag-derivation metadata for
+      // R_AARCH64_RELATIVE relocations, which makes it incompatible with
+      // `-Wl,--apply-dynamic-relocs`. This is enforced by lld, however there's nothing stopping
+      // Android binaries (particularly prebuilts) from building with this linker flag if they're
+      // not built with MTE globals. Thus, don't use the new relocation semantics if this DSO
+      // doesn't have MTE globals.
+      if (relocator.si->should_tag_memtag_globals()) {
+        int64_t* place = static_cast<int64_t*>(rel_target);
+        int64_t offset = *place;
+        result = relocator.si->apply_memtag_if_mte_globals(result + offset) - offset;
+      }
       LD_DEBUG(reloc && IsGeneral, "RELO RELATIVE %16p <- %16p",
                rel_target, reinterpret_cast<void*>(result));
       *static_cast<ElfW(Addr)*>(rel_target) = result;
@@ -600,7 +616,7 @@
     LD_DEBUG(reloc, "[ relocating %s relr ]", get_realpath());
     const ElfW(Relr)* begin = relr_;
     const ElfW(Relr)* end = relr_ + relr_count_;
-    if (!relocate_relr(begin, end, load_bias)) {
+    if (!relocate_relr(begin, end, load_bias, should_tag_memtag_globals())) {
       return false;
     }
   }
diff --git a/linker/linker_sleb128.h b/linker/linker_sleb128.h
index 6bb3199..f48fda8 100644
--- a/linker/linker_sleb128.h
+++ b/linker/linker_sleb128.h
@@ -69,3 +69,32 @@
   const uint8_t* current_;
   const uint8_t* const end_;
 };
+
+class uleb128_decoder {
+ public:
+  uleb128_decoder(const uint8_t* buffer, size_t count) : current_(buffer), end_(buffer + count) {}
+
+  uint64_t pop_front() {
+    uint64_t value = 0;
+
+    size_t shift = 0;
+    uint8_t byte;
+
+    do {
+      if (current_ >= end_) {
+        async_safe_fatal("uleb128_decoder ran out of bounds");
+      }
+      byte = *current_++;
+      value |= (static_cast<size_t>(byte & 127) << shift);
+      shift += 7;
+    } while (byte & 128);
+
+    return value;
+  }
+
+  bool has_bytes() { return current_ < end_; }
+
+ private:
+  const uint8_t* current_;
+  const uint8_t* const end_;
+};
diff --git a/linker/linker_soinfo.cpp b/linker/linker_soinfo.cpp
index 0549d36..176c133 100644
--- a/linker/linker_soinfo.cpp
+++ b/linker/linker_soinfo.cpp
@@ -44,6 +44,8 @@
 #include "linker_logger.h"
 #include "linker_relocate.h"
 #include "linker_utils.h"
+#include "platform/bionic/mte.h"
+#include "private/bionic_globals.h"
 
 SymbolLookupList::SymbolLookupList(soinfo* si)
     : sole_lib_(si->get_lookup_lib()), begin_(&sole_lib_), end_(&sole_lib_ + 1) {
@@ -304,6 +306,12 @@
   return is_gnu_hash() ? gnu_lookup(symbol_name, vi) : elf_lookup(symbol_name, vi);
 }
 
+ElfW(Addr) soinfo::apply_memtag_if_mte_globals(ElfW(Addr) sym_addr) const {
+  if (!should_tag_memtag_globals()) return sym_addr;
+  if (sym_addr == 0) return sym_addr;  // Handle undefined weak symbols.
+  return reinterpret_cast<ElfW(Addr)>(get_tagged_address(reinterpret_cast<void*>(sym_addr)));
+}
+
 const ElfW(Sym)* soinfo::gnu_lookup(SymbolName& symbol_name, const version_info* vi) const {
   const uint32_t hash = symbol_name.gnu_hash();
 
diff --git a/linker/linker_soinfo.h b/linker/linker_soinfo.h
index 9a13af2..4d02676 100644
--- a/linker/linker_soinfo.h
+++ b/linker/linker_soinfo.h
@@ -30,6 +30,7 @@
 
 #include <link.h>
 
+#include <list>
 #include <memory>
 #include <string>
 #include <vector>
@@ -66,6 +67,7 @@
                                          // soinfo is executed and this flag is
                                          // unset.
 #define FLAG_PRELINKED        0x00000400 // prelink_image has successfully processed this soinfo
+#define FLAG_GLOBALS_TAGGED   0x00000800 // globals have been tagged by MTE.
 #define FLAG_NEW_SOINFO       0x40000000 // new soinfo format
 
 #define SOINFO_VERSION 6
@@ -252,11 +254,14 @@
   void call_constructors();
   void call_destructors();
   void call_pre_init_constructors();
-  bool prelink_image();
+  bool prelink_image(bool deterministic_memtag_globals = false);
   bool link_image(const SymbolLookupList& lookup_list, soinfo* local_group_root,
                   const android_dlextinfo* extinfo, size_t* relro_fd_offset);
   bool protect_relro();
 
+  void tag_globals(bool deterministic_memtag_globals);
+  ElfW(Addr) apply_memtag_if_mte_globals(ElfW(Addr) sym_addr) const;
+
   void add_child(soinfo* child);
   void remove_all_links();
 
@@ -293,6 +298,9 @@
 #if defined(__work_around_b_24465209__)
     return (flags_ & FLAG_NEW_SOINFO) != 0 && version_ >= min_version;
 #else
+    // If you make this return non-true in the case where
+    // __work_around_b_24465209__ is not defined, you will have to change
+    // memtag_dynamic_entries() and vma_names().
     return true;
 #endif
   }
@@ -354,20 +362,66 @@
   size_t get_gap_size() const;
 
   const memtag_dynamic_entries_t* memtag_dynamic_entries() const {
-    CHECK(has_min_version(7));
+#ifdef __aarch64__
+#ifdef __work_around_b_24465209__
+#error "Assuming aarch64 does not use versioned soinfo."
+#endif
     return &memtag_dynamic_entries_;
+#endif
+    return nullptr;
   }
-  void* memtag_globals() const { return memtag_dynamic_entries()->memtag_globals; }
-  size_t memtag_globalssz() const { return memtag_dynamic_entries()->memtag_globalssz; }
-  bool has_memtag_mode() const { return memtag_dynamic_entries()->has_memtag_mode; }
-  unsigned memtag_mode() const { return memtag_dynamic_entries()->memtag_mode; }
-  bool memtag_heap() const { return memtag_dynamic_entries()->memtag_heap; }
-  bool memtag_stack() const { return memtag_dynamic_entries()->memtag_stack; }
+  void* memtag_globals() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->memtag_globals : nullptr;
+  }
+  size_t memtag_globalssz() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->memtag_globalssz : 0U;
+  }
+  bool has_memtag_mode() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->has_memtag_mode : false;
+  }
+  unsigned memtag_mode() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->memtag_mode : 0U;
+  }
+  bool memtag_heap() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->memtag_heap : false;
+  }
+  bool memtag_stack() const {
+    const memtag_dynamic_entries_t* entries = memtag_dynamic_entries();
+    return entries ? entries->memtag_stack : false;
+  }
 
   void set_should_pad_segments(bool should_pad_segments) {
    should_pad_segments_ = should_pad_segments;
   }
   bool should_pad_segments() const { return should_pad_segments_; }
+  bool should_tag_memtag_globals() const {
+    return !is_linker() && memtag_globals() && memtag_globalssz() > 0 && __libc_mte_enabled();
+  }
+  std::list<std::string>* vma_names() {
+#ifdef __aarch64__
+#ifdef __work_around_b_24465209__
+#error "Assuming aarch64 does not use versioned soinfo."
+#endif
+    return &vma_names_;
+#endif
+    return nullptr;
+};
+
+  void set_should_use_16kib_app_compat(bool should_use_16kib_app_compat) {
+    should_use_16kib_app_compat_ = should_use_16kib_app_compat;
+  }
+  bool should_use_16kib_app_compat() const { return should_use_16kib_app_compat_; }
+
+  void set_compat_relro_start(ElfW(Addr) start) { compat_relro_start_ = start; }
+  ElfW(Addr) compat_relro_start() const { return compat_relro_start_; }
+
+  void set_compat_relro_size(ElfW(Addr) size) { compat_relro_size_ = size; }
+  ElfW(Addr) compat_relro_size() const { return compat_relro_start_; }
 
  private:
   bool is_image_linked() const;
@@ -450,11 +504,19 @@
   ElfW(Addr) gap_start_;
   size_t gap_size_;
 
-  // version >= 7
+  // __aarch64__ only, which does not use versioning.
   memtag_dynamic_entries_t memtag_dynamic_entries_;
+  std::list<std::string> vma_names_;
 
   // Pad gaps between segments when memory mapping?
   bool should_pad_segments_ = false;
+
+  // Use app compat mode when loading 4KiB max-page-size ELFs on 16KiB page-size devices?
+  bool should_use_16kib_app_compat_ = false;
+
+  // RELRO region for 16KiB compat loading
+  ElfW(Addr) compat_relro_start_ = 0;
+  ElfW(Addr) compat_relro_size_ = 0;
 };
 
 // This function is used by dlvsym() to calculate hash of sym_ver
diff --git a/tests/Android.bp b/tests/Android.bp
index 6aecb40..a97f5a8 100644
--- a/tests/Android.bp
+++ b/tests/Android.bp
@@ -389,7 +389,9 @@
         "bug_26110743_test.cpp",
         "byteswap_test.cpp",
         "complex_test.cpp",
-        "cpu_target_features_test.cpp",
+        // Disabled while investigating
+        // b/378304366, b/375525252
+        // "cpu_target_features_test.cpp",
         "ctype_test.cpp",
         "dirent_test.cpp",
         "elf_test.cpp",
@@ -428,6 +430,7 @@
         "malloc_test.cpp",
         "math_test.cpp",
         "membarrier_test.cpp",
+        "memtag_globals_test.cpp",
         "memtag_stack_test.cpp",
         "mntent_test.cpp",
         "mte_test.cpp",
@@ -746,6 +749,40 @@
     },
 }
 
+cc_defaults {
+    name: "bionic_fortify_c_tests_defaults",
+    defaults: [
+        "bionic_clang_fortify_tests_w_flags",
+        "bionic_tests_defaults",
+    ],
+    cflags: [
+        "-U_FORTIFY_SOURCE",
+        // -fbuiltin is required here to counteract -fno-builtin from
+        // `bionic_tests_defaults`. With `-fno-builtin`, Clang won't
+        // const-evaluate calls to `strlen`, which is tested for here.
+        "-fbuiltin",
+    ],
+    srcs: [
+        "clang_fortify_c_only_tests.c",
+    ],
+    tidy: false,
+    shared: {
+        enabled: false,
+    },
+}
+
+cc_test_library {
+    name: "libfortify1-c-tests-clang",
+    defaults: ["bionic_fortify_c_tests_defaults"],
+    cflags: ["-D_FORTIFY_SOURCE=1"],
+}
+
+cc_test_library {
+    name: "libfortify2-c-tests-clang",
+    defaults: ["bionic_fortify_c_tests_defaults"],
+    cflags: ["-D_FORTIFY_SOURCE=2"],
+}
+
 // -----------------------------------------------------------------------------
 // Library of all tests (excluding the dynamic linker tests).
 // -----------------------------------------------------------------------------
@@ -757,8 +794,10 @@
         "libBionicStandardTests",
         "libBionicElfTlsTests",
         "libBionicFramePointerTests",
+        "libfortify1-c-tests-clang",
         "libfortify1-tests-clang",
         "libfortify1-new-tests-clang",
+        "libfortify2-c-tests-clang",
         "libfortify2-tests-clang",
         "libfortify2-new-tests-clang",
     ],
@@ -779,6 +818,7 @@
         "dlfcn_test.cpp",
         "execinfo_test.cpp",
         "link_test.cpp",
+        "page_size_16kib_compat_test.cpp",
         "pthread_dlfcn_test.cpp",
     ],
     static_libs: [
@@ -787,6 +827,7 @@
     ],
     include_dirs: [
         "bionic/libc",
+        "bionic/tests/libs",
     ],
     shared: {
         enabled: false,
@@ -853,6 +894,11 @@
         "ld_preload_test_helper",
         "ld_preload_test_helper_lib1",
         "ld_preload_test_helper_lib2",
+        "memtag_globals_binary",
+        "memtag_globals_binary_static",
+        "memtag_globals_dso",
+        "mte_globals_relr_regression_test_b_314038442",
+        "mte_globals_relr_regression_test_b_314038442_mte",
         "ns_hidden_child_helper",
         "preinit_getauxval_test_helper",
         "preinit_syscall_test_helper",
@@ -934,6 +980,7 @@
         "libtest_dt_runpath_d",
         "libtest_dt_runpath_x",
         "libtest_dt_runpath_y",
+        "libtest_elf_max_page_size_4kib",
         "libtest_elftls_dynamic",
         "libtest_elftls_dynamic_filler_1",
         "libtest_elftls_dynamic_filler_2",
@@ -1160,6 +1207,35 @@
 }
 
 cc_test {
+    name: "memtag_stack_abi_test",
+    enabled: false,
+    // This does not use bionic_tests_defaults because it is not supported on
+    // host.
+    arch: {
+        arm64: {
+            enabled: true,
+        },
+    },
+    // We don't use `sanitize:` so we generate the appropriate ELF note, but
+    // still support non-MTE devices.
+    // TODO(fmayer): also add a test that enables stack MTE for MTE devices,
+    // which would test for more bugs.
+    ldflags: ["-fsanitize=memtag-stack"],
+    // Turn off all other sanitizers from SANITIZE_TARGET.
+    sanitize: {
+        never: true,
+    },
+    shared_libs: [
+        "libbase",
+    ],
+    srcs: [
+        "memtag_stack_abi_test.cpp",
+    ],
+    header_libs: ["bionic_libc_platform_headers"],
+    test_suites: ["device-tests"],
+}
+
+cc_test {
     name: "bionic-stress-tests",
     defaults: [
         "bionic_tests_defaults",
@@ -1178,6 +1254,7 @@
 
     shared_libs: [
         "libbase",
+        "liblog",
     ],
 
     target: {
@@ -1247,6 +1324,11 @@
         "heap_tagging_static_disabled_helper",
         "heap_tagging_static_sync_helper",
         "heap_tagging_sync_helper",
+        "memtag_globals_binary",
+        "memtag_globals_binary_static",
+        "memtag_globals_dso",
+        "mte_globals_relr_regression_test_b_314038442",
+        "mte_globals_relr_regression_test_b_314038442_mte",
         "stack_tagging_helper",
         "stack_tagging_static_helper",
     ],
diff --git a/libc/private/bsd_sys_param.h b/tests/DoNotOptimize.h
similarity index 63%
rename from libc/private/bsd_sys_param.h
rename to tests/DoNotOptimize.h
index ab54aa0..711d339 100644
--- a/libc/private/bsd_sys_param.h
+++ b/tests/DoNotOptimize.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2014 The Android Open Source Project
+ * Copyright (C) 2012 The Android Open Source Project
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,8 +16,12 @@
 
 #pragma once
 
-#include <inttypes.h>
-
-/* OpenBSD has these in <sys/param.h>, but "ALIGN" isn't something we want to reserve. */
-#define ALIGNBYTES (sizeof(uintptr_t) - 1)
-#define ALIGN(p) ((__BIONIC_CAST(reinterpret_cast, uintptr_t, p) + ALIGNBYTES) & ~ALIGNBYTES)
+// From <benchmark/benchmark.h>.
+template <class Tp>
+static inline void DoNotOptimize(Tp const& value) {
+  asm volatile("" : : "r,m"(value) : "memory");
+}
+template <class Tp>
+static inline void DoNotOptimize(Tp& value) {
+  asm volatile("" : "+r,m"(value) : : "memory");
+}
diff --git a/tests/clang_fortify_c_only_tests.c b/tests/clang_fortify_c_only_tests.c
new file mode 100644
index 0000000..3bec848
--- /dev/null
+++ b/tests/clang_fortify_c_only_tests.c
@@ -0,0 +1,38 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+// This is a test specifically of bionic's FORTIFY machinery. Other stdlibs need not apply.
+#ifdef __BIONIC__
+
+// Ensure that strlen can be evaluated at compile-time. Clang doesn't support
+// this in C++, but does in C.
+_Static_assert(strlen("foo") == 3, "");
+
+#endif  // __BIONIC__
diff --git a/tests/clang_fortify_tests.cpp b/tests/clang_fortify_tests.cpp
index f08fd1f..da7926d 100644
--- a/tests/clang_fortify_tests.cpp
+++ b/tests/clang_fortify_tests.cpp
@@ -89,6 +89,10 @@
 #include <unistd.h>
 #include <wchar.h>
 
+#include <array>
+
+#include "DoNotOptimize.h"
+
 #ifndef COMPILATION_TESTS
 #include <android-base/silent_death_test.h>
 #include <gtest/gtest.h>
@@ -133,6 +137,24 @@
 
 const static int kBogusFD = -1;
 
+FORTIFY_TEST(strlen) {
+  auto run_strlen_with_contents = [&](std::array<char, 3> contents) {
+    // A lot of cruft is necessary to make this test DTRT. LLVM and Clang love to fold/optimize
+    // strlen calls, and that's the opposite of what we want to happen.
+
+    // Loop to convince LLVM that `contents` can never be known (since `xor volatile_value` can flip
+    // any bit in each elem of `contents`).
+    volatile char always_zero = 0;
+    for (char& c : contents) {
+      c ^= always_zero;
+    }
+    DoNotOptimize(strlen(&contents.front()));
+  };
+
+  EXPECT_NO_DEATH(run_strlen_with_contents({'f', 'o', '\0'}));
+  EXPECT_FORTIFY_DEATH(run_strlen_with_contents({'f', 'o', 'o'}));
+}
+
 FORTIFY_TEST(string) {
   char small_buffer[8] = {};
 
diff --git a/tests/complex_test.cpp b/tests/complex_test.cpp
index 8fdb2b2..ed0109a 100644
--- a/tests/complex_test.cpp
+++ b/tests/complex_test.cpp
@@ -20,6 +20,9 @@
 #if !defined(__INTRODUCED_IN)
 #define __INTRODUCED_IN(x)
 #endif
+#if !defined(__BIONIC_AVAILABILITY_GUARD)
+#define __BIONIC_AVAILABILITY_GUARD(x) 1
+#endif
 
 // libc++ actively gets in the way of including <complex.h> from C++, so we
 // have to be naughty.
diff --git a/tests/cpu_target_features_test.cpp b/tests/cpu_target_features_test.cpp
index 3458bca..d773772 100644
--- a/tests/cpu_target_features_test.cpp
+++ b/tests/cpu_target_features_test.cpp
@@ -54,15 +54,3 @@
   GTEST_SKIP() << "Not targeting an aarch64 architecture.";
 #endif
 }
-
-TEST(cpu_target_features, has_expected_arm_compiler_values) {
-#if defined(__arm__)
-  ExecTestHelper eth;
-  char* const argv[] = {nullptr};
-  const auto invocation = [&] { execvp("cpu-target-features", argv); };
-  eth.Run(invocation, 0, "(^|\n)__ARM_FEATURE_AES=1($|\n)");
-  eth.Run(invocation, 0, "(^|\n)__ARM_FEATURE_CRC32=1($|\n)");
-#else
-  GTEST_SKIP() << "Not targeting an arm architecture.";
-#endif
-}
diff --git a/tests/dlext_test.cpp b/tests/dlext_test.cpp
index 570da2a..8b26cb0 100644
--- a/tests/dlext_test.cpp
+++ b/tests/dlext_test.cpp
@@ -21,6 +21,7 @@
 #include <errno.h>
 #include <fcntl.h>
 #include <inttypes.h>
+#include <link.h>
 #include <stdio.h>
 #include <string.h>
 #include <unistd.h>
@@ -40,11 +41,13 @@
 #include <procinfo/process_map.h>
 #include <ziparchive/zip_archive.h>
 
+#include "bionic/mte.h"
+#include "bionic/page.h"
 #include "core_shared_libs.h"
-#include "gtest_globals.h"
-#include "utils.h"
 #include "dlext_private.h"
 #include "dlfcn_symlink_support.h"
+#include "gtest_globals.h"
+#include "utils.h"
 
 #define ASSERT_DL_NOTNULL(ptr) \
     ASSERT_TRUE((ptr) != nullptr) << "dlerror: " << dlerror()
@@ -1958,6 +1961,14 @@
   dlclose(ns_a_handle3);
 }
 
+static inline int MapPflagsToProtFlags(uint32_t flags) {
+  int prot_flags = 0;
+  if (PF_X & flags) prot_flags |= PROT_EXEC;
+  if (PF_W & flags) prot_flags |= PROT_WRITE;
+  if (PF_R & flags) prot_flags |= PROT_READ;
+  return prot_flags;
+}
+
 TEST(dlext, ns_anonymous) {
   static const char* root_lib = "libnstest_root.so";
   std::string shared_libs = g_core_shared_libs + ":" + g_public_lib;
@@ -1999,30 +2010,45 @@
   typedef const char* (*fn_t)();
   fn_t ns_get_dlopened_string_private = reinterpret_cast<fn_t>(ns_get_dlopened_string_addr);
 
-  std::vector<map_record> maps;
-  Maps::parse_maps(&maps);
-
+  Dl_info private_library_info;
+  ASSERT_NE(dladdr(reinterpret_cast<void*>(ns_get_dlopened_string_addr), &private_library_info), 0)
+      << dlerror();
+  std::vector<map_record> maps_to_copy;
+  bool has_executable_segment = false;
   uintptr_t addr_start = 0;
   uintptr_t addr_end = 0;
-  bool has_executable_segment = false;
-  std::vector<map_record> maps_to_copy;
+  std::tuple dl_iterate_arg = {&private_library_info, &maps_to_copy, &has_executable_segment,
+                               &addr_start, &addr_end};
+  ASSERT_EQ(
+      1, dl_iterate_phdr(
+             [](dl_phdr_info* info, size_t /*size*/, void* data) -> int {
+               auto [private_library_info, maps_to_copy, has_executable_segment, addr_start,
+                     addr_end] = *reinterpret_cast<decltype(dl_iterate_arg)*>(data);
+               if (info->dlpi_addr != reinterpret_cast<ElfW(Addr)>(private_library_info->dli_fbase))
+                 return 0;
 
-  for (const auto& rec : maps) {
-    if (rec.pathname == private_library_absolute_path) {
-      if (addr_start == 0) {
-        addr_start = rec.addr_start;
-      }
-      addr_end = rec.addr_end;
-      has_executable_segment = has_executable_segment || (rec.perms & PROT_EXEC) != 0;
-
-      maps_to_copy.push_back(rec);
-    }
-  }
+               for (size_t i = 0; i < info->dlpi_phnum; ++i) {
+                 const ElfW(Phdr)* phdr = info->dlpi_phdr + i;
+                 if (phdr->p_type != PT_LOAD) continue;
+                 *has_executable_segment |= phdr->p_flags & PF_X;
+                 uintptr_t mapping_start = page_start(info->dlpi_addr + phdr->p_vaddr);
+                 uintptr_t mapping_end = page_end(info->dlpi_addr + phdr->p_vaddr + phdr->p_memsz);
+                 if (*addr_start == 0 || mapping_start < *addr_start) *addr_start = mapping_start;
+                 if (*addr_end == 0 || mapping_end > *addr_end) *addr_end = mapping_end;
+                 maps_to_copy->push_back({
+                     .addr_start = mapping_start,
+                     .addr_end = mapping_end,
+                     .perms = MapPflagsToProtFlags(phdr->p_flags),
+                 });
+               }
+               return 1;
+             },
+             &dl_iterate_arg));
 
   // Some validity checks.
+  ASSERT_NE(maps_to_copy.size(), 0u);
   ASSERT_TRUE(addr_start > 0);
   ASSERT_TRUE(addr_end > 0);
-  ASSERT_TRUE(maps_to_copy.size() > 0);
   ASSERT_TRUE(ns_get_dlopened_string_addr > addr_start);
   ASSERT_TRUE(ns_get_dlopened_string_addr < addr_end);
 
@@ -2052,19 +2078,26 @@
   ASSERT_EQ(ret, 0) << "Failed to stat library";
   size_t file_size = file_stat.st_size;
 
-  for (const auto& rec : maps_to_copy) {
-    uintptr_t offset = rec.addr_start - addr_start;
-    size_t size = rec.addr_end - rec.addr_start;
-    void* addr = reinterpret_cast<void*>(reserved_addr + offset);
-    void* map = mmap(addr, size, PROT_READ | PROT_WRITE,
-                     MAP_ANON | MAP_PRIVATE | MAP_FIXED, -1, 0);
-    ASSERT_TRUE(map != MAP_FAILED);
-    // Attempting the below memcpy from a portion of the map that is off the end of
-    // the backing file will cause the kernel to throw a SIGBUS
-    size_t _size = ::android::procinfo::MappedFileSize(rec.addr_start, rec.addr_end,
-                                                       rec.offset, file_size);
-    memcpy(map, reinterpret_cast<void*>(rec.addr_start), _size);
-    mprotect(map, size, rec.perms);
+  {
+    // Disable MTE while copying the PROT_MTE-protected global variables from
+    // the existing mappings. We don't really care about turning on PROT_MTE for
+    // the new copy of the mappings, as this isn't the behaviour under test and
+    // tags will be ignored. This only applies for MTE-enabled devices.
+    ScopedDisableMTE disable_mte_for_copying_global_variables;
+    for (const auto& rec : maps_to_copy) {
+      uintptr_t offset = rec.addr_start - addr_start;
+      size_t size = rec.addr_end - rec.addr_start;
+      void* addr = reinterpret_cast<void*>(reserved_addr + offset);
+      void* map =
+          mmap(addr, size, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE | MAP_FIXED, -1, 0);
+      ASSERT_TRUE(map != MAP_FAILED);
+      // Attempting the below memcpy from a portion of the map that is off the end of
+      // the backing file will cause the kernel to throw a SIGBUS
+      size_t _size =
+          ::android::procinfo::MappedFileSize(rec.addr_start, rec.addr_end, rec.offset, file_size);
+      memcpy(map, reinterpret_cast<void*>(rec.addr_start), _size);
+      mprotect(map, size, rec.perms);
+    }
   }
 
   // call the function copy
diff --git a/tests/fenv_test.cpp b/tests/fenv_test.cpp
index 9cf9d98..bbf339f 100644
--- a/tests/fenv_test.cpp
+++ b/tests/fenv_test.cpp
@@ -16,6 +16,7 @@
 
 #include <gtest/gtest.h>
 
+#include "DoNotOptimize.h"
 #include "utils.h"
 
 #include <fenv.h>
diff --git a/tests/grp_pwd_test.cpp b/tests/grp_pwd_test.cpp
index 7b7e0e5..3f93c8a 100644
--- a/tests/grp_pwd_test.cpp
+++ b/tests/grp_pwd_test.cpp
@@ -480,6 +480,18 @@
       EXPECT_STREQ(getpwuid(AID_CROS_EC)->pw_name, "cros_ec");
     }
   }
+  // AID_MMD (1095) was added in API level 36, but "trunk stable" means
+  // that the 2024Q* builds are tested with the _previous_ release's CTS.
+  if (android::base::GetIntProperty("ro.build.version.sdk", 0) == 35) {
+#if !defined(AID_MMD)
+#define AID_MMD 1095
+#endif
+    ids.erase(AID_MMD);
+    expected_ids.erase(AID_MMD);
+    if (getpwuid(AID_MMD)) {
+      EXPECT_STREQ(getpwuid(AID_MMD)->pw_name, "mmd");
+    }
+  }
 
   EXPECT_EQ(expected_ids, ids) << return_differences();
 }
diff --git a/tests/headers/posix/stdatomic_h.c b/tests/headers/posix/stdatomic_h.c
new file mode 100644
index 0000000..05be859
--- /dev/null
+++ b/tests/headers/posix/stdatomic_h.c
@@ -0,0 +1,175 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdatomic.h>
+
+#include "header_checks.h"
+
+static void stdatomic_h() {
+  TYPE(atomic_flag);
+  TYPE(atomic_bool);
+  TYPE(atomic_char);
+  TYPE(atomic_schar);
+  TYPE(atomic_uchar);
+  TYPE(atomic_short);
+  TYPE(atomic_ushort);
+  TYPE(atomic_int);
+  TYPE(atomic_uint);
+  TYPE(atomic_long);
+  TYPE(atomic_ulong);
+  TYPE(atomic_llong);
+  TYPE(atomic_ullong);
+#if !defined(__GLIBC__)
+  TYPE(atomic_char16_t);
+  TYPE(atomic_char32_t);
+#endif
+  TYPE(atomic_wchar_t);
+  TYPE(atomic_int_least8_t);
+  TYPE(atomic_uint_least8_t);
+  TYPE(atomic_int_least16_t);
+  TYPE(atomic_uint_least16_t);
+  TYPE(atomic_int_least32_t);
+  TYPE(atomic_uint_least32_t);
+  TYPE(atomic_int_least64_t);
+  TYPE(atomic_uint_least64_t);
+  TYPE(atomic_int_fast8_t);
+  TYPE(atomic_uint_fast8_t);
+  TYPE(atomic_int_fast16_t);
+  TYPE(atomic_uint_fast16_t);
+  TYPE(atomic_int_fast32_t);
+  TYPE(atomic_uint_fast32_t);
+  TYPE(atomic_int_fast64_t);
+  TYPE(atomic_uint_fast64_t);
+  TYPE(atomic_intptr_t);
+  TYPE(atomic_uintptr_t);
+  TYPE(atomic_size_t);
+  TYPE(atomic_ptrdiff_t);
+  TYPE(atomic_intmax_t);
+  TYPE(atomic_uintmax_t);
+
+  memory_order m1 = memory_order_relaxed;
+  memory_order m2 = memory_order_consume;
+  memory_order m3 = memory_order_acquire;
+  memory_order m4 = memory_order_release;
+  memory_order m5 = memory_order_acq_rel;
+  memory_order m6 = memory_order_seq_cst;
+
+  MACRO(ATOMIC_BOOL_LOCK_FREE);
+  MACRO(ATOMIC_CHAR_LOCK_FREE);
+  MACRO(ATOMIC_CHAR16_T_LOCK_FREE);
+  MACRO(ATOMIC_CHAR32_T_LOCK_FREE);
+  MACRO(ATOMIC_WCHAR_T_LOCK_FREE);
+  MACRO(ATOMIC_SHORT_LOCK_FREE);
+  MACRO(ATOMIC_INT_LOCK_FREE);
+  MACRO(ATOMIC_LONG_LOCK_FREE);
+  MACRO(ATOMIC_LLONG_LOCK_FREE);
+  MACRO(ATOMIC_POINTER_LOCK_FREE);
+
+  atomic_flag f = ATOMIC_FLAG_INIT;
+  atomic_int i = ATOMIC_VAR_INIT(123);
+
+  // TODO: remove this #if after the next toolchain update (http://b/374104004).
+#if !defined(__GLIBC__)
+  i = kill_dependency(i);
+#endif
+
+#if !defined(atomic_compare_exchange_strong)
+#error atomic_compare_exchange_strong
+#endif
+#if !defined(atomic_compare_exchange_strong_explicit)
+#error atomic_compare_exchange_strong_explicit
+#endif
+#if !defined(atomic_compare_exchange_weak)
+#error atomic_compare_exchange_weak
+#endif
+#if !defined(atomic_compare_exchange_weak_explicit)
+#error atomic_compare_exchange_weak_explicit
+#endif
+#if !defined(atomic_exchange)
+#error atomic_exchange
+#endif
+#if !defined(atomic_exchange_explicit)
+#error atomic_exchange_explicit
+#endif
+#if !defined(atomic_fetch_add)
+#error atomic_fetch_add
+#endif
+#if !defined(atomic_fetch_add_explicit)
+#error atomic_fetch_add_explicit
+#endif
+#if !defined(atomic_fetch_and)
+#error atomic_fetch_and
+#endif
+#if !defined(atomic_fetch_and_explicit)
+#error atomic_fetch_and_explicit
+#endif
+#if !defined(atomic_fetch_or)
+#error atomic_fetch_or
+#endif
+#if !defined(atomic_fetch_or_explicit)
+#error atomic_fetch_or_explicit
+#endif
+#if !defined(atomic_fetch_sub)
+#error atomic_fetch_sub
+#endif
+#if !defined(atomic_fetch_sub_explicit)
+#error atomic_fetch_sub_explicit
+#endif
+#if !defined(atomic_fetch_xor)
+#error atomic_fetch_xor
+#endif
+#if !defined(atomic_fetch_xor_explicit)
+#error atomic_fetch_xor_explicit
+#endif
+#if !defined(atomic_init)
+#error atomic_init
+#endif
+#if !defined(atomic_is_lock_free)
+#error atomic_is_lock_free
+#endif
+#if !defined(atomic_load)
+#error atomic_load
+#endif
+#if !defined(atomic_load_explicit)
+#error atomic_load_explicit
+#endif
+#if !defined(atomic_store)
+#error atomic_store
+#endif
+#if !defined(atomic_store_explicit)
+#error atomic_store_explicit
+#endif
+
+  FUNCTION(atomic_flag_clear, void (*f)(volatile atomic_flag*));
+  FUNCTION(atomic_flag_clear_explicit, void (*f)(volatile atomic_flag*, memory_order));
+  FUNCTION(atomic_flag_test_and_set, bool (*f)(volatile atomic_flag*));
+  FUNCTION(atomic_flag_test_and_set_explicit, bool (*f)(volatile atomic_flag*, memory_order));
+
+  FUNCTION(atomic_signal_fence, void (*f)(memory_order));
+  FUNCTION(atomic_thread_fence, void (*f)(memory_order));
+}
diff --git a/tests/headers/posix/stdbool_h.c b/tests/headers/posix/stdbool_h.c
index f891a73..830c33c 100644
--- a/tests/headers/posix/stdbool_h.c
+++ b/tests/headers/posix/stdbool_h.c
@@ -31,10 +31,8 @@
 #include "header_checks.h"
 
 static void stdbool_h() {
-#if !defined(bool)
-#error bool
-#endif
-  MACRO_VALUE(true, 1);
-  MACRO_VALUE(false, 0);
+  TYPE(bool);
+  bool t = true;
+  bool f = false;
   MACRO_VALUE(__bool_true_false_are_defined, 1);
 }
diff --git a/tests/libs/Android.bp b/tests/libs/Android.bp
index 35f0f0c..5b86e78 100644
--- a/tests/libs/Android.bp
+++ b/tests/libs/Android.bp
@@ -48,6 +48,16 @@
 }
 
 // -----------------------------------------------------------------------------
+// Test library ELFs for linker page size related tests
+// -----------------------------------------------------------------------------
+cc_test_library {
+    name: "libtest_elf_max_page_size_4kib",
+    defaults: ["bionic_testlib_defaults"],
+    srcs: ["elf_max_page_size.c"],
+    ldflags: ["-z max-page-size=0x1000"],
+}
+
+// -----------------------------------------------------------------------------
 // Libraries and helper binaries for ELF TLS
 // -----------------------------------------------------------------------------
 cc_test_library {
@@ -1893,3 +1903,89 @@
         " $(location soong_zip) -o $(out).unaligned -L 0 -C $(genDir)/zipdir -D $(genDir)/zipdir &&" +
         " $(location bionic_tests_zipalign) 16384 $(out).unaligned $(out)",
 }
+
+cc_defaults {
+    name: "memtag_globals_defaults",
+    defaults: [
+        "bionic_testlib_defaults",
+        "bionic_targets_only"
+    ],
+    cflags: [
+        "-Wno-array-bounds",
+        "-Wno-unused-variable",
+    ],
+    header_libs: ["bionic_libc_platform_headers"],
+    sanitize: {
+        hwaddress: false,
+        memtag_heap: true,
+        memtag_globals: true,
+        diag: {
+            memtag_heap: true,
+        }
+    },
+}
+
+cc_test_library {
+    name: "memtag_globals_dso",
+    defaults: [ "memtag_globals_defaults" ],
+    srcs: ["memtag_globals_dso.cpp"],
+}
+
+cc_test {
+    name: "memtag_globals_binary",
+    defaults: [ "memtag_globals_defaults" ],
+    srcs: ["memtag_globals_binary.cpp"],
+    shared_libs: [ "memtag_globals_dso" ],
+    // This binary is used in the bionic-unit-tests as a data dependency, and is
+    // in the same folder as memtag_globals_dso. But, the default cc_test rules
+    // make this binary (when just explicitly built and shoved in
+    // /data/nativetest64/) end up in a subfolder called
+    // 'memtag_globals_binary'. When this happens, the explicit build fails to
+    // find the DSO because the default rpath is just ${ORIGIN}, and because we
+    // want this to be usable both from bionic-unit-tests and explicit builds,
+    // let's just not put it in a subdirectory.
+    no_named_install_directory: true,
+}
+
+cc_test {
+    name: "memtag_globals_binary_static",
+    defaults: [ "memtag_globals_defaults" ],
+    srcs: ["memtag_globals_binary.cpp"],
+    static_libs: [ "memtag_globals_dso" ],
+    no_named_install_directory: true,
+    static_executable: true,
+}
+
+// This is a regression test for b/314038442, where binaries built *without* MTE
+// globals would have out-of-bounds RELR relocations, which where then `ldg`'d,
+// which resulted in linker crashes.
+cc_test {
+  name: "mte_globals_relr_regression_test_b_314038442",
+  defaults: [
+        "bionic_testlib_defaults",
+        "bionic_targets_only"
+    ],
+    cflags: [ "-Wno-array-bounds" ],
+    ldflags: [ "-Wl,--pack-dyn-relocs=relr" ],
+    srcs: ["mte_globals_relr_regression_test_b_314038442.cpp"],
+    no_named_install_directory: true,
+    sanitize: {
+        memtag_globals: false,
+    },
+}
+
+// Same test as above, but also for MTE globals, just for the sake of it.
+cc_test {
+  name: "mte_globals_relr_regression_test_b_314038442_mte",
+  defaults: [
+        "bionic_testlib_defaults",
+        "bionic_targets_only"
+    ],
+    cflags: [ "-Wno-array-bounds" ],
+    ldflags: [ "-Wl,--pack-dyn-relocs=relr" ],
+    srcs: ["mte_globals_relr_regression_test_b_314038442.cpp"],
+    no_named_install_directory: true,
+    sanitize: {
+      memtag_globals: true,
+    },
+}
diff --git a/tests/libs/elf_max_page_size.c b/tests/libs/elf_max_page_size.c
new file mode 100644
index 0000000..24c7e89
--- /dev/null
+++ b/tests/libs/elf_max_page_size.c
@@ -0,0 +1,50 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "elf_max_page_size.h"
+
+const int ro0 = RO0;
+const int ro1 = RO1;
+int rw0 = RW0;
+
+/* Force some padding alignment */
+int rw1 __attribute__((aligned(0x10000))) = RW1;
+
+int bss0, bss1;
+
+int* const prw0 = &rw0;
+
+int loader_test_func(void) {
+  rw0 += RW0_INCREMENT;
+  rw1 += RW1_INCREMENT;
+
+  bss0 += BSS0_INCREMENT;
+  bss1 += BSS1_INCREMENT;
+
+  return ro0 + ro1 + rw0 + rw1 + bss0 + bss1 + *prw0;
+}
diff --git a/tests/libs/elf_max_page_size.h b/tests/libs/elf_max_page_size.h
new file mode 100644
index 0000000..846a8b6
--- /dev/null
+++ b/tests/libs/elf_max_page_size.h
@@ -0,0 +1,45 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#define RO0 23
+#define RO1 234
+#define RW0 2345
+#define RW1 23456
+#define BSS0 0
+#define BSS1 0
+
+#define RW0_INCREMENT 12
+#define RW1_INCREMENT 123
+#define BSS0_INCREMENT 1234
+#define BSS1_INCREMENT 12345
+
+#define TEST_RESULT_BASE (RO0 + RO1 + RW0 + RW1 + BSS0 + BSS1 + RW0)
+#define TEST_RESULT_INCREMENT \
+  (RW0_INCREMENT + RW1_INCREMENT + BSS0_INCREMENT + BSS1_INCREMENT + RW0_INCREMENT)
+
+typedef int (*loader_test_func_t)(void);
diff --git a/tests/libs/memtag_globals.h b/tests/libs/memtag_globals.h
new file mode 100644
index 0000000..a03abae
--- /dev/null
+++ b/tests/libs/memtag_globals.h
@@ -0,0 +1,43 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <utility>
+#include <vector>
+
+void check_tagged(const void* a);
+void check_untagged(const void* a);
+void check_matching_tags(const void* a, const void* b);
+void check_eq(const void* a, const void* b);
+
+void dso_check_assertions(bool enforce_tagged);
+void dso_print_variables();
+
+void print_variable_address(const char* name, const void* ptr);
+void print_variables(const char* header,
+                     const std::vector<std::pair<const char*, const void*>>& tagged_variables,
+                     const std::vector<std::pair<const char*, const void*>>& untagged_variables);
diff --git a/tests/libs/memtag_globals_binary.cpp b/tests/libs/memtag_globals_binary.cpp
new file mode 100644
index 0000000..9248728
--- /dev/null
+++ b/tests/libs/memtag_globals_binary.cpp
@@ -0,0 +1,195 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <string>
+#include <vector>
+
+#include "memtag_globals.h"
+
+// Adapted from the LLD test suite: lld/test/ELF/Inputs/aarch64-memtag-globals.s
+
+/// Global variables defined here, of various semantics.
+char global[30] = {};
+__attribute__((no_sanitize("memtag"))) int global_untagged = 0;
+const int const_global = 0;
+static const int hidden_const_global = 0;
+static char hidden_global[12] = {};
+__attribute__((visibility("hidden"))) int hidden_attr_global = 0;
+__attribute__((visibility("hidden"))) const int hidden_attr_const_global = 0;
+
+/// Should be untagged.
+__thread int tls_global;
+__thread static int hidden_tls_global;
+
+/// Tagged, from the other file.
+extern int global_extern;
+/// Untagged, from the other file.
+extern __attribute__((no_sanitize("memtag"))) int global_extern_untagged;
+/// Tagged here, but untagged in the definition found in the sister objfile
+/// (explicitly).
+extern int global_extern_untagged_definition_but_tagged_import;
+
+/// ABS64 relocations. Also, forces symtab entries for local and external
+/// globals.
+char* pointer_to_global = &global[0];
+char* pointer_inside_global = &global[17];
+char* pointer_to_global_end = &global[30];
+char* pointer_past_global_end = &global[48];
+int* pointer_to_global_untagged = &global_untagged;
+const int* pointer_to_const_global = &const_global;
+/// RELATIVE relocations.
+const int* pointer_to_hidden_const_global = &hidden_const_global;
+char* pointer_to_hidden_global = &hidden_global[0];
+int* pointer_to_hidden_attr_global = &hidden_attr_global;
+const int* pointer_to_hidden_attr_const_global = &hidden_attr_const_global;
+/// RELATIVE relocations with special AArch64 MemtagABI semantics, with the
+/// offset ('12' or '16') encoded in the place.
+char* pointer_to_hidden_global_end = &hidden_global[12];
+char* pointer_past_hidden_global_end = &hidden_global[16];
+/// ABS64 relocations.
+int* pointer_to_global_extern = &global_extern;
+int* pointer_to_global_extern_untagged = &global_extern_untagged;
+int* pointer_to_global_extern_untagged_definition_but_tagged_import =
+    &global_extern_untagged_definition_but_tagged_import;
+
+// Force materialization of these globals into the symtab.
+int* get_address_to_tls_global() {
+  return &tls_global;
+}
+int* get_address_to_hidden_tls_global() {
+  return &hidden_tls_global;
+}
+
+static const std::vector<std::pair<const char*, const void*>>& get_expected_tagged_vars() {
+  static std::vector<std::pair<const char*, const void*>> expected_tagged_vars = {
+      {"global", &global},
+      {"pointer_inside_global", pointer_inside_global},
+      {"pointer_to_global_end", pointer_to_global_end},
+      {"pointer_past_global_end", pointer_past_global_end},
+      {"hidden_global", &hidden_global},
+      {"hidden_attr_global", &hidden_attr_global},
+      {"global_extern", &global_extern},
+  };
+  return expected_tagged_vars;
+}
+
+static const std::vector<std::pair<const char*, const void*>>& get_expected_untagged_vars() {
+  static std::vector<std::pair<const char*, const void*>> expected_untagged_vars = {
+      {"global_extern_untagged", &global_extern_untagged},
+      {"global_extern_untagged_definition_but_tagged_import",
+       &global_extern_untagged_definition_but_tagged_import},
+      {"global_untagged", &global_untagged},
+      {"const_global", &const_global},
+      {"hidden_const_global", &hidden_const_global},
+      {"hidden_attr_const_global", &hidden_attr_const_global},
+      {"tls_global", &tls_global},
+      {"hidden_tls_global", &hidden_tls_global},
+  };
+  return expected_untagged_vars;
+}
+
+void exe_print_variables() {
+  print_variables("  Variables accessible from the binary:\n", get_expected_tagged_vars(),
+                  get_expected_untagged_vars());
+}
+
+// Dump the addresses of the global variables to stderr
+void dso_print();
+void dso_print_others();
+
+void exe_check_assertions(bool check_pointers_are_tagged) {
+  // Check that non-const variables are writeable.
+  *pointer_to_global = 0;
+  *pointer_inside_global = 0;
+  *(pointer_to_global_end - 1) = 0;
+  *pointer_to_global_untagged = 0;
+  *pointer_to_hidden_global = 0;
+  *pointer_to_hidden_attr_global = 0;
+  *(pointer_to_hidden_global_end - 1) = 0;
+  *pointer_to_global_extern = 0;
+  *pointer_to_global_extern_untagged = 0;
+  *pointer_to_global_extern_untagged_definition_but_tagged_import = 0;
+
+  if (check_pointers_are_tagged) {
+    for (const auto& [_, pointer] : get_expected_tagged_vars()) {
+      check_tagged(pointer);
+    }
+  }
+
+  for (const auto& [_, pointer] : get_expected_untagged_vars()) {
+    check_untagged(pointer);
+  }
+
+  check_matching_tags(pointer_to_global, pointer_inside_global);
+  check_matching_tags(pointer_to_global, pointer_to_global_end);
+  check_matching_tags(pointer_to_global, pointer_past_global_end);
+  check_eq(pointer_inside_global, pointer_to_global + 17);
+  check_eq(pointer_to_global_end, pointer_to_global + 30);
+  check_eq(pointer_past_global_end, pointer_to_global + 48);
+
+  check_matching_tags(pointer_to_hidden_global, pointer_to_hidden_global_end);
+  check_matching_tags(pointer_to_hidden_global, pointer_past_hidden_global_end);
+  check_eq(pointer_to_hidden_global_end, pointer_to_hidden_global + 12);
+  check_eq(pointer_past_hidden_global_end, pointer_to_hidden_global + 16);
+}
+
+void crash() {
+  *pointer_past_global_end = 0;
+}
+
+int main(int argc, char** argv) {
+  bool check_pointers_are_tagged = false;
+  // For an MTE-capable device, provide argv[1] == '1' to enable the assertions
+  // that pointers should be tagged.
+  if (argc >= 2 && argv[1][0] == '1') {
+    check_pointers_are_tagged = true;
+  }
+
+  char* heap_ptr = static_cast<char*>(malloc(1));
+  print_variable_address("heap address", heap_ptr);
+  *heap_ptr = 0;
+  if (check_pointers_are_tagged) check_tagged(heap_ptr);
+  free(heap_ptr);
+
+  exe_print_variables();
+  dso_print_variables();
+
+  exe_check_assertions(check_pointers_are_tagged);
+  dso_check_assertions(check_pointers_are_tagged);
+
+  printf("Assertions were passed. Now doing a global-buffer-overflow.\n");
+  fflush(stdout);
+  crash();
+  printf("global-buffer-overflow went uncaught.\n");
+  return 0;
+}
diff --git a/tests/libs/memtag_globals_dso.cpp b/tests/libs/memtag_globals_dso.cpp
new file mode 100644
index 0000000..9ed264e
--- /dev/null
+++ b/tests/libs/memtag_globals_dso.cpp
@@ -0,0 +1,165 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <vector>
+
+#include "memtag_globals.h"
+
+// Adapted from the LLD test suite: lld/test/ELF/Inputs/aarch64-memtag-globals.s
+
+int global_extern;
+static int global_extern_hidden;
+__attribute__((no_sanitize("memtag"))) int global_extern_untagged;
+__attribute__((no_sanitize("memtag"))) int global_extern_untagged_definition_but_tagged_import;
+
+void assertion_failure() {
+  exit(1);
+}
+
+void check_tagged(const void* a) {
+  uintptr_t a_uptr = reinterpret_cast<uintptr_t>(a);
+#if defined(__aarch64__)
+  if ((a_uptr >> 56) == 0) {
+    fprintf(stderr, "**********************************\n");
+    fprintf(stderr, "Failed assertion:\n");
+    fprintf(stderr, "  tag(0x%zx) != 0\n", a_uptr);
+    fprintf(stderr, "**********************************\n");
+
+    assertion_failure();
+  }
+#endif  // defined(__aarch64__)
+}
+
+void check_untagged(const void* a) {
+  uintptr_t a_uptr = reinterpret_cast<uintptr_t>(a);
+#if defined(__aarch64__)
+  if ((a_uptr >> 56) != 0) {
+    fprintf(stderr, "**********************************\n");
+    fprintf(stderr, "Failed assertion:\n");
+    fprintf(stderr, "  tag(0x%zx) == 0\n", a_uptr);
+    fprintf(stderr, "**********************************\n");
+
+    assertion_failure();
+  }
+#endif  // defined(__aarch64__)
+}
+
+void check_matching_tags(const void* a, const void* b) {
+  uintptr_t a_uptr = reinterpret_cast<uintptr_t>(a);
+  uintptr_t b_uptr = reinterpret_cast<uintptr_t>(b);
+#if defined(__aarch64__)
+  if (a_uptr >> 56 != b_uptr >> 56) {
+    fprintf(stderr, "**********************************\n");
+    fprintf(stderr, "Failed assertion:\n");
+    fprintf(stderr, "  tag(0x%zx) != tag(0x%zx)\n", a_uptr, b_uptr);
+    fprintf(stderr, "**********************************\n");
+
+    assertion_failure();
+  }
+#endif  // defined(__aarch64__)
+}
+
+void check_eq(const void* a, const void* b) {
+  if (a != b) {
+    fprintf(stderr, "**********************************\n");
+    fprintf(stderr, "Failed assertion:\n");
+    fprintf(stderr, "  %p != %p\n", a, b);
+    fprintf(stderr, "**********************************\n");
+
+    assertion_failure();
+  }
+}
+
+#define LONGEST_VARIABLE_NAME "51"
+void print_variable_address(const char* name, const void* ptr) {
+  printf("%" LONGEST_VARIABLE_NAME "s: %16p\n", name, ptr);
+}
+
+static const std::vector<std::pair<const char*, const void*>>& get_expected_tagged_vars() {
+  static std::vector<std::pair<const char*, const void*>> expected_tagged_vars = {
+      {"global_extern", &global_extern},
+      {"global_extern_hidden", &global_extern_hidden},
+  };
+  return expected_tagged_vars;
+}
+
+static const std::vector<std::pair<const char*, const void*>>& get_expected_untagged_vars() {
+  static std::vector<std::pair<const char*, const void*>> expected_untagged_vars = {
+      {"global_extern_untagged", &global_extern_untagged},
+      {"global_extern_untagged_definition_but_tagged_import",
+       &global_extern_untagged_definition_but_tagged_import},
+  };
+  return expected_untagged_vars;
+}
+
+void dso_print_variables() {
+  print_variables("  Variables declared in the DSO:\n", get_expected_tagged_vars(),
+                  get_expected_untagged_vars());
+}
+
+void print_variables(const char* header,
+                     const std::vector<std::pair<const char*, const void*>>& tagged_variables,
+                     const std::vector<std::pair<const char*, const void*>>& untagged_variables) {
+  printf("==========================================================\n");
+  printf("%s", header);
+  printf("==========================================================\n");
+  printf(" Variables expected to be tagged:\n");
+  printf("----------------------------------------------------------\n");
+  for (const auto& [name, pointer] : tagged_variables) {
+    print_variable_address(name, pointer);
+  }
+
+  printf("\n----------------------------------------------------------\n");
+  printf(" Variables expected to be untagged:\n");
+  printf("----------------------------------------------------------\n");
+  for (const auto& [name, pointer] : untagged_variables) {
+    print_variable_address(name, pointer);
+  }
+  printf("\n");
+}
+
+void dso_check_assertions(bool check_pointers_are_tagged) {
+  // Check that non-const variables are writeable.
+  global_extern = 0;
+  global_extern_hidden = 0;
+  global_extern_untagged = 0;
+  global_extern_untagged_definition_but_tagged_import = 0;
+
+  if (check_pointers_are_tagged) {
+    for (const auto& [_, pointer] : get_expected_tagged_vars()) {
+      check_tagged(pointer);
+    }
+  }
+
+  for (const auto& [_, pointer] : get_expected_untagged_vars()) {
+    check_untagged(pointer);
+  }
+}
diff --git a/tests/libs/mte_globals_relr_regression_test_b_314038442.cpp b/tests/libs/mte_globals_relr_regression_test_b_314038442.cpp
new file mode 100644
index 0000000..20bbba9
--- /dev/null
+++ b/tests/libs/mte_globals_relr_regression_test_b_314038442.cpp
@@ -0,0 +1,55 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <stdio.h>
+
+static volatile char array[0x10000];
+volatile char* volatile oob_ptr = &array[0x111111111];
+
+unsigned char get_tag(__attribute__((unused)) volatile void* ptr) {
+#if defined(__aarch64__)
+  return static_cast<unsigned char>(reinterpret_cast<uintptr_t>(ptr) >> 56) & 0xf;
+#else   // !defined(__aarch64__)
+  return 0;
+#endif  // defined(__aarch64__)
+}
+
+int main() {
+  printf("Program loaded successfully. %p %p. ", array, oob_ptr);
+  if (get_tag(array) != get_tag(oob_ptr)) {
+    printf("Tags are mismatched!\n");
+    return 1;
+  }
+  if (get_tag(array) == 0) {
+    printf("Tags are zero!\n");
+  } else {
+    printf("Tags are non-zero\n");
+  }
+  return 0;
+}
diff --git a/tests/malloc_stress_test.cpp b/tests/malloc_stress_test.cpp
index b5b06ab..00f5919 100644
--- a/tests/malloc_stress_test.cpp
+++ b/tests/malloc_stress_test.cpp
@@ -27,10 +27,58 @@
 #include <thread>
 #include <vector>
 
-#include <android-base/strings.h>
 #if defined(__BIONIC__)
 #include <meminfo/procmeminfo.h>
 #include <procinfo/process_map.h>
+
+#include <log/log.h>
+#include <log/log_read.h>
+#endif
+
+#if defined(__BIONIC__)
+static void PrintLogStats(uint64_t& last_time) {
+  logger_list* logger =
+      android_logger_list_open(android_name_to_log_id("main"), ANDROID_LOG_NONBLOCK, 0, getpid());
+  if (logger == nullptr) {
+    printf("Failed to open log for main\n");
+    return;
+  }
+
+  uint64_t last_message_time = last_time;
+  while (true) {
+    log_msg entry;
+    ssize_t retval = android_logger_list_read(logger, &entry);
+    if (retval == 0) {
+      break;
+    }
+    if (retval < 0) {
+      if (retval == -EINTR) {
+        continue;
+      }
+      // EAGAIN means there is nothing left to read when ANDROID_LOG_NONBLOCK is set.
+      if (retval != -EAGAIN) {
+        printf("Failed to read log entry: %s\n", strerrordesc_np(retval));
+      }
+      break;
+    }
+    if (entry.msg() == nullptr) {
+      continue;
+    }
+    // Only print allocator tagged log entries.
+    std::string_view tag(entry.msg() + 1);
+    if (tag != "scudo" && tag != "jemalloc") {
+      continue;
+    }
+    if (entry.nsec() > last_time) {
+      printf("  %s\n", &tag.back() + 2);
+      // Only update the last time outside this loop just in case two or more
+      // messages have the same timestamp.
+      last_message_time = entry.nsec();
+    }
+  }
+  android_logger_list_close(logger);
+  last_time = last_message_time;
+}
 #endif
 
 TEST(malloc_stress, multiple_threads_forever) {
@@ -45,6 +93,8 @@
 #endif
   uint64_t mallinfo_min = UINT64_MAX;
   uint64_t mallinfo_max = 0;
+
+  uint64_t last_message_time = 0;
   for (size_t i = 0; ; i++) {
     printf("Pass %zu\n", i);
 
@@ -74,8 +124,8 @@
     uint64_t rss_bytes = 0;
     uint64_t vss_bytes = 0;
     for (auto& vma : maps) {
-      if (vma.name == "[anon:libc_malloc]" || android::base::StartsWith(vma.name, "[anon:scudo:") ||
-          android::base::StartsWith(vma.name, "[anon:GWP-ASan")) {
+      if (vma.name == "[anon:libc_malloc]" || vma.name.starts_with("[anon:scudo:") ||
+          vma.name.starts_with("[anon:GWP-ASan")) {
         android::meminfo::Vma update_vma(vma);
         ASSERT_TRUE(proc_mem.FillInVmaStats(update_vma));
         rss_bytes += update_vma.usage.rss;
@@ -112,5 +162,15 @@
     printf("Allocated memory %zu %0.2fMB\n", mallinfo_bytes, mallinfo_bytes / (1024.0 * 1024.0));
     printf("  Min %" PRIu64 " %0.2fMB\n", mallinfo_min, mallinfo_min / (1024.0 * 1024.0));
     printf("  Max %" PRIu64 " %0.2fMB\n", mallinfo_max, mallinfo_max / (1024.0 * 1024.0));
+
+#if defined(__BIONIC__)
+    if (((i + 1) % 100) == 0) {
+      // Send native allocator stats to the log
+      mallopt(M_LOG_STATS, 0);
+
+      printf("Log stats:\n");
+      PrintLogStats(last_message_time);
+    }
+#endif
   }
 }
diff --git a/tests/malloc_test.cpp b/tests/malloc_test.cpp
index 813f348..3f1ba79 100644
--- a/tests/malloc_test.cpp
+++ b/tests/malloc_test.cpp
@@ -47,6 +47,7 @@
 #include <android-base/file.h>
 #include <android-base/test_utils.h>
 
+#include "DoNotOptimize.h"
 #include "utils.h"
 
 #if defined(__BIONIC__)
diff --git a/tests/memtag_globals_test.cpp b/tests/memtag_globals_test.cpp
new file mode 100644
index 0000000..ff93e7b
--- /dev/null
+++ b/tests/memtag_globals_test.cpp
@@ -0,0 +1,113 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <gtest/gtest.h>
+
+#if defined(__BIONIC__)
+#include "gtest_globals.h"
+#include "utils.h"
+#endif  // defined(__BIONIC__)
+
+#include <android-base/test_utils.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <string>
+#include <tuple>
+
+#include "platform/bionic/mte.h"
+
+class MemtagGlobalsTest : public testing::TestWithParam<bool> {};
+
+TEST_P(MemtagGlobalsTest, test) {
+  SKIP_WITH_HWASAN << "MTE globals tests are incompatible with HWASan";
+#if defined(__BIONIC__) && defined(__aarch64__)
+  std::string binary = GetTestLibRoot() + "/memtag_globals_binary";
+  bool is_static = MemtagGlobalsTest::GetParam();
+  if (is_static) {
+    binary += "_static";
+  }
+
+  chmod(binary.c_str(), 0755);
+  ExecTestHelper eth;
+  eth.SetArgs({binary.c_str(), nullptr});
+  eth.Run(
+      [&]() {
+        execve(binary.c_str(), eth.GetArgs(), eth.GetEnv());
+        GTEST_FAIL() << "Failed to execve: " << strerror(errno) << " " << binary.c_str();
+      },
+      // We catch the global-buffer-overflow and crash only when MTE globals is
+      // supported. Note that MTE globals is unsupported for fully static
+      // executables, but we should still make sure the binary passes its
+      // assertions, just that global variables won't be tagged.
+      (mte_supported() && !is_static) ? -SIGSEGV : 0, "Assertions were passed");
+#else
+  GTEST_SKIP() << "bionic/arm64 only";
+#endif
+}
+
+INSTANTIATE_TEST_SUITE_P(MemtagGlobalsTest, MemtagGlobalsTest, testing::Bool(),
+                         [](const ::testing::TestParamInfo<MemtagGlobalsTest::ParamType>& info) {
+                           if (info.param) return "MemtagGlobalsTest_static";
+                           return "MemtagGlobalsTest";
+                         });
+
+TEST(MemtagGlobalsTest, RelrRegressionTestForb314038442) {
+  SKIP_WITH_HWASAN << "MTE globals tests are incompatible with HWASan";
+#if defined(__BIONIC__) && defined(__aarch64__)
+  std::string binary = GetTestLibRoot() + "/mte_globals_relr_regression_test_b_314038442";
+  chmod(binary.c_str(), 0755);
+  ExecTestHelper eth;
+  eth.SetArgs({binary.c_str(), nullptr});
+  eth.Run(
+      [&]() {
+        execve(binary.c_str(), eth.GetArgs(), eth.GetEnv());
+        GTEST_FAIL() << "Failed to execve: " << strerror(errno) << " " << binary.c_str();
+      },
+      /* exit code */ 0, "Program loaded successfully.*Tags are zero!");
+#else
+  GTEST_SKIP() << "bionic/arm64 only";
+#endif
+}
+
+TEST(MemtagGlobalsTest, RelrRegressionTestForb314038442WithMteGlobals) {
+  if (!mte_supported()) GTEST_SKIP() << "Must have MTE support.";
+#if defined(__BIONIC__) && defined(__aarch64__)
+  std::string binary = GetTestLibRoot() + "/mte_globals_relr_regression_test_b_314038442_mte";
+  chmod(binary.c_str(), 0755);
+  ExecTestHelper eth;
+  eth.SetArgs({binary.c_str(), nullptr});
+  eth.Run(
+      [&]() {
+        execve(binary.c_str(), eth.GetArgs(), eth.GetEnv());
+        GTEST_FAIL() << "Failed to execve: " << strerror(errno) << " " << binary.c_str();
+      },
+      /* exit code */ 0, "Program loaded successfully.*Tags are non-zero");
+#else
+  GTEST_SKIP() << "bionic/arm64 only";
+#endif
+}
diff --git a/tests/memtag_stack_abi_test.cpp b/tests/memtag_stack_abi_test.cpp
new file mode 100644
index 0000000..4725c8d
--- /dev/null
+++ b/tests/memtag_stack_abi_test.cpp
@@ -0,0 +1,102 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <filesystem>
+#include <fstream>
+#include <iterator>
+#include <string>
+#include <thread>
+
+#include <dlfcn.h>
+#include <stdlib.h>
+
+#include <android-base/logging.h>
+#include <gtest/gtest.h>
+
+static size_t NumberBuffers() {
+  size_t bufs = 0;
+  std::ifstream file("/proc/self/maps");
+  CHECK(file.is_open());
+  std::string line;
+  while (std::getline(file, line)) {
+    if (line.find("stack_mte_ring") != std::string::npos) {
+      ++bufs;
+    }
+  }
+  return bufs;
+}
+
+static size_t NumberThreads() {
+  std::filesystem::directory_iterator di("/proc/self/task");
+  return std::distance(begin(di), end(di));
+}
+
+TEST(MemtagStackAbiTest, MainThread) {
+#if defined(__BIONIC__) && defined(__aarch64__)
+  ASSERT_EQ(NumberBuffers(), 1U);
+  ASSERT_EQ(NumberBuffers(), NumberThreads());
+#else
+  GTEST_SKIP() << "requires bionic arm64";
+#endif
+}
+
+TEST(MemtagStackAbiTest, JoinableThread) {
+#if defined(__BIONIC__) && defined(__aarch64__)
+  ASSERT_EQ(NumberBuffers(), 1U);
+  ASSERT_EQ(NumberBuffers(), NumberThreads());
+  std::thread th([] {
+    ASSERT_EQ(NumberBuffers(), 2U);
+    ASSERT_EQ(NumberBuffers(), NumberThreads());
+  });
+  th.join();
+  ASSERT_EQ(NumberBuffers(), 1U);
+  ASSERT_EQ(NumberBuffers(), NumberThreads());
+#else
+  GTEST_SKIP() << "requires bionic arm64";
+#endif
+}
+
+TEST(MemtagStackAbiTest, DetachedThread) {
+#if defined(__BIONIC__) && defined(__aarch64__)
+  ASSERT_EQ(NumberBuffers(), 1U);
+  ASSERT_EQ(NumberBuffers(), NumberThreads());
+  std::thread th([] {
+    ASSERT_EQ(NumberBuffers(), 2U);
+    ASSERT_EQ(NumberBuffers(), NumberThreads());
+  });
+  th.detach();
+  // Leave the thread some time to exit.
+  for (int i = 0; NumberBuffers() != 1 && i < 3; ++i) {
+    sleep(1);
+  }
+  ASSERT_EQ(NumberBuffers(), 1U);
+  ASSERT_EQ(NumberBuffers(), NumberThreads());
+#else
+  GTEST_SKIP() << "requires bionic arm64";
+#endif
+}
diff --git a/tests/page_size_16kib_compat_test.cpp b/tests/page_size_16kib_compat_test.cpp
new file mode 100644
index 0000000..a5d91b8
--- /dev/null
+++ b/tests/page_size_16kib_compat_test.cpp
@@ -0,0 +1,62 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "page_size_compat_helpers.h"
+
+#include <android-base/properties.h>
+
+extern "C" void android_set_16kb_appcompat_mode(bool enable_app_compat);
+
+TEST(PageSize16KiBCompatTest, ElfAlignment4KiB_LoadElf) {
+  if (getpagesize() != 0x4000) {
+    GTEST_SKIP() << "This test is only applicable to 16kB page-size devices";
+  }
+
+  bool app_compat_enabled =
+      android::base::GetBoolProperty("bionic.linker.16kb.app_compat.enabled", false);
+  std::string lib = GetTestLibRoot() + "/libtest_elf_max_page_size_4kib.so";
+  void* handle = nullptr;
+
+  OpenTestLibrary(lib, !app_compat_enabled, &handle);
+
+  if (app_compat_enabled) CallTestFunction(handle);
+}
+
+TEST(PageSize16KiBCompatTest, ElfAlignment4KiB_LoadElf_perAppOption) {
+  if (getpagesize() != 0x4000) {
+    GTEST_SKIP() << "This test is only applicable to 16kB page-size devices";
+  }
+
+  android_set_16kb_appcompat_mode(true);
+  std::string lib = GetTestLibRoot() + "/libtest_elf_max_page_size_4kib.so";
+  void* handle = nullptr;
+
+  OpenTestLibrary(lib, false /*should_fail*/, &handle);
+  CallTestFunction(handle);
+  android_set_16kb_appcompat_mode(false);
+}
diff --git a/tests/page_size_compat_helpers.h b/tests/page_size_compat_helpers.h
new file mode 100644
index 0000000..2f0f1d0
--- /dev/null
+++ b/tests/page_size_compat_helpers.h
@@ -0,0 +1,75 @@
+/*
+ * Copyright (C) 2024 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "elf_max_page_size.h"
+#include "gtest_globals.h"
+
+#include <android-base/stringprintf.h>
+
+#include <string>
+
+#include <dlfcn.h>
+#include <gtest/gtest.h>
+#include <unistd.h>
+
+static inline void OpenTestLibrary(std::string lib, bool expect_fail, void** handle) {
+  void* _handle = dlopen(lib.c_str(), RTLD_NODELETE);
+  const char* dlopen_error = dlerror();
+
+  if (expect_fail) {
+    ASSERT_EQ(_handle, nullptr);
+
+    const std::string expected_error = android::base::StringPrintf(
+        "dlopen failed: \"%s\" program alignment (%d) cannot be smaller than system page size (%d)",
+        lib.c_str(), 4096, getpagesize());
+
+    ASSERT_EQ(expected_error, dlopen_error);
+  } else {
+    ASSERT_NE(_handle, nullptr) << "Failed to dlopen shared library \"" << lib
+                                << "\": " << dlopen_error;
+  }
+
+  *handle = _handle;
+}
+
+static inline void CallTestFunction(void* handle) {
+  loader_test_func_t loader_test_func = (loader_test_func_t)dlsym(handle, "loader_test_func");
+  const char* dlsym_error = dlerror();
+
+  ASSERT_EQ(dlsym_error, nullptr) << "Failed to locate symbol \"loader_test_func\": "
+                                  << dlsym_error;
+
+  int res = loader_test_func();
+  ASSERT_EQ(res, TEST_RESULT_BASE + TEST_RESULT_INCREMENT);
+
+  // Call loader_test_func() twice to ensure we can modify writeable data and bss data
+  res = loader_test_func();
+  ASSERT_EQ(res, TEST_RESULT_BASE + (2 * TEST_RESULT_INCREMENT));
+}
diff --git a/tests/prebuilt-elf-files/arm64/libtest_invalid-empty_shdr_table.so b/tests/prebuilt-elf-files/arm64/libtest_invalid-empty_shdr_table.so
index c8b5430..21a8f26 100755
--- a/tests/prebuilt-elf-files/arm64/libtest_invalid-empty_shdr_table.so
+++ b/tests/prebuilt-elf-files/arm64/libtest_invalid-empty_shdr_table.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/arm64/libtest_invalid-local-tls.so b/tests/prebuilt-elf-files/arm64/libtest_invalid-local-tls.so
index 20c5765..c902bbe 100755
--- a/tests/prebuilt-elf-files/arm64/libtest_invalid-local-tls.so
+++ b/tests/prebuilt-elf-files/arm64/libtest_invalid-local-tls.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/arm64/libtest_invalid-unaligned_shdr_offset.so b/tests/prebuilt-elf-files/arm64/libtest_invalid-unaligned_shdr_offset.so
index 6e5a6e3..fb86bca 100755
--- a/tests/prebuilt-elf-files/arm64/libtest_invalid-unaligned_shdr_offset.so
+++ b/tests/prebuilt-elf-files/arm64/libtest_invalid-unaligned_shdr_offset.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_content.so b/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_content.so
index 14b80b5..0416db2 100755
--- a/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_content.so
+++ b/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_content.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_offset.so b/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_offset.so
index 0aaca72..90892a6 100755
--- a/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_offset.so
+++ b/tests/prebuilt-elf-files/arm64/libtest_invalid-zero_shdr_table_offset.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/gen-libtest_invalid-local-tls.sh b/tests/prebuilt-elf-files/gen-libtest_invalid-local-tls.sh
index 0f3e736..98a2b00 100755
--- a/tests/prebuilt-elf-files/gen-libtest_invalid-local-tls.sh
+++ b/tests/prebuilt-elf-files/gen-libtest_invalid-local-tls.sh
@@ -19,12 +19,18 @@
 build() {
   arch=$1
   target=$2
+
+  if [[ "$arch" == "arm64" || "$arch" == "x86_64" ]]; then
+    alignment="-Wl,-z,max-page-size=16384"
+  fi
+
   $NDK21E/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -O2 --target=$target \
       -fpic -shared -o $arch/libtest_invalid-local-tls.so -fno-emulated-tls \
-      -fuse-ld=gold test.c test2.c
+      $alignment -fuse-ld=gold test.c test2.c
 }
 
 build arm armv7a-linux-androideabi29
 build arm64 aarch64-linux-android29
 build x86 i686-linux-android29
 build x86_64 x86_64-linux-android29
+
diff --git a/tests/prebuilt-elf-files/x86_64/libtest_invalid-empty_shdr_table.so b/tests/prebuilt-elf-files/x86_64/libtest_invalid-empty_shdr_table.so
index af1538d..00fefd4 100755
--- a/tests/prebuilt-elf-files/x86_64/libtest_invalid-empty_shdr_table.so
+++ b/tests/prebuilt-elf-files/x86_64/libtest_invalid-empty_shdr_table.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/x86_64/libtest_invalid-local-tls.so b/tests/prebuilt-elf-files/x86_64/libtest_invalid-local-tls.so
index 5b689ba..31d2b37 100755
--- a/tests/prebuilt-elf-files/x86_64/libtest_invalid-local-tls.so
+++ b/tests/prebuilt-elf-files/x86_64/libtest_invalid-local-tls.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/x86_64/libtest_invalid-unaligned_shdr_offset.so b/tests/prebuilt-elf-files/x86_64/libtest_invalid-unaligned_shdr_offset.so
index 87631af..f9c310f 100755
--- a/tests/prebuilt-elf-files/x86_64/libtest_invalid-unaligned_shdr_offset.so
+++ b/tests/prebuilt-elf-files/x86_64/libtest_invalid-unaligned_shdr_offset.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_content.so b/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_content.so
index 27d1138..3d1f5d3 100755
--- a/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_content.so
+++ b/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_content.so
Binary files differ
diff --git a/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_offset.so b/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_offset.so
index 3e2c1d1..aeea1d2 100755
--- a/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_offset.so
+++ b/tests/prebuilt-elf-files/x86_64/libtest_invalid-zero_shdr_table_offset.so
Binary files differ
diff --git a/tests/pthread_test.cpp b/tests/pthread_test.cpp
index 2bf755b..5ce7d4d 100644
--- a/tests/pthread_test.cpp
+++ b/tests/pthread_test.cpp
@@ -45,6 +45,7 @@
 #include <android-base/test_utils.h>
 
 #include "private/bionic_constants.h"
+#include "private/bionic_time_conversions.h"
 #include "SignalUtils.h"
 #include "utils.h"
 
@@ -2437,23 +2438,25 @@
   ts.tv_sec = -1;
   ASSERT_EQ(ETIMEDOUT, lock_function(&m, &ts));
 
-  // check we wait long enough for the lock.
+  // Check we wait long enough for the lock before timing out...
+
+  // What time is it before we start?
   ASSERT_EQ(0, clock_gettime(clock, &ts));
-  const int64_t start_ns = ts.tv_sec * NS_PER_S + ts.tv_nsec;
-
-  // add a second to get deadline.
+  const int64_t start_ns = to_ns(ts);
+  // Add a second to get deadline, and wait until we time out.
   ts.tv_sec += 1;
-
   ASSERT_EQ(ETIMEDOUT, lock_function(&m, &ts));
 
+  // What time is it now we've timed out?
+  timespec ts2;
+  clock_gettime(clock, &ts2);
+  const int64_t end_ns = to_ns(ts2);
+
   // The timedlock must have waited at least 1 second before returning.
-  clock_gettime(clock, &ts);
-  const int64_t end_ns = ts.tv_sec * NS_PER_S + ts.tv_nsec;
-  ASSERT_GT(end_ns - start_ns, NS_PER_S);
+  ASSERT_GE(end_ns - start_ns, NS_PER_S);
 
   // If the mutex is unlocked, pthread_mutex_timedlock should succeed.
   ASSERT_EQ(0, pthread_mutex_unlock(&m));
-
   ASSERT_EQ(0, clock_gettime(clock, &ts));
   ts.tv_sec += 1;
   ASSERT_EQ(0, lock_function(&m, &ts));
@@ -2474,12 +2477,19 @@
 #endif  // __BIONIC__
 }
 
-TEST(pthread, pthread_mutex_clocklock) {
+TEST(pthread, pthread_mutex_clocklock_MONOTONIC) {
 #if defined(__BIONIC__)
   pthread_mutex_timedlock_helper(
       CLOCK_MONOTONIC, [](pthread_mutex_t* __mutex, const timespec* __timeout) {
         return pthread_mutex_clocklock(__mutex, CLOCK_MONOTONIC, __timeout);
       });
+#else   // __BIONIC__
+  GTEST_SKIP() << "pthread_mutex_clocklock not available";
+#endif  // __BIONIC__
+}
+
+TEST(pthread, pthread_mutex_clocklock_REALTIME) {
+#if defined(__BIONIC__)
   pthread_mutex_timedlock_helper(
       CLOCK_REALTIME, [](pthread_mutex_t* __mutex, const timespec* __timeout) {
         return pthread_mutex_clocklock(__mutex, CLOCK_REALTIME, __timeout);
@@ -3127,3 +3137,39 @@
   GTEST_SKIP() << "bionic-only test";
 #endif
 }
+
+TEST(pthread, pthread_getaffinity_np_failure) {
+  // Trivial test of the errno-preserving/returning behavior.
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wnonnull"
+  errno = 0;
+  ASSERT_EQ(EINVAL, pthread_getaffinity_np(pthread_self(), 0, nullptr));
+  ASSERT_ERRNO(0);
+#pragma clang diagnostic pop
+}
+
+TEST(pthread, pthread_getaffinity) {
+  cpu_set_t set;
+  CPU_ZERO(&set);
+  ASSERT_EQ(0, pthread_getaffinity_np(pthread_self(), sizeof(set), &set));
+  ASSERT_GT(CPU_COUNT(&set), 0);
+}
+
+TEST(pthread, pthread_setaffinity_np_failure) {
+  // Trivial test of the errno-preserving/returning behavior.
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wnonnull"
+  errno = 0;
+  ASSERT_EQ(EINVAL, pthread_setaffinity_np(pthread_self(), 0, nullptr));
+  ASSERT_ERRNO(0);
+#pragma clang diagnostic pop
+}
+
+TEST(pthread, pthread_setaffinity) {
+  cpu_set_t set;
+  CPU_ZERO(&set);
+  ASSERT_EQ(0, pthread_getaffinity_np(pthread_self(), sizeof(set), &set));
+  // It's hard to make any more general claim than this,
+  // but it ought to be safe to ask for the same affinity you already have.
+  ASSERT_EQ(0, pthread_setaffinity_np(pthread_self(), sizeof(set), &set));
+}
diff --git a/tests/sched_test.cpp b/tests/sched_test.cpp
index 0231de4..448fae9 100644
--- a/tests/sched_test.cpp
+++ b/tests/sched_test.cpp
@@ -305,8 +305,35 @@
 }
 
 TEST(sched, sched_getaffinity_failure) {
+  // Trivial test of the errno-preserving/returning behavior.
 #pragma clang diagnostic push
 #pragma clang diagnostic ignored "-Wnonnull"
   ASSERT_EQ(-1, sched_getaffinity(getpid(), 0, nullptr));
+  ASSERT_ERRNO(EINVAL);
 #pragma clang diagnostic pop
 }
+
+TEST(pthread, sched_getaffinity) {
+  cpu_set_t set;
+  CPU_ZERO(&set);
+  ASSERT_EQ(0, sched_getaffinity(getpid(), sizeof(set), &set));
+  ASSERT_GT(CPU_COUNT(&set), 0);
+}
+
+TEST(sched, sched_setaffinity_failure) {
+  // Trivial test of the errno-preserving/returning behavior.
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wnonnull"
+  ASSERT_EQ(-1, sched_setaffinity(getpid(), 0, nullptr));
+  ASSERT_ERRNO(EINVAL);
+#pragma clang diagnostic pop
+}
+
+TEST(pthread, sched_setaffinity) {
+  cpu_set_t set;
+  CPU_ZERO(&set);
+  ASSERT_EQ(0, sched_getaffinity(getpid(), sizeof(set), &set));
+  // It's hard to make any more general claim than this,
+  // but it ought to be safe to ask for the same affinity you already have.
+  ASSERT_EQ(0, sched_setaffinity(getpid(), sizeof(set), &set));
+}
diff --git a/tests/stdatomic_test.cpp b/tests/stdatomic_test.cpp
index 1c51b11..8a54080 100644
--- a/tests/stdatomic_test.cpp
+++ b/tests/stdatomic_test.cpp
@@ -40,7 +40,7 @@
   atomic_int v = 123;
   ASSERT_EQ(123, atomic_load(&v));
 
-  atomic_init(&v, 456);
+  atomic_store_explicit(&v, 456, memory_order_relaxed);
   ASSERT_EQ(456, atomic_load(&v));
 
   atomic_flag f = ATOMIC_FLAG_INIT;
@@ -258,9 +258,9 @@
   // Run a memory ordering smoke test.
   void* result;
   three_atomics a;
-  atomic_init(&a.x, 0ul);
-  atomic_init(&a.y, 0ul);
-  atomic_init(&a.z, 0ul);
+  atomic_store_explicit(&a.x, 0ul, memory_order_relaxed);
+  atomic_store_explicit(&a.y, 0ul, memory_order_relaxed);
+  atomic_store_explicit(&a.z, 0ul, memory_order_relaxed);
   pthread_t t1,t2;
   ASSERT_EQ(0, pthread_create(&t1, nullptr, reader, &a));
   ASSERT_EQ(0, pthread_create(&t2, nullptr, writer, &a));
diff --git a/tests/string_test.cpp b/tests/string_test.cpp
index 6e1fcfc..502405f 100644
--- a/tests/string_test.cpp
+++ b/tests/string_test.cpp
@@ -740,15 +740,11 @@
     // Set the second half of ptr to the expected pattern in ptr2.
     memset(state.ptr + state.MAX_LEN, '\1', state.MAX_LEN);
     memcpy(state.ptr + state.MAX_LEN, state.ptr1, copy_len);
-    size_t expected_end;
     if (copy_len > ptr1_len) {
       memset(state.ptr + state.MAX_LEN + ptr1_len, '\0', copy_len - ptr1_len);
-      expected_end = ptr1_len;
-    } else {
-      expected_end = copy_len;
     }
 
-    ASSERT_EQ(state.ptr2 + expected_end, stpncpy(state.ptr2, state.ptr1, copy_len));
+    ASSERT_EQ(state.ptr2, strncpy(state.ptr2, state.ptr1, copy_len));
 
     // Verify ptr1 was not modified.
     ASSERT_EQ(0, memcmp(state.ptr1, state.ptr, state.MAX_LEN));
diff --git a/tests/struct_layout_test.cpp b/tests/struct_layout_test.cpp
index 1f04344..b9fd315 100644
--- a/tests/struct_layout_test.cpp
+++ b/tests/struct_layout_test.cpp
@@ -30,7 +30,7 @@
 #define CHECK_OFFSET(name, field, offset) \
     check_offset(#name, #field, offsetof(name, field), offset);
 #ifdef __LP64__
-  CHECK_SIZE(pthread_internal_t, 816);
+  CHECK_SIZE(pthread_internal_t, 824);
   CHECK_OFFSET(pthread_internal_t, next, 0);
   CHECK_OFFSET(pthread_internal_t, prev, 8);
   CHECK_OFFSET(pthread_internal_t, tid, 16);
@@ -57,6 +57,7 @@
   CHECK_OFFSET(pthread_internal_t, errno_value, 768);
   CHECK_OFFSET(pthread_internal_t, bionic_tcb, 776);
   CHECK_OFFSET(pthread_internal_t, stack_mte_ringbuffer_vma_name_buffer, 784);
+  CHECK_OFFSET(pthread_internal_t, should_allocate_stack_mte_ringbuffer, 816);
   CHECK_SIZE(bionic_tls, 12200);
   CHECK_OFFSET(bionic_tls, key_data, 0);
   CHECK_OFFSET(bionic_tls, locale, 2080);
@@ -74,7 +75,7 @@
   CHECK_OFFSET(bionic_tls, bionic_systrace_disabled, 12193);
   CHECK_OFFSET(bionic_tls, padding, 12194);
 #else
-  CHECK_SIZE(pthread_internal_t, 704);
+  CHECK_SIZE(pthread_internal_t, 708);
   CHECK_OFFSET(pthread_internal_t, next, 0);
   CHECK_OFFSET(pthread_internal_t, prev, 4);
   CHECK_OFFSET(pthread_internal_t, tid, 8);
@@ -101,6 +102,7 @@
   CHECK_OFFSET(pthread_internal_t, errno_value, 664);
   CHECK_OFFSET(pthread_internal_t, bionic_tcb, 668);
   CHECK_OFFSET(pthread_internal_t, stack_mte_ringbuffer_vma_name_buffer, 672);
+  CHECK_OFFSET(pthread_internal_t, should_allocate_stack_mte_ringbuffer, 704);
   CHECK_SIZE(bionic_tls, 11080);
   CHECK_OFFSET(bionic_tls, key_data, 0);
   CHECK_OFFSET(bionic_tls, locale, 1040);
diff --git a/tests/sys_time_test.cpp b/tests/sys_time_test.cpp
index ff9271f..b0e52aa 100644
--- a/tests/sys_time_test.cpp
+++ b/tests/sys_time_test.cpp
@@ -23,6 +23,7 @@
 
 #include <android-base/file.h>
 
+#include "private/bionic_time_conversions.h"
 #include "utils.h"
 
 // http://b/11383777
@@ -147,14 +148,6 @@
   ASSERT_EQ(0, syscall(__NR_gettimeofday, &tv2, nullptr));
 
   // What's the difference between the two?
-  tv2.tv_sec -= tv1.tv_sec;
-  tv2.tv_usec -= tv1.tv_usec;
-  if (tv2.tv_usec < 0) {
-    --tv2.tv_sec;
-    tv2.tv_usec += 1000000;
-  }
-
   // To try to avoid flakiness we'll accept answers within 10,000us (0.01s).
-  ASSERT_EQ(0, tv2.tv_sec);
-  ASSERT_LT(tv2.tv_usec, 10'000);
+  ASSERT_LT(to_us(tv2) - to_us(tv1), 10'000);
 }
diff --git a/tests/unistd_test.cpp b/tests/unistd_test.cpp
index 78b55c1..9ad3b6d 100644
--- a/tests/unistd_test.cpp
+++ b/tests/unistd_test.cpp
@@ -16,6 +16,7 @@
 
 #include <gtest/gtest.h>
 
+#include "DoNotOptimize.h"
 #include "SignalUtils.h"
 #include "utils.h"
 
diff --git a/tests/utils.h b/tests/utils.h
index 3c83b73..4740e59 100644
--- a/tests/utils.h
+++ b/tests/utils.h
@@ -295,16 +295,6 @@
   size_t start_count_ = CountOpenFds();
 };
 
-// From <benchmark/benchmark.h>.
-template <class Tp>
-static inline void DoNotOptimize(Tp const& value) {
-  asm volatile("" : : "r,m"(value) : "memory");
-}
-template <class Tp>
-static inline void DoNotOptimize(Tp& value) {
-  asm volatile("" : "+r,m"(value) : : "memory");
-}
-
 static inline bool running_with_mte() {
 #ifdef __aarch64__
   int level = prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0);