Merge "Bump tzcode from 2016g to 2022a*."
diff --git a/docs/clang_fortify_anatomy.md b/docs/clang_fortify_anatomy.md
new file mode 100644
index 0000000..4b95fdc
--- /dev/null
+++ b/docs/clang_fortify_anatomy.md
@@ -0,0 +1,841 @@
+*This document was originally written for a broad audience, and it was*
+*determined that it'd be good to hold in Bionic's docs, too. Due to the*
+*ever-changing nature of code, it tries to link to a stable tag of*
+*Bionic's libc, rather than the live code in Bionic. Same for Clang.*
+*Reader beware. :)*
+
+# The Anatomy of Clang FORTIFY
+
+## Objective
+
+The intent of this document is to run through the minutiae of how Clang FORTIFY
+actually works in Bionic at the time of writing. Other FORTIFY implementations
+that target Clang should use very similar mechanics. This document exists in part
+because many Clang-specific features serve multiple purposes simultaneously, so
+getting up-to-speed on how things function can be quite difficult.
+
+## Background
+
+FORTIFY is a broad suite of extensions to libc aimed at catching misuses of
+common library functions. Textually, these extensions exist purely in libc, but
+all implementations of FORTIFY rely heavily on C language extensions in order
+to function at all.
+
+Broadly, FORTIFY implementations try to guard against many misuses of C
+standard(-ish) libraries:
+- Buffer overruns in functions where pointers+sizes are passed (e.g., `memcpy`,
+ `poll`), or where sizes exist implicitly (e.g., `strcpy`).
+- Arguments with incorrect values passed to libc functions (e.g.,
+ out-of-bounds bits in `umask`).
+- Missing arguments to functions (e.g., `open()` with `O_CREAT`, but no mode
+ bits).
+
+FORTIFY is traditionally enabled by passing `-D_FORTIFY_SOURCE=N` to your
+compiler. `N==0` disables FORTIFY, whereas `N==1`, `N==2`, and `N==3` enable
+increasingly strict versions of it. In general, FORTIFY doesn't require user
+code changes; that said, some code patterns
+are [incompatible with stricter versions of FORTIFY checking]. This is largely
+because FORTIFY has significant flexibility in what it considers to be an
+"out-of-bounds" access.
+
+FORTIFY implementations use a mix of compiler diagnostics and runtime checks to
+flag and/or mitigate the impacts of the misuses mentioned above.
+
+Further, given FORTIFY's design, the effectiveness of FORTIFY is a function of
+-- among other things -- the optimization level you're compiling your code at.
+Many FORTIFY implementations are implicitly disabled when building with `-O0`,
+since FORTIFY's design for both Clang and GCC relies on optimizations in order
+to provide useful run-time checks. For the purpose of this document, all
+analysis of FORTIFY functions and commentary on builtins assume that code is
+being built with some optimization level > `-O0`.
+
+### A note on GCC
+
+This document talks specifically about Bionic's FORTIFY implementation targeted
+at Clang. While GCC also provides a set of language extensions necessary to
+implement FORTIFY, these tools are different from what Clang offers. This
+divergence is an artifact of Clang and GCC's differing architecture as
+compilers.
+
+Textually, quite a bit can be shared between a FORTIFY implementation for GCC
+and one for Clang (e.g., see [ChromeOS' Glibc patch]), but this kind of sharing
+requires things like macros that expand to unbalanced braces depending on your
+compiler:
+
+```c
+/*
+ * Highly simplified; if you're interested in FORTIFY's actual implementation,
+ * please see the patch linked above.
+ */
+#ifdef __clang__
+# define FORTIFY_PRECONDITIONS
+# define FORTIFY_FUNCTION_END
+#else
+# define FORTIFY_PRECONDITIONS {
+# define FORTIFY_FUNCTION_END }
+#endif
+
+/*
+ * FORTIFY_WARNING_ONLY_IF_SIZE_OF_BUF_LESS_THAN is not defined, due to its
+ * complexity and irrelevance. It turns into a compile-time warning if the
+ * compiler can determine `*buf` has fewer than `size` bytes available.
+ */
+
+char *getcwd(char *buf, size_t size)
+FORTIFY_PRECONDITIONS
+ FORTIFY_WARNING_ONLY_IF_SIZE_OF_BUF_LESS_THAN(buf, size, "`buf` is too smol.")
+{
+ // Actual shared function implementation goes here.
+}
+FORTIFY_FUNCTION_END
+```
+
+All talk of GCC-focused implementations and how to merge Clang and GCC
+implementations is out-of-scope for this doc, however.
+
+## The Life of a Clang FORTIFY Function
+
+As referenced in the Background section, FORTIFY performs many different checks
+for many functions. This section intends to go through real-world examples of
+FORTIFY functions in Bionic, breaking down how each part of these functions
+work, and how the pieces fit together to provide FORTIFY-like functionality.
+
+While FORTIFY implementations may differ between stdlibs, they broadly follow
+the same patterns when implementing their checks for Clang, and they try to
+make similar promises with respect to FORTIFY compiling to be zero-overhead in
+some cases, etc. Moreover, while this document specifically examines Bionic,
+many stdlibs will operate _very similarly_ to Bionic in their Clang FORTIFY
+implementations.
+
+**In general, when reading the below, be prepared for exceptions, subtlety, and
+corner cases. The individual function breakdowns below try to not offer
+redundant information. Each one focuses on different aspects of FORTIFY.**
+
+### Terminology
+
+Because FORTIFY should be mostly transparent to developers, there are inherent
+naming collisions here: `memcpy(x, y, z)` turns into fundamentally different
+generated code depending on the value of `_FORTIFY_SOURCE`. Further, said
+`memcpy` call with `_FORTIFY_SOURCE` enabled needs to be able to refer to the
+`memcpy` that would have been called, had `_FORTIFY_SOURCE` been disabled.
+Hence, the following convention is followed in the subsections below for all
+prose (namely, multiline code blocks are exempted from this):
+
+- Standard library function names preceded by `__builtin_` refer to the use of
+ the function with `_FORTIFY_SOURCE` disabled.
+- Standard library function names without a prefix refer to the use of the
+ function with `_FORTIFY_SOURCE` enabled.
+
+This convention also applies in `clang`. `__builtin_memcpy` will always call
+`memcpy` as though `_FORTIFY_SOURCE` were disabled.
+
+## Breakdown of `mempcpy`
+
+The [FORTIFY'ed version of `mempcpy`] is a full, featureful example of a
+FORTIFY'ed function from Bionic. From the user's perspective, it supports a few
+things:
+- Producing a compile-time error if the number of bytes to copy trivially
+ exceeds the number of bytes available at the destination pointer.
+- If the `mempcpy` has the potential to write to more bytes than what is
+ available at the destination, a run-time check is inserted to crash the
+ program if more bytes are written than what is allowed.
+- Compiling away to be zero overhead when none of the buffer sizes can be
+ determined at compile-time[^1].
+
+The declaration in Bionic's headers for `__builtin_mempcpy` is:
+```c
+void* mempcpy(void* __dst, const void* __src, size_t __n) __INTRODUCED_IN(23);
+```
+
+Which is annotated with nothing special, except for Bionic's versioner, which
+is Android-specific (and orthogonal to FORTIFY anyway), so it will be ignored.
+
+The [source for `mempcpy`] in Bionic's headers for is:
+```c
+__BIONIC_FORTIFY_INLINE
+void* mempcpy(void* const dst __pass_object_size0, const void* src, size_t copy_amount)
+ __overloadable
+ __clang_error_if(__bos_unevaluated_lt(__bos0(dst), copy_amount),
+ "'mempcpy' called with size bigger than buffer") {
+#if __BIONIC_FORTIFY_RUNTIME_CHECKS_ENABLED
+ size_t bos_dst = __bos0(dst);
+ if (!__bos_trivially_ge(bos_dst, copy_amount)) {
+ return __builtin___mempcpy_chk(dst, src, copy_amount, bos_dst);
+ }
+#endif
+ return __builtin_mempcpy(dst, src, copy_amount);
+}
+```
+
+Expanding some of the important macros here, this function expands to roughly:
+```c
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+void* mempcpy(
+ void* const dst __attribute__((pass_object_size(0))),
+ const void* src,
+ size_t copy_amount)
+ __attribute__((overloadable))
+ __attribute__((diagnose_if(
+ __builtin_object_size(dst, 0) != -1 && __builtin_object_size(dst, 0) <= copy_amount),
+ "'mempcpy' called with size bigger than buffer"))) {
+#if __BIONIC_FORTIFY_RUNTIME_CHECKS_ENABLED
+ size_t bos_dst = __builtin_object_size(dst, 0);
+ if (!(__bos_trivially_ge(bos_dst, copy_amount))) {
+ return __builtin___mempcpy_chk(dst, src, copy_amount, bos_dst);
+ }
+#endif
+ return __builtin_mempcpy(dst, src, copy_amount);
+}
+```
+
+So let's walk through this step by step, to see how FORTIFY does what it says on
+the tin here.
+
+[^1]: "Zero overhead" in a way [similar to C++11's `std::unique_ptr`]: this will
+turn into a direct call `__builtin_mempcpy` (or an optimized form thereof) with
+no other surrounding checks at runtime. However, the additional complexity may
+hinder optimizations that are performed before the optimizer can prove that the
+`if (...) { ... }` can be optimized out. Depending on how late this happens,
+the additional complexity may skew inlining costs, hide opportunities for e.g.,
+`memcpy` coalescing, etc etc.
+
+### How does Clang select `mempcpy`?
+
+First, it's critical to notice that `mempcpy` is marked `overloadable`. This
+function is a `static inline __attribute__((always_inline))` overload of
+`__builtin_mempcpy`:
+- `__attribute__((overloadable))` allows us to perform overloading in C.
+- `__attribute__((overloadable))` mangles all calls to functions marked with
+ `__attribute__((overloadable))`.
+- `__attribute__((overloadable))` allows exactly one function signature with a
+ given name to not be marked with `__attribute__((overloadable))`. Calls to
+ this overload will not be mangled.
+
+Second, one might note that this `mempcpy` implementation has the same C-level
+signature as `__builtin_mempcpy`. `pass_object_size` is a Clang attribute that
+is generally needed by FORTIFY, but it carries the side-effect that functions
+may be overloaded simply on the presence (or lack of presence) of
+`pass_object_size` attributes. Given two overloads of a function that only
+differ on the presence of `pass_object_size` attributes, the candidate with
+`pass_object_size` attributes is preferred.
+
+Finally, the prior paragraph gets thrown out if one tries to take the address of
+`mempcpy`. It is impossible to take the address of a function with one or more
+parameters that are annotated with `pass_object_size`. Hence,
+`&__builtin_mempcpy == &mempcpy`. Further, because this is an issue of overload
+resolution, `(&mempcpy)(x, y, z);` is functionally identical to
+`__builtin_mempcpy(x, y, z);`.
+
+All of this accomplishes the following:
+- Direct calls to `mempcpy` should call the FORTIFY-protected `mempcpy`.
+- Indirect calls to `&mempcpy` should call `__builtin_mempcpy`.
+
+### How does Clang offer compile-time diagnostics?
+
+Once one is convinced that the FORTIFY-enabled overload of `mempcpy` will be
+selected for direct calls, Clang's `diagnose_if` and `__builtin_object_size` do
+all of the work from there.
+
+Subtleties here primarily fall out of the discussion in the above section about
+`&__builtin_mempcpy == &mempcpy`:
+```c
+#define _FORTIFY_SOURCE 2
+#include <string.h>
+void example_code() {
+ char buf[4]; // ...Assume sizeof(char) == 1.
+ const char input_buf[] = "Hello, World";
+ mempcpy(buf, input_buf, 4); // Valid, no diagnostic issued.
+
+ mempcpy(buf, input_buf, 5); // Emits a compile-time error since sizeof(buf) < 5.
+ __builtin_mempcpy(buf, input_buf, 5); // No compile-time error.
+ (&mempcpy)(buf, input_buf, 5); // No compile-time error, since __builtin_mempcpy is selected.
+}
+```
+
+Otherwise, the rest of this subsection is dedicated to preliminary discussion
+about `__builtin_object_size`.
+
+Clang's frontend can do one of two things with `__builtin_object_size(p, n)`:
+- Evaluate it as a constant.
+ - This can either mean declaring that the number of bytes at `p` is definitely
+ impossible to know, so the default value is used, or the number of bytes at
+ `p` can be known without optimizations.
+- Declare that the expression cannot form a constant, and lower it to
+ `@llvm.objectsize`, which is discussed in depth later.
+
+In the examples above, since `diagnose_if` is evaluated with context from the
+caller, Clang should be able to trivially determine that `buf` refers to a
+`char` array with 4 elements.
+
+The primary consequence of the above is that diagnostics can only be emitted if
+no optimizations are required to detect a broken code pattern. To be specific,
+clang's constexpr evaluator must be able to determine the logical object that
+any given pointer points to in order to fold `__builtin_object_size` to a
+constant, non-default answer:
+
+```c
+#define _FORTIFY_SOURCE 2
+#include <string.h>
+void example_code() {
+ char buf[4]; // ...Assume sizeof(char) == 1.
+ const char input_buf[] = "Hello, World";
+ mempcpy(buf, input_buf, 4); // Valid, no diagnostic issued.
+ mempcpy(buf, input_buf, 5); // Emits a compile-time error since sizeof(buf) < 5.
+ char *buf_ptr = buf;
+ mempcpy(buf_ptr, input_buf, 5); // No compile-time error; `buf_ptr`'s target can't be determined.
+}
+```
+
+### How does Clang insert run-time checks?
+
+This section expands on the following statement: FORTIFY has zero runtime cost
+in instances where there is no chance of catching a bug at run-time. Otherwise,
+it introduces a tiny additional run-time cost to ensure that functions aren't
+misused.
+
+In prior sections, the following was established:
+- `overloadable` and `pass_object_size` prompt Clang to always select this
+ overload of `mempcpy` over `__builtin_mempcpy` for direct calls.
+- If a call to `mempcpy` was trivially broken, Clang would produce a
+ compile-time error, rather than producing a binary.
+
+Hence, the case we're interested in here is one where Clang's frontend selected
+a FORTIFY'ed function's implementation for a function call, but was unable to
+find anything seriously wrong with said function call. Since the frontend is
+powerless to detect bugs at this point, our focus shifts to the mechanisms that
+LLVM uses to support FORTIFY.
+
+Going back to Bionic's `mempcpy` implementation, we have the following (ignoring
+diagnose_if and assuming run-time checks are enabled):
+```c
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+void* mempcpy(
+ void* const dst __attribute__((pass_object_size(0))),
+ const void* src,
+ size_t copy_amount)
+ __attribute__((overloadable)) {
+ size_t bos_dst = __builtin_object_size(dst, 0);
+ if (bos_dst != -1 &&
+ !(__builtin_constant_p(copy_amount) && bos_dst >= copy_amount)) {
+ return __builtin___mempcpy_chk(dst, src, copy_amount, bos_dst);
+ }
+ return __builtin_mempcpy(dst, src, copy_amount);
+}
+```
+
+In other words, we have a `static`, `always_inline` function which:
+- If `__builtin_object_size(dst, 0)` cannot be determined (in which case, it
+ returns -1), calls `__builtin_mempcpy`.
+- Otherwise, if `copy_amount` can be folded to a constant, and if
+ `__builtin_object_size(dst, 0) >= copy_amount`, calls `__builtin_mempcpy`.
+- Otherwise, calls `__builtin___mempcpy_chk`.
+
+
+How can this be "zero overhead"? Let's focus on the following part of the
+function:
+
+```c
+ size_t bos_dst = __builtin_object_size(dst, 0);
+ if (bos_dst != -1 &&
+ !(__builtin_constant_p(copy_amount) && bos_dst >= copy_amount)) {
+```
+
+If Clang's frontend cannot determine a value for `__builtin_object_size`, Clang
+lowers it to LLVM's `@llvm.objectsize` intrinsic. The `@llvm.objectsize`
+invocation corresponding to `__builtin_object_size(p, 0)` is guaranteed to
+always fold to a constant value by the time LLVM emits machine code.
+
+Hence, `bos_dst` is guaranteed to be a constant; if it's -1, the above branch
+can be eliminated entirely, since it folds to `if (false && ...)`. Further, the
+RHS of the `&&` in this branch has us call `__builtin_mempcpy` if `copy_amount`
+is a known value less than `bos_dst` (yet another constant value). Therefore,
+the entire condition is always knowable when LLVM is done with LLVM IR-level
+optimizations, so no condition is ever emitted to machine code in practice.
+
+#### Why is "zero overhead" in quotes? Why is `unique_ptr` relevant?
+
+`__builtin_object_size` and `__builtin_constant_p` are forced to be constants
+after most optimizations take place. Until LLVM replaces both of these with
+constants and optimizes them out, we have additional branches and function calls
+in our IR. This can have negative effects, such as distorting inlining costs and
+inhibiting optimizations that are conservative around branches in control-flow.
+
+So FORTIFY is free in these cases _in isolation of any of the code around it_.
+Due to its implementation, it may impact the optimizations that occur on code
+around the literal call to the FORTIFY-hardened libc function.
+
+`unique_ptr` was just the first thing that came to the author's mind for "the
+type should be zero cost with any level of optimization enabled, but edge-cases
+might make it only-mostly-free to use."
+
+### How is checking actually performed?
+
+In cases where checking can be performed (e.g., where we call
+`__builtin___mempcpy_chk(dst, src, copy_amount, bos_dst);`), Bionic provides [an
+implementation for `__mempcpy_chk`]. This is:
+
+```c
+extern "C" void* __mempcpy_chk(void* dst, const void* src, size_t count, size_t dst_len) {
+ __check_count("mempcpy", "count", count);
+ __check_buffer_access("mempcpy", "write into", count, dst_len);
+ return mempcpy(dst, src, count);
+}
+```
+This function itself boils down to a few small branches which abort the program
+if they fail, and a direct call to `__builtin_mempcpy`.
+
+### Wrapping up
+
+In the above breakdown, it was shown how Clang and Bionic work together to:
+- represent FORTIFY-hardened overloads of functions,
+- report misuses of stdlib functions at compile-time, and
+- insert run-time checks for uses of functions that might be incorrect, but only
+ if we have the potential of proving the incorrectness of these.
+
+## Breakdown of open
+
+In Bionic, the [FORTIFY'ed implementation of `open`] is quite large. Much like
+`mempcpy`, the `__builtin_open` declaration is simple:
+
+```c
+int open(const char* __path, int __flags, ...);
+```
+
+With some macros expanded, the FORTIFY-hardened header implementation is:
+```c
+int __open_2(const char*, int);
+int __open_real(const char*, int, ...) __asm__(open);
+
+#define __open_modes_useful(flags) (((flags) & O_CREAT) || ((flags) & O_TMPFILE) == O_TMPFILE)
+
+static
+int open(const char* pathname, int flags, mode_t modes, ...) __overloadable
+ __attribute__((diagnose_if(1, "error", "too many arguments")));
+
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+int open(const char* const __attribute__((pass_object_size(1))) pathname, int flags)
+ __attribute__((overloadable))
+ __attribute__((diagnose_if(
+ __open_modes_useful(flags),
+ "error",
+ "'open' called with O_CREAT or O_TMPFILE, but missing mode"))) {
+#if __ANDROID_API__ >= 17 && __BIONIC_FORTIFY_RUNTIME_CHECKS_ENABLED
+ return __open_2(pathname, flags);
+#else
+ return __open_real(pathname, flags);
+#endif
+}
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+int open(const char* const __attribute__((pass_object_size(1))) pathname, int flags, mode_t modes)
+ __attribute__((overloadable))
+ __clang_warning_if(!__open_modes_useful(flags) && modes,
+ "'open' has superfluous mode bits; missing O_CREAT?") {
+ return __open_real(pathname, flags, modes);
+}
+```
+
+Which may be a lot to take in.
+
+Before diving too deeply, please note that the remainder of these subsections
+assume that the programmer didn't make any egregious typos. Moreover, there's no
+real way that Bionic tries to prevent calls to `open` like
+`open("foo", 0, "how do you convert a const char[N] to mode_t?");`. The only
+real C-compatible solution the author can think of is "stamp out many overloads
+to catch sort-of-common instances of this very uncommon typo." This isn't great.
+
+More directly, no effort is made below to recognize calls that, due to
+incompatible argument types, cannot go to any `open` implementation other than
+`__builtin_open`, since it's recognized right here. :)
+
+### Implementation breakdown
+
+This `open` implementation does a few things:
+- Turns calls to `open` with too many arguments into a compile-time error.
+- Diagnoses calls to `open` with missing modes at compile-time and run-time
+ (both cases turn into errors).
+- Emits warnings on calls to `open` with useless mode bits, unless the mode bits
+ are all 0.
+
+One common bit of code not explained below is the `__open_real` declaration above:
+```c
+int __open_real(const char*, int, ...) __asm__(open);
+```
+
+This exists as a way for us to call `__builtin_open` without needing clang to
+have a pre-defined `__builtin_open` function.
+
+#### Compile-time error on too many arguments
+
+```c
+static
+int open(const char* pathname, int flags, mode_t modes, ...) __overloadable
+ __attribute__((diagnose_if(1, "error", "too many arguments")));
+```
+
+Which matches most calls to open that supply too many arguments, since
+`int(const char *, int, ...)` matches less strongly than
+`int(const char *, int, mode_t, ...)` for calls where the 3rd arg can be
+converted to `mode_t` without too much effort. Because of the `diagnose_if`
+attribute, all of these calls turn into compile-time errors.
+
+#### Compile-time or run-time error on missing arguments
+The following overload handles all two-argument calls to `open`.
+```c
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+int open(const char* const __attribute__((pass_object_size(1))) pathname, int flags)
+ __attribute__((overloadable))
+ __attribute__((diagnose_if(
+ __open_modes_useful(flags),
+ "error",
+ "'open' called with O_CREAT or O_TMPFILE, but missing mode"))) {
+#if __ANDROID_API__ >= 17 && __BIONIC_FORTIFY_RUNTIME_CHECKS_ENABLED
+ return __open_2(pathname, flags);
+#else
+ return __open_real(pathname, flags);
+#endif
+}
+```
+
+Like `mempcpy`, `diagnose_if` handles emitting a compile-time error if the call
+to `open` is broken in a way that's visible to Clang's frontend. This
+essentially boils down to "`open` is being called with a `flags` value that
+requires mode bits to be set."
+
+If that fails to catch a bug, we [unconditionally call `__open_2`], which
+performs a run-time check:
+```c
+int __open_2(const char* pathname, int flags) {
+ if (needs_mode(flags)) __fortify_fatal("open: called with O_CREAT/O_TMPFILE but no mode");
+ return FDTRACK_CREATE_NAME("open", __openat(AT_FDCWD, pathname, force_O_LARGEFILE(flags), 0));
+}
+```
+
+#### Compile-time warning if modes are pointless
+
+Finally, we have the following `open` call:
+```c
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+int open(const char* const __attribute__((pass_object_size(1))) pathname, int flags, mode_t modes)
+ __attribute__((overloadable))
+ __clang_warning_if(!__open_modes_useful(flags) && modes,
+ "'open' has superfluous mode bits; missing O_CREAT?") {
+ return __open_real(pathname, flags, modes);
+}
+```
+
+This simply issues a warning if Clang's frontend can determine that `flags`
+isn't necessary. Due to conventions in existing code, a `modes` value of `0` is
+not diagnosed.
+
+#### What about `&open`?
+One yet-unaddressed aspect of the above is how `&open` works. This is thankfully
+a short answer:
+- It happens that `open` takes a parameter of type `const char*`.
+- It happens that `pass_object_size` -- an attribute only applicable to
+ parameters of type `T*` -- makes it impossible to take the address of a
+ function.
+
+Since clang doesn't support a "this function should never have its address
+taken," attribute, Bionic uses the next best thing: `pass_object_size`. :)
+
+## Breakdown of poll
+
+(Preemptively: at the time of writing, Clang has no literal `__builtin_poll`
+builtin. `__builtin_poll` is referenced below to remain consistent with the
+convention established in the Terminology section.)
+
+Bionic's `poll` implementation is closest to `mempcpy` above, though it has a
+few interesting aspects worth examining.
+
+The [full header implementation of `poll`] is, with some macros expanded:
+```c
+#define __bos_fd_count_trivially_safe(bos_val, fds, fd_count) \
+ ((bos_val) == -1) || \
+ (__builtin_constant_p(fd_count) && \
+ (bos_val) >= sizeof(*fds) * (fd_count)))
+
+static
+__inline__
+__attribute__((no_stack_protector))
+__attribute__((always_inline))
+int poll(struct pollfd* const fds __attribute__((pass_object_size(1))), nfds_t fd_count, int timeout)
+ __attribute__((overloadable))
+ __attriubte__((diagnose_if(
+ __builtin_object_size(fds, 1) != -1 && __builtin_object_size(fds, 1) < sizeof(*fds) * fd_count,
+ "error",
+ "in call to 'poll', fd_count is larger than the given buffer"))) {
+ size_t bos_fds = __builtin_object_size(fds, 1);
+ if (!__bos_fd_count_trivially_safe(bos_fds, fds, fd_count)) {
+ return __poll_chk(fds, fd_count, timeout, bos_fds);
+ }
+ return (&poll)(fds, fd_count, timeout);
+}
+```
+
+To get the commonality with `mempcpy` and `open` out of the way:
+- This function is an overload with `__builtin_poll`.
+- The signature is the same, modulo the presence of a `pass_object_size`
+ attribute. Hence, for direct calls, overload resolution will always prefer it
+ over `__builtin_poll`. Taking the address of `poll` is forbidden, so all
+ references to `&poll` actually reference `__builtin_poll`.
+- When `fds` is too small to hold `fd_count` `pollfd`s, Clang will emit a
+ compile-time error if possible using `diagnose_if`.
+- If this can't be observed until run-time, `__poll_chk` verifies this.
+- When `fds` is a constant according to `__builtin_constant_p`, this always
+ compiles into `__poll_chk` for always-broken calls to `poll`, or
+ `__builtin_poll` for always-safe calls to `poll`.
+
+The critical bits to highlight here are on this line:
+```c
+int poll(struct pollfd* const fds __attribute__((pass_object_size(1))), nfds_t fd_count, int timeout)
+```
+
+And this line:
+```c
+ return (&poll)(fds, fd_count, timeout);
+```
+
+Starting with the simplest, we call `__builtin_poll` with `(&poll)(...);`. As
+referenced above, taking the address of an overloaded function where all but one
+overload has a `pass_object_size` attribute on one or more parameters always
+resolves to the function without any `pass_object_size` attributes.
+
+The other line deserves a section. The subtlety of it is almost entirely in the
+use of `pass_object_size(1)` instead of `pass_object_size(0)`. on the `fds`
+parameter, and the corresponding use of `__builtin_object_size(fds, 1);` in the
+body of `poll`.
+
+### Subtleties of __builtin_object_size(p, N)
+
+Earlier in this document, it was said that a full description of each
+attribute/builtin necessary to power FORTIFY was out of scope. This is... only
+somewhat the case when we talk about `__builtin_object_size` and
+`pass_object_size`, especially when their second argument is `1`.
+
+#### tl;dr
+`__builtin_object_size(p, N)` and `pass_object_size(N)`, where `(N & 1) == 1`,
+can only be accurately determined by Clang. LLVM's `@llvm.objectsize` intrinsic
+ignores the value of `N & 1`, since handling `(N & 1) == 1` accurately requires
+data that's currently entirely inaccessible to LLVM, and that is difficult to
+preserve through LLVM's optimization passes.
+
+`pass_object_size`'s "lifting" of the evaluation of
+`__builtin_object_size(p, N)` to the caller is critical, since it allows Clang
+full visibility into the expression passed to e.g., `poll(&foo->bar, baz, qux)`.
+It's not a perfect solution, but it allows `N == 1` to be fully accurate in at
+least some cases.
+
+#### Background
+Clang's implementation of `__builtin_object_size` aims to be compatible with
+GCC's, which has [a decent bit of documentation]. Put simply,
+`__builtin_object_size(p, N)` is intended to evaluate at compile-time how many
+bytes can be accessed after `p` in a well-defined way. Straightforward examples
+of this are:
+```c
+char buf[8];
+assert(__builtin_object_size(buf, N) == 8);
+assert(__builtin_object_size(buf + 1, N) == 7);
+```
+
+This should hold for all values of N that are valid to pass to
+`__builtin_object_size`. The `N` value of `__builtin_object_size` is a mask of
+settings.
+
+##### (N & 2) == ?
+
+This is mostly for completeness sake; in Bionic's FORTIFY implementation, N is
+always either 0 or 1.
+
+If there are multiple possible values of `p` in a call to
+`__builtin_object_size(p, N)`, the second bit in `N` determines the behavior of
+the compiler. If `(N & 2) == 0`, `__builtin_object_size` should return the
+greatest possible size for each possible value of `p`. Otherwise, it should
+return the least possible value. For example:
+
+```c
+char smol_buf[7];
+char buf[8];
+char *p = rand() ? smol_buf : buf;
+assert(__builtin_object_size(p, 0) == 8);
+assert(__builtin_object_size(p, 2) == 7);
+```
+
+##### (N & 1) == 0
+
+`__builtin_object_size(p, 0)` is more or less as simple as the example in the
+Background section directly above. When Clang attempts to evaluate
+`__builtin_object_size(p, 0);` and when LLVM tries to determine the result of a
+corresponding `@llvm.objectsize` call to, they search for the storage underlying
+the pointer in question. If that can be determined, Clang or LLVM can provide an
+answer; otherwise, they cannot.
+
+##### (N & 1) == 1, and the true magic of pass_object_size
+
+`__builtin_object_size(p, 1)` has a less uniform implementation between LLVM and
+Clang. According to GCC's documentation, "If the least significant bit [of
+__builtin_object_size's second argument] is clear, objects are whole variables,
+if it is set, a closest surrounding subobject is considered the object a pointer
+points to."
+
+The "closest surrounding subobject," means that `(N & 1) == 1` depends on type
+information in order to operate in many cases. Consider the following examples:
+```c
+struct Foo {
+ int a;
+ int b;
+};
+
+struct Foo foo;
+assert(__builtin_object_size(&foo, 0) == sizeof(foo));
+assert(__builtin_object_size(&foo, 1) == sizeof(foo));
+assert(__builtin_object_size(&foo->a, 0) == sizeof(foo));
+assert(__builtin_object_size(&foo->a, 1) == sizeof(int));
+
+struct Foo foos[2];
+assert(__builtin_object_size(&foos[0], 0) == 2 * sizeof(foo));
+assert(__builtin_object_size(&foos[0], 1) == sizeof(foo));
+assert(__builtin_object_size(&foos[0]->a, 0) == 2 * sizeof(foo));
+assert(__builtin_object_size(&foos[0]->a, 1) == sizeof(int));
+```
+
+...And perhaps somewhat surprisingly:
+```c
+void example(struct Foo *foo) {
+ // (As a reminder, `-1` is "I don't know" when `(N & 2) == 0`.)
+ assert(__builtin_object_size(foo, 0) == -1);
+ assert(__builtin_object_size(foo, 1) == -1);
+ assert(__builtin_object_size(foo->a, 0) == -1);
+ assert(__builtin_object_size(foo->a, 1) == sizeof(int));
+}
+```
+
+In Clang, [this type-aware requirement poses problems for us]: Clang's frontend
+knows everything we could possibly want about the types of variables, but
+optimizations are only performed by LLVM. LLVM has no reliable source for C or
+C++ data types, so calls to `__builtin_object_size(p, N)` that cannot be
+resolved by clang are lowered to the equivalent of
+`__builtin_object_size(p, N & ~1)` in LLVM IR.
+
+Moreover, Clang's frontend is the best-equipped part of the compiler to
+accurately determine the answer for `__builtin_object_size(p, N)`, given we know
+what `p` is. LLVM is the best-equipped part of the compiler to determine the
+value of `p`. This ordering issue is unfortunate.
+
+This is where `pass_object_size(N)` comes in. To summarize [the docs for
+`pass_object_size`], it evaluates `__builtin_object_size(p, N)` within the
+context of the caller of the function annotated with `pass_object_size`, and
+passes the value of that into the callee as an invisible parameter. All calls to
+`__builtin_object_size(parameter, N)` are substituted with references to this
+invisible parameter.
+
+Putting this plainly, Clang's frontend struggles to evaluate the following:
+```c
+int foo(void *p) {
+ return __builtin_object_size(p, 1);
+}
+
+void bar() {
+ struct { int i, j } k;
+ // The frontend can't figure this interprocedural objectsize out, so it gets lowered to
+ // LLVM, which determines that the answer here is sizeof(k).
+ int baz = foo(&k.i);
+}
+```
+
+However, with the magic of `pass_object_size`, we get one level of inlining to
+look through:
+```c
+int foo(void *const __attribute__((pass_object_size(1))) p) {
+ return __builtin_object_size(p, 1);
+}
+
+void bar() {
+ struct { int i, j } k;
+ // Due to pass_object_size, this is equivalent to:
+ // int baz = foo(&k.i, __builtin_object_size(&k.i, 1));
+ // ...and `int foo(void *)` is actually equivalent to:
+ // int foo(void *const, size_t size) {
+ // return size;
+ // }
+ int baz = foo(&k.i);
+}
+```
+
+So we can obtain an accurate result in this case.
+
+##### What about pass_object_size(0)?
+It's sort of tangential, but if you find yourself wondering about the utility of
+`pass_object_size(0)` ... it's somewhat split. `pass_object_size(0)` in Bionic's
+FORTIFY exists mostly for visual consistency, simplicity, and as a useful way to
+have e.g., `&mempcpy` == `&__builtin_mempcpy`.
+
+Outside of these fringe benefits, all of the functions with
+`pass_object_size(0)` on parameters are marked with `always_inline`, so
+"lifting" the `__builtin_object_size` call isn't ultimately very helpful. In
+theory, users can always have something like:
+
+```c
+// In some_header.h
+// This function does cool and interesting things with the `__builtin_object_size` of its parameter,
+// and is able to work with that as though the function were defined inline.
+void out_of_line_function(void *__attribute__((pass_object_size(0))));
+```
+
+Though the author isn't aware of uses like this in practice, beyond a few folks
+on LLVM's mailing list seeming interested in trying it someday.
+
+#### Wrapping up
+In the (long) section above, two things were covered:
+- The use of `(&poll)(...);` is a convenient shorthand for calling
+ `__builtin_poll`.
+- `__builtin_object_size(p, N)` with `(N & 1) == 1` is not easy for Clang to
+ answer accurately, since it relies on type info only available in the
+ frontend, and it sometimes relies on optimizations only available in the
+ middle-end. `pass_object_size` helps mitigate this.
+
+## Miscellaneous Notes
+The above should be a roughly comprehensive view of how FORTIFY works in the
+real world. The main thing it fails to mention is the use of [the `diagnose_as_builtin` attribute] in Clang.
+
+As time has moved on, Clang has increasingly gained support for emitting
+warnings that were previously emitted by FORTIFY machinery.
+`diagnose_as_builtin` allows us to remove the `diagnose_if`s from some of the
+`static inline` overloads of stdlib functions above, so Clang may diagnose them
+instead.
+
+Clang's built-in diagnostics are often better than `diagnose_if` diagnostics,
+since Clang can format its diagnostics to include e.g., information about the
+sizes of buffers in a suspect call to a function. `diagnose_if` can only have
+the compiler output constant strings.
+
+[ChromeOS' Glibc patch]: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/90fa9b27731db10a6010c7f7c25b24028145b091/sys-libs/glibc/files/local/glibc-2.33/0007-glibc-add-clang-style-FORTIFY.patch
+[FORTIFY'ed implementation of `open`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/include/bits/fortify/fcntl.h#41
+[FORTIFY'ed version of `mempcpy`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/include/bits/fortify/string.h#45
+[a decent bit of documentation]: https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html
+[an implementation for `__mempcpy_chk`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/bionic/fortify.cpp#501
+[full header implementation of `poll`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/include/bits/fortify/poll.h#43
+[incompatible with stricter versions of FORTIFY checking]: https://godbolt.org/z/fGfEYxfnf
+[similar to C++11's `std::unique_ptr`]: https://stackoverflow.com/questions/58339165/why-can-a-t-be-passed-in-register-but-a-unique-ptrt-cannot
+[source for `mempcpy`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/include/string.h#55
+[the `diagnose_as_builtin` attribute]: https://releases.llvm.org/14.0.0/tools/clang/docs/AttributeReference.html#diagnose-as-builtin
+[the docs for `pass_object_size`]: https://releases.llvm.org/14.0.0/tools/clang/docs/AttributeReference.html#pass-object-size-pass-dynamic-object-size
+[this type-aware requirement poses problems for us]: https://github.com/llvm/llvm-project/issues/55742
+[unconditionally call `__open_2`]: https://android.googlesource.com/platform/bionic/+/refs/heads/android12-release/libc/bionic/open.cpp#70
diff --git a/libc/SYSCALLS.TXT b/libc/SYSCALLS.TXT
index 45b2a1b..9aeb07c 100644
--- a/libc/SYSCALLS.TXT
+++ b/libc/SYSCALLS.TXT
@@ -26,7 +26,7 @@
# This file is processed by a python script named gensyscalls.py, run via
# genrules in Android.bp.
-int execve(const char*, char* const*, char* const*) all
+int __execve:execve(const char*, char* const*, char* const*) all
uid_t getuid:getuid32() lp32
uid_t getuid:getuid() lp64
@@ -320,7 +320,7 @@
int __eventfd:eventfd2(unsigned int, int) all
-void _exit|_Exit:exit_group(int) all
+void __exit_group:exit_group(int) all
void __exit:exit(int) all
int inotify_init1(int) all
diff --git a/libc/arch-arm64/bionic/setjmp.S b/libc/arch-arm64/bionic/setjmp.S
index d2fafdb..8e00b56 100644
--- a/libc/arch-arm64/bionic/setjmp.S
+++ b/libc/arch-arm64/bionic/setjmp.S
@@ -194,7 +194,7 @@
cmp x2, x12
bne __bionic_setjmp_checksum_mismatch
-#if __has_feature(hwaddress_sanitizer)
+ // Update stack memory tags (MTE + hwasan).
stp x0, x30, [sp, #-16]!
.cfi_adjust_cfa_offset 16
.cfi_rel_offset x0, 0
@@ -206,7 +206,7 @@
bic x2, x2, #1
ldr x0, [x0, #(_JB_X30_SP * 8 + 8)]
eor x0, x0, x2
- bl __hwasan_handle_longjmp
+ bl memtag_handle_longjmp
mov x1, x19 // Restore 'value'.
// Restore original x0 and lr.
@@ -214,7 +214,6 @@
.cfi_adjust_cfa_offset -16
.cfi_restore x0
.cfi_restore x30
-#endif
// Do we need to restore the signal mask?
ldr x2, [x0, #(_JB_SIGFLAG * 8)]
diff --git a/libc/arch-arm64/bionic/vfork.S b/libc/arch-arm64/bionic/vfork.S
index df7b063..9878e8d 100644
--- a/libc/arch-arm64/bionic/vfork.S
+++ b/libc/arch-arm64/bionic/vfork.S
@@ -45,6 +45,9 @@
ldr w10, [x9, #20]
str w0, [x9, #20]
+ // Clear vfork_child_stack_bottom_.
+ str xzr, [x9, #776]
+
mov x0, #(CLONE_VM | CLONE_VFORK | SIGCHLD)
mov x1, xzr
mov x2, xzr
@@ -62,9 +65,6 @@
cneg x0, x0, hi
b.hi __set_errno_internal
-#if __has_feature(hwaddress_sanitizer)
- cbz x0, .L_exit
-
// Clean up stack shadow in the parent process.
// https://github.com/google/sanitizers/issues/925
paciasp
@@ -75,7 +75,7 @@
.cfi_rel_offset x30, 8
add x0, sp, #16
- bl __hwasan_handle_vfork
+ bl memtag_handle_vfork
ldp x0, x30, [sp], #16
.cfi_adjust_cfa_offset -16
@@ -84,8 +84,6 @@
autiasp
.cfi_negate_ra_state
-#endif
-
.L_exit:
ret
END(vfork)
diff --git a/libc/bionic/exec.cpp b/libc/bionic/exec.cpp
index fd2c401..40612e7 100644
--- a/libc/bionic/exec.cpp
+++ b/libc/bionic/exec.cpp
@@ -39,10 +39,12 @@
#include <string.h>
#include <unistd.h>
-#include "private/__bionic_get_shell_path.h"
#include "private/FdPath.h"
+#include "private/__bionic_get_shell_path.h"
+#include "pthread_internal.h"
extern "C" char** environ;
+extern "C" int __execve(const char* pathname, char* const* argv, char* const* envp);
enum { ExecL, ExecLE, ExecLP };
@@ -181,3 +183,9 @@
if (errno == ENOENT) errno = EBADF;
return -1;
}
+
+__attribute__((no_sanitize("memtag"))) int execve(const char* pathname, char* const* argv,
+ char* const* envp) {
+ __get_thread()->vfork_child_stack_bottom = __builtin_frame_address(0);
+ return __execve(pathname, argv, envp);
+}
diff --git a/libc/bionic/exit.cpp b/libc/bionic/exit.cpp
index a5aed78..52fd193 100644
--- a/libc/bionic/exit.cpp
+++ b/libc/bionic/exit.cpp
@@ -30,9 +30,18 @@
#include <unistd.h>
#include "private/bionic_defs.h"
+#include "pthread_internal.h"
extern "C" void __cxa_finalize(void* dso_handle);
extern "C" void __cxa_thread_finalize();
+extern "C" __noreturn void __exit_group(int status);
+
+__attribute__((no_sanitize("memtag"))) void _exit(int status) {
+ __get_thread()->vfork_child_stack_bottom = __builtin_frame_address(0);
+ __exit_group(status);
+}
+
+__strong_alias(_Exit, _exit);
__BIONIC_WEAK_FOR_NATIVE_BRIDGE
void exit(int status) {
diff --git a/libc/bionic/heap_tagging.cpp b/libc/bionic/heap_tagging.cpp
index 9b67d85..78d21b0 100644
--- a/libc/bionic/heap_tagging.cpp
+++ b/libc/bionic/heap_tagging.cpp
@@ -32,6 +32,8 @@
#include <bionic/pthread_internal.h>
#include <platform/bionic/malloc.h>
+#include <sanitizer/hwasan_interface.h>
+#include <sys/auxv.h>
extern "C" void scudo_malloc_disable_memory_tagging();
extern "C" void scudo_malloc_set_track_allocation_stacks(int);
@@ -44,30 +46,37 @@
#if !__has_feature(hwaddress_sanitizer)
heap_tagging_level = __libc_shared_globals()->initial_heap_tagging_level;
#endif
- switch (heap_tagging_level) {
- case M_HEAP_TAGGING_LEVEL_TBI:
- __libc_globals.mutate([](libc_globals* globals) {
+
+ __libc_globals.mutate([](libc_globals* globals) {
+ switch (heap_tagging_level) {
+ case M_HEAP_TAGGING_LEVEL_TBI:
// Arrange for us to set pointer tags to POINTER_TAG, check tags on
// deallocation and untag when passing pointers to the allocator.
globals->heap_pointer_tag = (reinterpret_cast<uintptr_t>(POINTER_TAG) << TAG_SHIFT) |
(0xffull << CHECK_SHIFT) | (0xffull << UNTAG_SHIFT);
- });
-#if defined(USE_SCUDO)
- scudo_malloc_disable_memory_tagging();
-#endif // USE_SCUDO
- break;
-#if defined(USE_SCUDO)
- case M_HEAP_TAGGING_LEVEL_SYNC:
- scudo_malloc_set_track_allocation_stacks(1);
- break;
+ break;
+ case M_HEAP_TAGGING_LEVEL_SYNC:
+ case M_HEAP_TAGGING_LEVEL_ASYNC:
+ atomic_store(&globals->memtag_stack, __libc_shared_globals()->initial_memtag_stack);
+ break;
+ default:
+ break;
+ };
+ });
+#if defined(USE_SCUDO)
+ switch (heap_tagging_level) {
+ case M_HEAP_TAGGING_LEVEL_TBI:
case M_HEAP_TAGGING_LEVEL_NONE:
scudo_malloc_disable_memory_tagging();
break;
-#endif // USE_SCUDO
+ case M_HEAP_TAGGING_LEVEL_SYNC:
+ scudo_malloc_set_track_allocation_stacks(1);
+ break;
default:
break;
}
+#endif // USE_SCUDO
#endif // aarch64
}
@@ -99,16 +108,21 @@
switch (tag_level) {
case M_HEAP_TAGGING_LEVEL_NONE:
- if (heap_tagging_level == M_HEAP_TAGGING_LEVEL_TBI) {
- __libc_globals.mutate([](libc_globals* globals) {
+ __libc_globals.mutate([](libc_globals* globals) {
+ if (heap_tagging_level == M_HEAP_TAGGING_LEVEL_TBI) {
// Preserve the untag mask (we still want to untag pointers when passing them to the
// allocator), but clear the fixed tag and the check mask, so that pointers are no longer
// tagged and checks no longer happen.
globals->heap_pointer_tag = static_cast<uintptr_t>(0xffull << UNTAG_SHIFT);
- });
- } else if (!set_tcf_on_all_threads(PR_MTE_TCF_NONE)) {
- error_log("SetHeapTaggingLevel: set_tcf_on_all_threads failed");
- return false;
+ }
+ atomic_store(&globals->memtag_stack, false);
+ });
+
+ if (heap_tagging_level != M_HEAP_TAGGING_LEVEL_TBI) {
+ if (!set_tcf_on_all_threads(PR_MTE_TCF_NONE)) {
+ error_log("SetHeapTaggingLevel: set_tcf_on_all_threads failed");
+ return false;
+ }
}
#if defined(USE_SCUDO)
scudo_malloc_disable_memory_tagging();
@@ -158,3 +172,69 @@
return true;
}
+
+#ifdef __aarch64__
+static inline __attribute__((no_sanitize("memtag"))) void untag_memory(void* from, void* to) {
+ __asm__ __volatile__(
+ ".arch_extension mte\n"
+ "1:\n"
+ "stg %[Ptr], [%[Ptr]], #16\n"
+ "cmp %[Ptr], %[End]\n"
+ "b.lt 1b\n"
+ : [Ptr] "+&r"(from)
+ : [End] "r"(to)
+ : "memory");
+}
+#endif
+
+#ifdef __aarch64__
+// 128Mb of stack should be enough for anybody.
+static constexpr size_t kUntagLimit = 128 * 1024 * 1024;
+#endif // __aarch64__
+
+extern "C" __LIBC_HIDDEN__ __attribute__((no_sanitize("memtag"))) void memtag_handle_longjmp(
+ void* sp_dst __unused) {
+#ifdef __aarch64__
+ if (__libc_globals->memtag_stack) {
+ void* sp = __builtin_frame_address(0);
+ size_t distance = reinterpret_cast<uintptr_t>(sp_dst) - reinterpret_cast<uintptr_t>(sp);
+ if (distance > kUntagLimit) {
+ async_safe_fatal(
+ "memtag_handle_longjmp: stack adjustment too large! %p -> %p, distance %zx > %zx\n", sp,
+ sp_dst, distance, kUntagLimit);
+ } else {
+ untag_memory(sp, sp_dst);
+ }
+ }
+#endif // __aarch64__
+
+#if __has_feature(hwaddress_sanitizer)
+ __hwasan_handle_longjmp(sp_dst);
+#endif // __has_feature(hwaddress_sanitizer)
+}
+
+extern "C" __LIBC_HIDDEN__ __attribute__((no_sanitize("memtag"), no_sanitize("hwaddress"))) void
+memtag_handle_vfork(void* sp __unused) {
+#ifdef __aarch64__
+ if (__libc_globals->memtag_stack) {
+ void* child_sp = __get_thread()->vfork_child_stack_bottom;
+ __get_thread()->vfork_child_stack_bottom = nullptr;
+ if (child_sp) {
+ size_t distance = reinterpret_cast<uintptr_t>(sp) - reinterpret_cast<uintptr_t>(child_sp);
+ if (distance > kUntagLimit) {
+ async_safe_fatal(
+ "memtag_handle_vfork: stack adjustment too large! %p -> %p, distance %zx > %zx\n",
+ child_sp, sp, distance, kUntagLimit);
+ } else {
+ untag_memory(child_sp, sp);
+ }
+ } else {
+ async_safe_fatal("memtag_handle_vfork: child SP unknown\n");
+ }
+ }
+#endif // __aarch64__
+
+#if __has_feature(hwaddress_sanitizer)
+ __hwasan_handle_vfork(sp);
+#endif // __has_feature(hwaddress_sanitizer)
+}
diff --git a/libc/bionic/libc_init_static.cpp b/libc/bionic/libc_init_static.cpp
index 575da62..66aaeaa 100644
--- a/libc/bionic/libc_init_static.cpp
+++ b/libc/bionic/libc_init_static.cpp
@@ -259,17 +259,18 @@
// M_HEAP_TAGGING_LEVEL_NONE, if MTE isn't enabled for this process we enable
// M_HEAP_TAGGING_LEVEL_TBI.
static HeapTaggingLevel __get_heap_tagging_level(const void* phdr_start, size_t phdr_ct,
- uintptr_t load_bias) {
+ uintptr_t load_bias, bool* stack) {
+ unsigned note_val =
+ __get_memtag_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct, load_bias);
+ *stack = note_val & NT_MEMTAG_STACK;
+
HeapTaggingLevel level;
if (get_environment_memtag_setting(&level)) return level;
- unsigned note_val =
- __get_memtag_note(reinterpret_cast<const ElfW(Phdr)*>(phdr_start), phdr_ct, load_bias);
-
// Note, previously (in Android 12), any value outside of bits [0..3] resulted
// in a check-fail. In order to be permissive of further extensions, we
- // relaxed this restriction. For now, we still only support MTE heap.
- if (!(note_val & NT_MEMTAG_HEAP)) return M_HEAP_TAGGING_LEVEL_TBI;
+ // relaxed this restriction.
+ if (!(note_val & (NT_MEMTAG_HEAP | NT_MEMTAG_STACK))) return M_HEAP_TAGGING_LEVEL_TBI;
unsigned mode = note_val & NT_MEMTAG_LEVEL_MASK;
switch (mode) {
@@ -295,8 +296,10 @@
// This function is called from the linker before the main executable is relocated.
__attribute__((no_sanitize("hwaddress", "memtag"))) void __libc_init_mte(const void* phdr_start,
size_t phdr_ct,
- uintptr_t load_bias) {
- HeapTaggingLevel level = __get_heap_tagging_level(phdr_start, phdr_ct, load_bias);
+ uintptr_t load_bias,
+ void* stack_top) {
+ bool memtag_stack;
+ HeapTaggingLevel level = __get_heap_tagging_level(phdr_start, phdr_ct, load_bias, &memtag_stack);
if (level == M_HEAP_TAGGING_LEVEL_SYNC || level == M_HEAP_TAGGING_LEVEL_ASYNC) {
unsigned long prctl_arg = PR_TAGGED_ADDR_ENABLE | PR_MTE_TAG_SET_NONZERO;
@@ -308,6 +311,17 @@
if (prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg | PR_MTE_TCF_SYNC, 0, 0, 0) == 0 ||
prctl(PR_SET_TAGGED_ADDR_CTRL, prctl_arg, 0, 0, 0) == 0) {
__libc_shared_globals()->initial_heap_tagging_level = level;
+ __libc_shared_globals()->initial_memtag_stack = memtag_stack;
+
+ if (memtag_stack) {
+ void* page_start =
+ reinterpret_cast<void*>(PAGE_START(reinterpret_cast<uintptr_t>(stack_top)));
+ if (mprotect(page_start, PAGE_SIZE, PROT_READ | PROT_WRITE | PROT_MTE | PROT_GROWSDOWN)) {
+ async_safe_fatal("error: failed to set PROT_MTE on main thread stack: %s\n",
+ strerror(errno));
+ }
+ }
+
return;
}
}
@@ -319,7 +333,7 @@
}
}
#else // __aarch64__
-void __libc_init_mte(const void*, size_t, uintptr_t) {}
+void __libc_init_mte(const void*, size_t, uintptr_t, void*) {}
#endif // __aarch64__
void __libc_init_profiling_handlers() {
@@ -331,11 +345,9 @@
signal(BIONIC_SIGNAL_ART_PROFILER, SIG_IGN);
}
-__noreturn static void __real_libc_init(void *raw_args,
- void (*onexit)(void) __unused,
- int (*slingshot)(int, char**, char**),
- structors_array_t const * const structors,
- bionic_tcb* temp_tcb) {
+__attribute__((no_sanitize("memtag"))) __noreturn static void __real_libc_init(
+ void* raw_args, void (*onexit)(void) __unused, int (*slingshot)(int, char**, char**),
+ structors_array_t const* const structors, bionic_tcb* temp_tcb) {
BIONIC_STOP_UNWIND;
// Initialize TLS early so system calls and errno work.
@@ -349,7 +361,7 @@
__libc_init_main_thread_final();
__libc_init_common();
__libc_init_mte(reinterpret_cast<ElfW(Phdr)*>(getauxval(AT_PHDR)), getauxval(AT_PHNUM),
- /*load_bias = */ 0);
+ /*load_bias = */ 0, /*stack_top = */ raw_args);
__libc_init_scudo();
__libc_init_profiling_handlers();
__libc_init_fork_handler();
@@ -379,11 +391,9 @@
//
// The 'structors' parameter contains pointers to various initializer
// arrays that must be run before the program's 'main' routine is launched.
-__attribute__((no_sanitize("hwaddress")))
-__noreturn void __libc_init(void* raw_args,
- void (*onexit)(void) __unused,
- int (*slingshot)(int, char**, char**),
- structors_array_t const * const structors) {
+__attribute__((no_sanitize("hwaddress", "memtag"))) __noreturn void __libc_init(
+ void* raw_args, void (*onexit)(void) __unused, int (*slingshot)(int, char**, char**),
+ structors_array_t const* const structors) {
bionic_tcb temp_tcb = {};
#if __has_feature(hwaddress_sanitizer)
// Install main thread TLS early. It will be initialized later in __libc_init_main_thread. For now
diff --git a/libc/bionic/pthread_create.cpp b/libc/bionic/pthread_create.cpp
index 121b26f..417ce76 100644
--- a/libc/bionic/pthread_create.cpp
+++ b/libc/bionic/pthread_create.cpp
@@ -40,15 +40,16 @@
#include <async_safe/log.h>
+#include "platform/bionic/macros.h"
+#include "platform/bionic/mte.h"
+#include "private/ErrnoRestorer.h"
#include "private/ScopedRWLock.h"
#include "private/bionic_constants.h"
#include "private/bionic_defs.h"
#include "private/bionic_globals.h"
-#include "platform/bionic/macros.h"
#include "private/bionic_ssp.h"
#include "private/bionic_systrace.h"
#include "private/bionic_tls.h"
-#include "private/ErrnoRestorer.h"
// x86 uses segment descriptors rather than a direct pointer to TLS.
#if defined(__i386__)
@@ -88,7 +89,13 @@
static void __init_alternate_signal_stack(pthread_internal_t* thread) {
// Create and set an alternate signal stack.
- void* stack_base = mmap(nullptr, SIGNAL_STACK_SIZE, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+ int prot = PROT_READ | PROT_WRITE;
+#ifdef __aarch64__
+ if (atomic_load(&__libc_globals->memtag_stack)) {
+ prot |= PROT_MTE;
+ }
+#endif
+ void* stack_base = mmap(nullptr, SIGNAL_STACK_SIZE, prot, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (stack_base != MAP_FAILED) {
// Create a guard to catch stack overflows in signal handlers.
if (mprotect(stack_base, PTHREAD_GUARD_SIZE, PROT_NONE) == -1) {
@@ -224,12 +231,19 @@
return {};
}
const size_t writable_size = mmap_size - stack_guard_size - PTHREAD_GUARD_SIZE;
- if (mprotect(space + stack_guard_size,
- writable_size,
- PROT_READ | PROT_WRITE) != 0) {
- async_safe_format_log(ANDROID_LOG_WARN, "libc",
- "pthread_create failed: couldn't mprotect R+W %zu-byte thread mapping region: %s",
- writable_size, strerror(errno));
+ int prot = PROT_READ | PROT_WRITE;
+ const char* prot_str = "R+W";
+#ifdef __aarch64__
+ if (atomic_load(&__libc_globals->memtag_stack)) {
+ prot |= PROT_MTE;
+ prot_str = "R+W+MTE";
+ }
+#endif
+ if (mprotect(space + stack_guard_size, writable_size, prot) != 0) {
+ async_safe_format_log(
+ ANDROID_LOG_WARN, "libc",
+ "pthread_create failed: couldn't mprotect %s %zu-byte thread mapping region: %s", prot_str,
+ writable_size, strerror(errno));
munmap(space, mmap_size);
return {};
}
diff --git a/libc/bionic/pthread_internal.h b/libc/bionic/pthread_internal.h
index 071a5bc..7222b62 100644
--- a/libc/bionic/pthread_internal.h
+++ b/libc/bionic/pthread_internal.h
@@ -160,6 +160,13 @@
bionic_tls* bionic_tls;
int errno_value;
+
+ // The last observed value of SP in a vfork child process.
+ // The part of the stack between this address and the value of SP when the vfork parent process
+ // regains control may have stale MTE tags and needs cleanup. This field is only meaningful while
+ // the parent is waiting for the vfork child to return control by calling either exec*() or
+ // exit().
+ void* vfork_child_stack_bottom;
};
struct ThreadMapping {
diff --git a/libc/include/bits/fortify/fcntl.h b/libc/include/bits/fortify/fcntl.h
index 7063541..7fe60f4 100644
--- a/libc/include/bits/fortify/fcntl.h
+++ b/libc/include/bits/fortify/fcntl.h
@@ -41,7 +41,7 @@
#if defined(__BIONIC_FORTIFY)
#define __open_too_many_args_error "too many arguments"
#define __open_too_few_args_error "called with O_CREAT or O_TMPFILE, but missing mode"
-#define __open_useless_modes_warning "has superfluous mode bits; missing O_CREAT?"
+#define __open_useless_modes_warning "has superfluous mode bits; missing O_CREAT or O_TMPFILE?"
/* O_TMPFILE shares bits with O_DIRECTORY. */
#define __open_modes_useful(flags) (((flags) & O_CREAT) || ((flags) & O_TMPFILE) == O_TMPFILE)
diff --git a/libc/malloc_debug/Config.cpp b/libc/malloc_debug/Config.cpp
index 5fd511f..6a81277 100644
--- a/libc/malloc_debug/Config.cpp
+++ b/libc/malloc_debug/Config.cpp
@@ -83,6 +83,19 @@
},
{
+ "backtrace_size",
+ {BACKTRACE_SPECIFIC_SIZES, &Config::SetBacktraceSize},
+ },
+ {
+ "backtrace_min_size",
+ {BACKTRACE_SPECIFIC_SIZES, &Config::SetBacktraceMinSize},
+ },
+ {
+ "backtrace_max_size",
+ {BACKTRACE_SPECIFIC_SIZES, &Config::SetBacktraceMaxSize},
+ },
+
+ {
"backtrace",
{BACKTRACE | TRACK_ALLOCS, &Config::SetBacktrace},
},
@@ -297,6 +310,23 @@
return true;
}
+bool Config::SetBacktraceSize(const std::string& option, const std::string& value) {
+ if (!ParseValue(option, value, 1, SIZE_MAX, &backtrace_min_size_bytes_)) {
+ return false;
+ }
+ backtrace_max_size_bytes_ = backtrace_min_size_bytes_;
+
+ return true;
+}
+
+bool Config::SetBacktraceMinSize(const std::string& option, const std::string& value) {
+ return ParseValue(option, value, 1, SIZE_MAX, &backtrace_min_size_bytes_);
+}
+
+bool Config::SetBacktraceMaxSize(const std::string& option, const std::string& value) {
+ return ParseValue(option, value, 1, SIZE_MAX, &backtrace_max_size_bytes_);
+}
+
bool Config::SetExpandAlloc(const std::string& option, const std::string& value) {
return ParseValue(option, value, DEFAULT_EXPAND_BYTES, 1, MAX_EXPAND_BYTES, &expand_alloc_bytes_);
}
@@ -403,6 +433,8 @@
backtrace_enabled_ = false;
backtrace_dump_on_exit_ = false;
backtrace_dump_prefix_ = DEFAULT_BACKTRACE_DUMP_PREFIX;
+ backtrace_min_size_bytes_ = 0;
+ backtrace_max_size_bytes_ = SIZE_MAX;
check_unreachable_signal_ = SIGRTMAX - 16;
// Process each option name we can find.
diff --git a/libc/malloc_debug/Config.h b/libc/malloc_debug/Config.h
index 5e3ff71..ef1d2a9 100644
--- a/libc/malloc_debug/Config.h
+++ b/libc/malloc_debug/Config.h
@@ -47,6 +47,7 @@
constexpr uint64_t ABORT_ON_ERROR = 0x800;
constexpr uint64_t VERBOSE = 0x1000;
constexpr uint64_t CHECK_UNREACHABLE_ON_SIGNAL = 0x2000;
+constexpr uint64_t BACKTRACE_SPECIFIC_SIZES = 0x4000;
// In order to guarantee posix compliance, set the minimum alignment
// to 8 bytes for 32 bit systems and 16 bytes for 64 bit systems.
@@ -90,6 +91,9 @@
uint8_t fill_alloc_value() const { return fill_alloc_value_; }
uint8_t fill_free_value() const { return fill_free_value_; }
+ size_t backtrace_min_size_bytes() const { return backtrace_min_size_bytes_; }
+ size_t backtrace_max_size_bytes() const { return backtrace_max_size_bytes_; }
+
int record_allocs_signal() const { return record_allocs_signal_; }
size_t record_allocs_num_entries() const { return record_allocs_num_entries_; }
const std::string& record_allocs_file() const { return record_allocs_file_; }
@@ -121,6 +125,10 @@
bool SetBacktraceDumpOnExit(const std::string& option, const std::string& value);
bool SetBacktraceDumpPrefix(const std::string& option, const std::string& value);
+ bool SetBacktraceSize(const std::string& option, const std::string& value);
+ bool SetBacktraceMinSize(const std::string& option, const std::string& value);
+ bool SetBacktraceMaxSize(const std::string& option, const std::string& value);
+
bool SetExpandAlloc(const std::string& option, const std::string& value);
bool SetFreeTrack(const std::string& option, const std::string& value);
@@ -145,6 +153,8 @@
size_t backtrace_frames_ = 0;
bool backtrace_dump_on_exit_ = false;
std::string backtrace_dump_prefix_;
+ size_t backtrace_min_size_bytes_ = 0;
+ size_t backtrace_max_size_bytes_ = 0;
size_t fill_on_alloc_bytes_ = 0;
size_t fill_on_free_bytes_ = 0;
diff --git a/libc/malloc_debug/PointerData.cpp b/libc/malloc_debug/PointerData.cpp
index d062086..5ab2232 100644
--- a/libc/malloc_debug/PointerData.cpp
+++ b/libc/malloc_debug/PointerData.cpp
@@ -136,7 +136,22 @@
return true;
}
-size_t PointerData::AddBacktrace(size_t num_frames) {
+static inline bool ShouldBacktraceAllocSize(size_t size_bytes) {
+ static bool only_backtrace_specific_sizes =
+ g_debug->config().options() & BACKTRACE_SPECIFIC_SIZES;
+ if (!only_backtrace_specific_sizes) {
+ return true;
+ }
+ static size_t min_size_bytes = g_debug->config().backtrace_min_size_bytes();
+ static size_t max_size_bytes = g_debug->config().backtrace_max_size_bytes();
+ return size_bytes >= min_size_bytes && size_bytes <= max_size_bytes;
+}
+
+size_t PointerData::AddBacktrace(size_t num_frames, size_t size_bytes) {
+ if (!ShouldBacktraceAllocSize(size_bytes)) {
+ return kBacktraceEmptyIndex;
+ }
+
std::vector<uintptr_t> frames;
std::vector<unwindstack::FrameData> frames_info;
if (g_debug->config().options() & BACKTRACE_FULL) {
@@ -198,7 +213,7 @@
void PointerData::Add(const void* ptr, size_t pointer_size) {
size_t hash_index = 0;
if (backtrace_enabled_) {
- hash_index = AddBacktrace(g_debug->config().backtrace_frames());
+ hash_index = AddBacktrace(g_debug->config().backtrace_frames(), pointer_size);
}
std::lock_guard<std::mutex> pointer_guard(pointer_mutex_);
@@ -336,11 +351,11 @@
}
}
-void* PointerData::AddFreed(const void* ptr) {
+void* PointerData::AddFreed(const void* ptr, size_t size_bytes) {
size_t hash_index = 0;
size_t num_frames = g_debug->config().free_track_backtrace_num_frames();
if (num_frames) {
- hash_index = AddBacktrace(num_frames);
+ hash_index = AddBacktrace(num_frames, size_bytes);
}
void* last = nullptr;
diff --git a/libc/malloc_debug/PointerData.h b/libc/malloc_debug/PointerData.h
index 97eec0a..3194bab 100644
--- a/libc/malloc_debug/PointerData.h
+++ b/libc/malloc_debug/PointerData.h
@@ -137,13 +137,13 @@
static void IteratePointers(std::function<void(uintptr_t pointer)> fn);
- static size_t AddBacktrace(size_t num_frames);
+ static size_t AddBacktrace(size_t num_frames, size_t size_bytes);
static void RemoveBacktrace(size_t hash_index);
static void Add(const void* pointer, size_t size);
static void Remove(const void* pointer);
- static void* AddFreed(const void* pointer);
+ static void* AddFreed(const void* pointer, size_t size_bytes);
static void LogFreeError(const FreePointerInfoType& info, size_t usable_size);
static void LogFreeBacktrace(const void* ptr);
static void VerifyFreedPointer(const FreePointerInfoType& info);
diff --git a/libc/malloc_debug/README.md b/libc/malloc_debug/README.md
index 6b046db..3667624 100644
--- a/libc/malloc_debug/README.md
+++ b/libc/malloc_debug/README.md
@@ -160,6 +160,41 @@
on the signal will be backtrace\_dump\_prefix.**PID**.txt. The filename chosen
when the program exits will be backtrace\_dump\_prefix.**PID**.exit.txt.
+### backtrace\_min\_size=ALLOCATION\_SIZE\_BYTES
+As of U, setting this in combination with the backtrace option means
+that only allocations of a size greater than or equal to
+**ALLOCATION\_SIZE\_BYTES** will be backtraced. When used in combination
+with the backtrace\_max\_size option, then allocations greater than or
+equal to backtrace\_min\_size and less than or equal to
+backtrace\_max\_size will be backtraced. The backtrace\_size option
+overrides this option, and should not be used at the same time.
+
+This option can also be used in combination with other tools such
+as [libmemunreachable](https://android.googlesource.com/platform/system/memory/libmemunreachable/+/master/README.md)
+to only get backtraces for sizes of allocations listed as being leaked.
+
+### backtrace\_max\_size=ALLOCATION\_SIZE\_BYTES
+As of U, setting this in combination with the backtrace option means
+that only allocations of a size less than or equal to
+**ALLOCATION\_SIZE\_BYTES** will be backtraced. When used in combination
+with the backtrace\_min\_size option, then allocations greater than or
+equal to backtrace\_min\_size and less than or equal to
+backtrace\_max\_size will be backtraced. The backtrace\_size option
+overrides this option, and should not be used at the same time.
+
+This option can also be used in combination with other tools such
+as [libmemunreachable](https://android.googlesource.com/platform/system/memory/libmemunreachable/+/master/README.md)
+to only get backtraces for sizes of allocations listed as being leaked.
+
+### backtrace\_size=ALLOCATION\_SIZE\_BYTES
+As of U, setting this in combination with the backtrace option means
+that only allocations of size **ALLOCATION\_SIZE\_BYTES** will be backtraced.
+This option overrides the backtrace\_min\_size and the backtrace\_max\_size.
+
+This option can also be used in combination with other tools such
+as [libmemunreachable](https://android.googlesource.com/platform/system/memory/libmemunreachable/+/master/README.md)
+to only get backtraces for sizes of allocations listed as being leaked.
+
### backtrace\_full
As of Q, any time that a backtrace is gathered, a different algorithm is used
that is extra thorough and can unwind through Java frames. This will run
diff --git a/libc/malloc_debug/malloc_debug.cpp b/libc/malloc_debug/malloc_debug.cpp
index 35cda83..617858a 100644
--- a/libc/malloc_debug/malloc_debug.cpp
+++ b/libc/malloc_debug/malloc_debug.cpp
@@ -522,8 +522,8 @@
if (g_debug->config().options() & FILL_ON_FREE) {
size_t fill_bytes = g_debug->config().fill_on_free_bytes();
- bytes = (bytes < fill_bytes) ? bytes : fill_bytes;
- memset(pointer, g_debug->config().fill_free_value(), bytes);
+ fill_bytes = (bytes < fill_bytes) ? bytes : fill_bytes;
+ memset(pointer, g_debug->config().fill_free_value(), fill_bytes);
}
if (g_debug->TrackPointers()) {
@@ -536,7 +536,7 @@
// frees at the same time and we wind up trying to really free this
// pointer from another thread, while still trying to free it in
// this function.
- pointer = PointerData::AddFreed(pointer);
+ pointer = PointerData::AddFreed(pointer, bytes);
if (pointer != nullptr) {
if (g_debug->HeaderEnabled()) {
pointer = g_debug->GetHeader(pointer)->orig_pointer;
diff --git a/libc/malloc_debug/tests/malloc_debug_config_tests.cpp b/libc/malloc_debug/tests/malloc_debug_config_tests.cpp
index d08ba98..0a0eaef 100644
--- a/libc/malloc_debug/tests/malloc_debug_config_tests.cpp
+++ b/libc/malloc_debug/tests/malloc_debug_config_tests.cpp
@@ -779,3 +779,69 @@
"which does not take a value\n");
ASSERT_STREQ((log_msg + usage_string).c_str(), getFakeLogPrint().c_str());
}
+
+TEST_F(MallocDebugConfigTest, size) {
+ ASSERT_TRUE(InitConfig("backtrace_size=37")) << getFakeLogPrint();
+ ASSERT_EQ(BACKTRACE_SPECIFIC_SIZES, config->options());
+ ASSERT_EQ(37U, config->backtrace_min_size_bytes());
+ ASSERT_EQ(37U, config->backtrace_max_size_bytes());
+
+ ASSERT_FALSE(InitConfig("backtrace_size")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_size=0")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_size=-1")) << getFakeLogPrint();
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string log_msg("6 malloc_debug malloc_testing: bad value for option 'backtrace_size'\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_size', value "
+ "must be >= 1: 0\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_size', value "
+ "cannot be negative: -1\n" +
+ usage_string);
+ ASSERT_STREQ(log_msg.c_str(), getFakeLogPrint().c_str());
+}
+
+TEST_F(MallocDebugConfigTest, min_size) {
+ ASSERT_TRUE(InitConfig("backtrace_min_size=9")) << getFakeLogPrint();
+ ASSERT_EQ(BACKTRACE_SPECIFIC_SIZES, config->options());
+ ASSERT_EQ(9U, config->backtrace_min_size_bytes());
+ ASSERT_EQ(SIZE_MAX, config->backtrace_max_size_bytes());
+
+ ASSERT_FALSE(InitConfig("backtrace_min_size")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_min_size=0")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_min_size=-1")) << getFakeLogPrint();
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string log_msg("6 malloc_debug malloc_testing: bad value for option 'backtrace_min_size'\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_min_size', "
+ "value must be >= 1: 0\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_min_size', "
+ "value cannot be negative: -1\n" +
+ usage_string);
+ ASSERT_STREQ(log_msg.c_str(), getFakeLogPrint().c_str());
+}
+
+TEST_F(MallocDebugConfigTest, max_size) {
+ ASSERT_TRUE(InitConfig("backtrace_max_size=13")) << getFakeLogPrint();
+ ASSERT_EQ(BACKTRACE_SPECIFIC_SIZES, config->options());
+ ASSERT_EQ(0U, config->backtrace_min_size_bytes());
+ ASSERT_EQ(13U, config->backtrace_max_size_bytes());
+
+ ASSERT_FALSE(InitConfig("backtrace_max_size")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_max_size=0")) << getFakeLogPrint();
+ ASSERT_FALSE(InitConfig("backtrace_max_size=-1")) << getFakeLogPrint();
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string log_msg("6 malloc_debug malloc_testing: bad value for option 'backtrace_max_size'\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_max_size', "
+ "value must be >= 1: 0\n" +
+ usage_string +
+ "6 malloc_debug malloc_testing: bad value for option 'backtrace_max_size', "
+ "value cannot be negative: -1\n" +
+ usage_string);
+ ASSERT_STREQ(log_msg.c_str(), getFakeLogPrint().c_str());
+}
diff --git a/libc/malloc_debug/tests/malloc_debug_unit_tests.cpp b/libc/malloc_debug/tests/malloc_debug_unit_tests.cpp
index b97e638..961ddd3 100644
--- a/libc/malloc_debug/tests/malloc_debug_unit_tests.cpp
+++ b/libc/malloc_debug/tests/malloc_debug_unit_tests.cpp
@@ -2754,3 +2754,178 @@
"6 malloc_debug Unreachable check failed, run setenforce 0 and try again.\n",
getFakeLogPrint().c_str());
}
+
+TEST_F(MallocDebugTest, backtrace_only_some_sizes_with_backtrace_size) {
+ Init("leak_track backtrace backtrace_size=120");
+
+ backtrace_fake_add(std::vector<uintptr_t>{0x1000, 0x2000, 0x3000});
+
+ void* pointer1 = debug_malloc(119);
+ ASSERT_TRUE(pointer1 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xa000, 0xb000, 0xc000, 0xd000});
+
+ void* pointer2 = debug_malloc(120);
+ ASSERT_TRUE(pointer2 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xfe000, 0xde000, 0xce000, 0xbe000, 0xae000});
+
+ void* pointer3 = debug_malloc(121);
+ ASSERT_TRUE(pointer3 != nullptr);
+
+ debug_finalize();
+ initialized = false;
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string expected_log = android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 121 at %p (leak 1 of 3)\n", pointer3);
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 120 at %p (leak 2 of 3)\n", pointer2);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0x1000\n";
+ expected_log += "6 malloc_debug #01 pc 0x2000\n";
+ expected_log += "6 malloc_debug #02 pc 0x3000\n";
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 119 at %p (leak 3 of 3)\n", pointer1);
+ ASSERT_STREQ(expected_log.c_str(), getFakeLogPrint().c_str());
+}
+
+TEST_F(MallocDebugTest, backtrace_only_some_sizes_with_backtrace_min_size) {
+ Init("leak_track backtrace backtrace_min_size=1000");
+
+ backtrace_fake_add(std::vector<uintptr_t>{0x1000, 0x2000, 0x3000});
+
+ void* pointer1 = debug_malloc(500);
+ ASSERT_TRUE(pointer1 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xa000, 0xb000, 0xc000, 0xd000});
+
+ void* pointer2 = debug_malloc(1000);
+ ASSERT_TRUE(pointer2 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xfe000, 0xde000, 0xce000, 0xbe000, 0xae000});
+
+ void* pointer3 = debug_malloc(1001);
+ ASSERT_TRUE(pointer3 != nullptr);
+
+ debug_finalize();
+ initialized = false;
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string expected_log = android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1001 at %p (leak 1 of 3)\n",
+ pointer3);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0xa000\n";
+ expected_log += "6 malloc_debug #01 pc 0xb000\n";
+ expected_log += "6 malloc_debug #02 pc 0xc000\n";
+ expected_log += "6 malloc_debug #03 pc 0xd000\n";
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1000 at %p (leak 2 of 3)\n",
+ pointer2);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0x1000\n";
+ expected_log += "6 malloc_debug #01 pc 0x2000\n";
+ expected_log += "6 malloc_debug #02 pc 0x3000\n";
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 500 at %p (leak 3 of 3)\n", pointer1);
+ ASSERT_STREQ(expected_log.c_str(), getFakeLogPrint().c_str());
+}
+
+TEST_F(MallocDebugTest, backtrace_only_some_sizes_with_backtrace_max_size) {
+ Init("leak_track backtrace backtrace_max_size=1000");
+
+ backtrace_fake_add(std::vector<uintptr_t>{0x1000, 0x2000, 0x3000});
+
+ void* pointer1 = debug_malloc(1000);
+ ASSERT_TRUE(pointer1 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xa000, 0xb000, 0xc000, 0xd000});
+
+ void* pointer2 = debug_malloc(1001);
+ ASSERT_TRUE(pointer2 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xfe000, 0xde000, 0xce000, 0xbe000, 0xae000});
+
+ void* pointer3 = debug_malloc(5000);
+ ASSERT_TRUE(pointer3 != nullptr);
+
+ debug_finalize();
+ initialized = false;
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string expected_log = android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 5000 at %p (leak 1 of 3)\n",
+ pointer3);
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1001 at %p (leak 2 of 3)\n",
+ pointer2);
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1000 at %p (leak 3 of 3)\n",
+ pointer1);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0x1000\n";
+ expected_log += "6 malloc_debug #01 pc 0x2000\n";
+ expected_log += "6 malloc_debug #02 pc 0x3000\n";
+
+ ASSERT_STREQ(expected_log.c_str(), getFakeLogPrint().c_str());
+}
+
+TEST_F(MallocDebugTest, backtrace_only_some_sizes_with_backtrace_min_max_size) {
+ Init("leak_track backtrace backtrace_min_size=50 backtrace_max_size=1000");
+
+ backtrace_fake_add(std::vector<uintptr_t>{0x1000, 0x2000, 0x3000});
+
+ void* pointer1 = debug_malloc(49);
+ ASSERT_TRUE(pointer1 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xa000, 0xb000, 0xc000, 0xd000});
+
+ void* pointer2 = debug_malloc(50);
+ ASSERT_TRUE(pointer2 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0xfe000, 0xde000, 0xce000, 0xbe000, 0xae000});
+
+ void* pointer3 = debug_malloc(1000);
+ ASSERT_TRUE(pointer3 != nullptr);
+
+ backtrace_fake_add(std::vector<uintptr_t>{0x1a000, 0x1b000, 0x1c000, 0x1d000, 0x1e000});
+
+ void* pointer4 = debug_malloc(1001);
+ ASSERT_TRUE(pointer4 != nullptr);
+
+ debug_finalize();
+ initialized = false;
+
+ ASSERT_STREQ("", getFakeLogBuf().c_str());
+ std::string expected_log = android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1001 at %p (leak 1 of 4)\n",
+ pointer4);
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 1000 at %p (leak 2 of 4)\n",
+ pointer3);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0xa000\n";
+ expected_log += "6 malloc_debug #01 pc 0xb000\n";
+ expected_log += "6 malloc_debug #02 pc 0xc000\n";
+ expected_log += "6 malloc_debug #03 pc 0xd000\n";
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 50 at %p (leak 3 of 4)\n", pointer2);
+ expected_log += "6 malloc_debug Backtrace at time of allocation:\n";
+ expected_log += "6 malloc_debug #00 pc 0x1000\n";
+ expected_log += "6 malloc_debug #01 pc 0x2000\n";
+ expected_log += "6 malloc_debug #02 pc 0x3000\n";
+
+ expected_log += android::base::StringPrintf(
+ "6 malloc_debug +++ malloc_testing leaked block of size 49 at %p (leak 4 of 4)\n", pointer1);
+
+ ASSERT_STREQ(expected_log.c_str(), getFakeLogPrint().c_str());
+}
diff --git a/libc/private/bionic_asm_arm64.h b/libc/private/bionic_asm_arm64.h
index 5d83d9b..ffc7181 100644
--- a/libc/private/bionic_asm_arm64.h
+++ b/libc/private/bionic_asm_arm64.h
@@ -76,3 +76,4 @@
#define NT_MEMTAG_LEVEL_ASYNC 1
#define NT_MEMTAG_LEVEL_SYNC 2
#define NT_MEMTAG_HEAP 4
+#define NT_MEMTAG_STACK 8
diff --git a/libc/private/bionic_globals.h b/libc/private/bionic_globals.h
index e105c18..3c86ef5 100644
--- a/libc/private/bionic_globals.h
+++ b/libc/private/bionic_globals.h
@@ -47,6 +47,7 @@
vdso_entry vdso[VDSO_END];
long setjmp_cookie;
uintptr_t heap_pointer_tag;
+ _Atomic(bool) memtag_stack;
// In order to allow a complete switch between dispatch tables without
// the need for copying each function by function in the structure,
@@ -112,6 +113,7 @@
const char* scudo_ring_buffer = nullptr;
HeapTaggingLevel initial_heap_tagging_level = M_HEAP_TAGGING_LEVEL_NONE;
+ bool initial_memtag_stack = false;
};
__LIBC_HIDDEN__ libc_shared_globals* __libc_shared_globals();
diff --git a/linker/linker_main.cpp b/linker/linker_main.cpp
index 9e5be34..f0e7b1b 100644
--- a/linker/linker_main.cpp
+++ b/linker/linker_main.cpp
@@ -68,7 +68,8 @@
static void set_bss_vma_name(soinfo* si);
-void __libc_init_mte(const void* phdr_start, size_t phdr_count, uintptr_t load_bias);
+void __libc_init_mte(const void* phdr_start, size_t phdr_count, uintptr_t load_bias,
+ void* stack_top);
// These should be preserved static to avoid emitting
// RELATIVE relocations for the part of the code running
@@ -408,7 +409,7 @@
}
}
- __libc_init_mte(somain->phdr, somain->phnum, somain->load_bias);
+ __libc_init_mte(somain->phdr, somain->phnum, somain->load_bias, args.argv);
#endif
// Register the main executable and the linker upfront to have
diff --git a/linker/linker_transparent_hugepage_support.cpp b/linker/linker_transparent_hugepage_support.cpp
index 65ba4cd..0631577 100644
--- a/linker/linker_transparent_hugepage_support.cpp
+++ b/linker/linker_transparent_hugepage_support.cpp
@@ -39,6 +39,6 @@
return false;
}
return enabled.find("[never]") == std::string::npos;
- };
+ }();
return transparent_hugepages_supported;
}
diff --git a/tests/struct_layout_test.cpp b/tests/struct_layout_test.cpp
index 0123ed9..10c100a 100644
--- a/tests/struct_layout_test.cpp
+++ b/tests/struct_layout_test.cpp
@@ -30,7 +30,7 @@
#define CHECK_OFFSET(name, field, offset) \
check_offset(#name, #field, offsetof(name, field), offset);
#ifdef __LP64__
- CHECK_SIZE(pthread_internal_t, 776);
+ CHECK_SIZE(pthread_internal_t, 784);
CHECK_OFFSET(pthread_internal_t, next, 0);
CHECK_OFFSET(pthread_internal_t, prev, 8);
CHECK_OFFSET(pthread_internal_t, tid, 16);
@@ -55,6 +55,7 @@
CHECK_OFFSET(pthread_internal_t, dlerror_buffer, 248);
CHECK_OFFSET(pthread_internal_t, bionic_tls, 760);
CHECK_OFFSET(pthread_internal_t, errno_value, 768);
+ CHECK_OFFSET(pthread_internal_t, vfork_child_stack_bottom, 776);
CHECK_SIZE(bionic_tls, 12200);
CHECK_OFFSET(bionic_tls, key_data, 0);
CHECK_OFFSET(bionic_tls, locale, 2080);
@@ -72,7 +73,7 @@
CHECK_OFFSET(bionic_tls, bionic_systrace_disabled, 12193);
CHECK_OFFSET(bionic_tls, padding, 12194);
#else
- CHECK_SIZE(pthread_internal_t, 668);
+ CHECK_SIZE(pthread_internal_t, 672);
CHECK_OFFSET(pthread_internal_t, next, 0);
CHECK_OFFSET(pthread_internal_t, prev, 4);
CHECK_OFFSET(pthread_internal_t, tid, 8);
@@ -97,6 +98,7 @@
CHECK_OFFSET(pthread_internal_t, dlerror_buffer, 148);
CHECK_OFFSET(pthread_internal_t, bionic_tls, 660);
CHECK_OFFSET(pthread_internal_t, errno_value, 664);
+ CHECK_OFFSET(pthread_internal_t, vfork_child_stack_bottom, 668);
CHECK_SIZE(bionic_tls, 11080);
CHECK_OFFSET(bionic_tls, key_data, 0);
CHECK_OFFSET(bionic_tls, locale, 1040);
diff --git a/tests/unistd_test.cpp b/tests/unistd_test.cpp
index 49fec12..02da585 100644
--- a/tests/unistd_test.cpp
+++ b/tests/unistd_test.cpp
@@ -1652,6 +1652,8 @@
TEST(UNISTD_TEST, close_range) {
#if defined(__GLIBC__)
GTEST_SKIP() << "glibc too old";
+#elif defined(ANDROID_HOST_MUSL)
+ GTEST_SKIP() << "musl does not have close_range";
#else // __GLIBC__
int fd = open("/proc/version", O_RDONLY);
ASSERT_GE(fd, 0);
@@ -1683,4 +1685,4 @@
ASSERT_TRUE(android::base::ReadFdToString(tf2.fd, &content));
ASSERT_EQ("hello world", content);
#endif // __GLIBC__
-}
\ No newline at end of file
+}