libbinder: RPC cap transaction size at 100KB

Why?
- Android code uses -fno-exceptions and generally doesn't check for OOM
  conditions (unlike the Linux kernel itself!). Even if we check for
  allocation success, a successful allocation here may mean even a 1
  byte allocation on another thread or by the server will cause a
  failure.
- kernel binder can have by default 1MB of concurrent transactions
  at a time. A transaction of size 100KB is already exceedingly
  dangerous to the runtime, since in a big process, this could cause
  other processes to reach the limit.

In the future, we could increase this cap (lowering is potentially
difficult) or make it customizable.

Bug: 167966510
Test: binderRpcTest, binderRpcBenchmark, binder_rpc_fuzzer
Change-Id: Ia215f1a00412654ce08e6bced14d4da4a0a46987
diff --git a/libs/binder/RpcState.h b/libs/binder/RpcState.h
index 83d0344..31f8a22 100644
--- a/libs/binder/RpcState.h
+++ b/libs/binder/RpcState.h
@@ -101,10 +101,10 @@
      */
     void terminate();
 
-    // alternative to std::vector<uint8_t> that doesn't abort on too big of allocations
-    struct ByteVec {
-        explicit ByteVec(size_t size)
-              : mData(size > 0 ? new (std::nothrow) uint8_t[size] : nullptr), mSize(size) {}
+    // Alternative to std::vector<uint8_t> that doesn't abort on allocation failure and caps
+    // large allocations to avoid being requested from allocating too much data.
+    struct CommandData {
+        explicit CommandData(size_t size);
         bool valid() { return mSize == 0 || mData != nullptr; }
         size_t size() { return mSize; }
         uint8_t* data() { return mData.get(); }
@@ -128,7 +128,7 @@
                                            const RpcWireHeader& command);
     [[nodiscard]] status_t processTransactInternal(const base::unique_fd& fd,
                                                    const sp<RpcSession>& session,
-                                                   ByteVec transactionData);
+                                                   CommandData transactionData);
     [[nodiscard]] status_t processDecStrong(const base::unique_fd& fd,
                                             const RpcWireHeader& command);
 
@@ -163,7 +163,7 @@
 
         // async transaction queue, _only_ for local binder
         struct AsyncTodo {
-            ByteVec data;
+            CommandData data;
             uint64_t asyncNumber = 0;
 
             bool operator<(const AsyncTodo& o) const {