Guard against overflow errors for transparent regions

Applications may send arbitrary transparent regions to SurfaceFlinger
by overriding ViewGroup#gatherTransparentRegion, or a savvy binary may
apply a custom transparent region directly on a transaction. Arbitrary
transparent regions may cause SurfaceFlinger to crash on malformed
input.

We can guard against some of these crashes by constraining the
transparent region to be within the layer bounds

Bug: 230227800
Test: Stress tests
Test: libcompositionengine_test
Change-Id: I38f6f42dce38cdf2f34ba41af658adfdad290417
diff --git a/services/surfaceflinger/CompositionEngine/src/Output.cpp b/services/surfaceflinger/CompositionEngine/src/Output.cpp
index 509312f..aea6798 100644
--- a/services/surfaceflinger/CompositionEngine/src/Output.cpp
+++ b/services/surfaceflinger/CompositionEngine/src/Output.cpp
@@ -586,8 +586,29 @@
     // Remove the transparent area from the visible region
     if (!layerFEState->isOpaque) {
         if (tr.preserveRects()) {
-            // transform the transparent region
-            transparentRegion = tr.transform(layerFEState->transparentRegionHint);
+            // Clip the transparent region to geomLayerBounds first
+            // The transparent region may be influenced by applications, for
+            // instance, by overriding ViewGroup#gatherTransparentRegion with a
+            // custom view. Once the layer stack -> display mapping is known, we
+            // must guard against very wrong inputs to prevent underflow or
+            // overflow errors. We do this here by constraining the transparent
+            // region to be within the pre-transform layer bounds, since the
+            // layer bounds are expected to play nicely with the full
+            // transform.
+            const Region clippedTransparentRegionHint =
+                    layerFEState->transparentRegionHint.intersect(
+                            Rect(layerFEState->geomLayerBounds));
+
+            if (clippedTransparentRegionHint.isEmpty()) {
+                if (!layerFEState->transparentRegionHint.isEmpty()) {
+                    ALOGD("Layer: %s had an out of bounds transparent region",
+                          layerFE->getDebugName());
+                    layerFEState->transparentRegionHint.dump("transparentRegionHint");
+                }
+                transparentRegion.clear();
+            } else {
+                transparentRegion = tr.transform(clippedTransparentRegionHint);
+            }
         } else {
             // transformation too complex, can't do the
             // transparent region optimization.