Fix the order of applying layer alpha and color matrices in GLESRenderEngine

If the system want to do a color inversion,
there is an unexpected result due to applying the layer alpha first.
Solution: RenderEngine should apply
the layer's alpha after multiplying by the color matrix.

For example, given the following color inversion matrix:
| r.r  r.g  r.b |   | -1.0   0.0   0.0 |
| g.r  g.g  g.b | = |  0.0  -1.0   0.0 |
| b.r  b.g  b.b |   |  0.0   0.0  -1.0 |
| Tr   Tg   Tb  |   |  1.0   1.0   1.0 |

We want to get a black layer when applying the color transform to a white layer;
However, the white layer applies layer alpha before color matrix. We can get
the below calculation: When layer alpha is 0, R_out value was always Tr.
Then we do not get a black layer when the system applies color inversion to a
white layer.
R_out = (R_in * layer_alpha) * r.r + Tr

Therefore we apply the color matrix first, so we can get the below calculation.
When the layer alpha is 0, we can get a black layer.
R_out = (R_in * r.r + Tr) * layer_alpha
      = (R_in * r.r * layer_alpha) + (Tr * layer_alpha)

adds RenderEngineTest::fillBufferColorTransformZeroLayerAlpha

Bug: 157204341
Test: color inversion when switching pages in the Setting app
Test: RenderEngineTest
Change-Id: I206184aae84042290b00e411709e04e54579780a
diff --git a/libs/renderengine/gl/ProgramCache.cpp b/libs/renderengine/gl/ProgramCache.cpp
index 7fc0499..5ff9240 100644
--- a/libs/renderengine/gl/ProgramCache.cpp
+++ b/libs/renderengine/gl/ProgramCache.cpp
@@ -740,15 +740,6 @@
         if (needs.isOpaque()) {
             fs << "gl_FragColor.a = 1.0;";
         }
-        if (needs.hasAlpha()) {
-            // modulate the current alpha value with alpha set
-            if (needs.isPremultiplied()) {
-                // ... and the color too if we're premultiplied
-                fs << "gl_FragColor *= color.a;";
-            } else {
-                fs << "gl_FragColor.a *= color.a;";
-            }
-        }
     }
 
     if (needs.hasTransformMatrix() || (needs.getInputTF() != needs.getOutputTF()) ||
@@ -768,6 +759,23 @@
         }
     }
 
+    /*
+     * Whether applying layer alpha before or after color transform doesn't matter,
+     * as long as we can undo premultiplication. But we cannot un-premultiply
+     * for color transform if the layer alpha = 0, e.g. 0 / (0 + 0.0019) = 0.
+     */
+    if (!needs.drawShadows()) {
+        if (needs.hasAlpha()) {
+            // modulate the current alpha value with alpha set
+            if (needs.isPremultiplied()) {
+                // ... and the color too if we're premultiplied
+                fs << "gl_FragColor *= color.a;";
+            } else {
+                fs << "gl_FragColor.a *= color.a;";
+            }
+        }
+    }
+
     if (needs.hasRoundedCorners()) {
         if (needs.isPremultiplied()) {
             fs << "gl_FragColor *= vec4(applyCornerRadius(outCropCoords));";