Merge "Don't leak threads."
diff --git a/docs/debug/tracing.md b/docs/debug/tracing.md
index 0038440..ebc0ac3 100644
--- a/docs/debug/tracing.md
+++ b/docs/debug/tracing.md
@@ -32,8 +32,15 @@
Consider first familiarizing yourself with Perfetto documentation for recording traces on Android:
https://perfetto.dev/docs/quickstart/android-tracing.
-So far it is only possible to capture hypervisor trace events by providing the full trace config
-file to Perfetto. Here is the minimal
+The [record_android_trace](
+https://cs.android.com/android/platform/superproject/+/master:external/perfetto/tools/record_android_trace)
+script supports a shortcut to capture all hypervisor events that are known to Perfetto:
+
+```shell
+external/perfetto/tools/record_android_trace hyp -t 15s -b 32mb -o /tmp/hyp.pftrace
+```
+
+Alternatively you can use full trace config to capture hypervisor. Example usage:
```shell
cat<<EOF>config.pbtx
@@ -66,9 +73,19 @@
#### Capturing hypervisor traces on QEMU
-TODO(b/271412868): fill in this section
+Perfetto supports capturing traces on Linux: https://perfetto.dev/docs/quickstart/linux-tracing.
+However, since pKVM hypervisor is only supported on arm64, you will need to cross-compile Perfetto
+binaries for linux-arm64 (unless you have an arm64 workstation).
-TODO(b/271412868): Stay tuned, more docs coming soon!
+1. Checkout Perfetto repository: https://perfetto.dev/docs/contributing/getting-started
+2. Follow https://perfetto.dev/docs/contributing/build-instructions#cross-compiling-for-linux-arm-64
+ to compile Perfetto binaries for arm64 architecture.
+3. Copy the tracebox binary to QEMU
+4. Run `tracebox` binary on QEMU to capture traces, it's interface is very similar to the
+`record_android_trace` binary. E.g. to capture all hypervisor events run:
+```shell
+tracebox -t 15s -b 32mb hyp
+```
## Microdroid VM tracing
diff --git a/microdroid/README.md b/microdroid/README.md
index 41278a5..28785fd 100644
--- a/microdroid/README.md
+++ b/microdroid/README.md
@@ -138,6 +138,7 @@
TEST_ROOT=/data/local/tmp/virt
adb shell /apex/com.android.virt/bin/vm run-app \
--log $TEST_ROOT/log.txt \
+--console $TEST_ROOT/console.txt \
PATH_TO_YOUR_APP \
$TEST_ROOT/MyApp.apk.idsig \
$TEST_ROOT/instance.img \
@@ -145,9 +146,9 @@
```
The last command lets you know the CID assigned to the VM. The console output
-from the VM is stored to `$TEST_ROOT/log.txt` file for debugging purpose. If you
-omit the `--log $TEST_ROOT/log.txt` option, it will be emitted to the current
-console.
+from the VM is stored to `$TEST_ROOT/console.txt` and logcat is stored to
+`$TEST_ROOT/log.txt` file for debugging purpose. If you omit `--log` or
+`--console` option, they will be emitted to the current console.
Stopping the VM can be done as follows:
@@ -159,12 +160,50 @@
invoked with the `--daemonize` flag. If the flag was not used, press Ctrl+C on
the console where the `vm run-app` command was invoked.
-## ADB
+## Debuggable microdroid
-On userdebug builds, you can have an adb connection to microdroid. To do so,
-first, delete `$TEST_ROOT/instance.img`; this is because changing debug settings
-requires a new instance. Then add the `--debug=full` flag to the
-`/apex/com.android.virt/bin/vm run-app` command, and then
+### Debugging features
+Microdroid supports following debugging features:
+
+- VM log
+- console output
+- kernel output
+- logcat output
+- [ramdump](../docs/debug/ramdump.md)
+- crashdump
+- [adb](#adb)
+- [gdb](#debugging-the-payload-on-microdroid)
+
+### Enabling debugging features
+There's two ways to enable the debugging features:
+
+#### Option 1) Running microdroid on AVF debug policy configured device
+
+microdroid can be started with debugging features by debug policies from the
+host. Host bootloader may provide debug policies to host OS's device tree for
+VMs.
+
+For protected VM, such device tree will be available in microdroid. microdroid
+can check which debuging features is enabled.
+
+Here are list of device tree properties for debugging features.
+
+- `/avf/guest/common/log`: `<1>` to enable kernel log and logcat. Ignored
+ otherwise.
+- `/avf/guest/common/ramdump`: `<1>` to enable ramdump. Ignored otherwise.
+- `/avf/guest/microdroid/adb`: `<1>` to enable `adb`. Ignored otherwise.
+
+#### Option 2) Lauching microdroid with debug level.
+
+microdroid can be started with debugging features. To do so, first, delete
+`$TEST_ROOT/instance.img`; this is because changing debug settings requires a
+new instance. Then add the `--debug=full` flag to the
+`/apex/com.android.virt/bin/vm run-app` command. This will enable all debugging
+features.
+
+### ADB
+
+If `adb` connection is enabled, launch following command.
```sh
vm_shell
@@ -175,7 +214,7 @@
Once you have an adb connection with `vm_shell`, `localhost:8000` will be the
serial of microdroid.
-## Debugging the payload on microdroid
+### Debugging the payload on microdroid
Like a normal adb device, you can debug native processes using `lldbclient.py`
script, either by running a new process, or attaching to an existing process.
diff --git a/microdroid/init.rc b/microdroid/init.rc
index 5187a12..c997bfd 100644
--- a/microdroid/init.rc
+++ b/microdroid/init.rc
@@ -35,7 +35,10 @@
restorecon /mnt/extra-apk
# Wait for apexd to finish activating APEXes before starting more processes.
- wait_for_prop apexd.status activated
+ # Microdroid starts apexd in VM mode in which apexd doesn't wait for init after setting
+ # apexd.status to activated, but immediately transitions to ready. Therefore, it's not safe to
+ # wait for the activated status, by the time this line is reached it may be already be ready.
+ wait_for_prop apexd.status ready
perform_apex_config
# Notify to microdroid_manager that perform_apex_config is done.
diff --git a/pvmfw/README.md b/pvmfw/README.md
index c652f01..04ad8c4 100644
--- a/pvmfw/README.md
+++ b/pvmfw/README.md
@@ -234,3 +234,31 @@
[dice-dt]: https://www.kernel.org/doc/Documentation/devicetree/bindings/reserved-memory/google%2Copen-dice.yaml
[Layering]: https://pigweed.googlesource.com/open-dice/+/refs/heads/main/docs/specification.md#layering-details
[Trusty-BCC]: https://android.googlesource.com/trusty/lib/+/1696be0a8f3a7103/lib/hwbcc/common/swbcc.c#554
+
+#### pVM Device Tree Overlay
+
+Config header can provide a DTBO to be overlaid on top of the baseline device
+tree from crosvm.
+
+The DTBO may contain debug policies as follows.
+
+```
+/ {
+ fragment@avf {
+ target-path = "/";
+
+ __overlay__ {
+ avf {
+ /* your debug policy here */
+ };
+ };
+ };
+}; /* end of avf */
+```
+
+For specifying DTBO, host bootloader should apply the DTBO to both host
+OS's device tree and config header of `pvmfw`. Both `virtualizationmanager` and
+`pvmfw` will prepare for debugging features.
+
+For details about device tree properties for debug policies, see
+[microdroid's debugging policy guide](../microdroid/README.md#option-1-running-microdroid-on-avf-debug-policy-configured-device).