Merge "Add document about RAM dump"
diff --git a/docs/debug/ramdump.md b/docs/debug/ramdump.md
new file mode 100644
index 0000000..a0d9bf2
--- /dev/null
+++ b/docs/debug/ramdump.md
@@ -0,0 +1,159 @@
+# Doing RAM dump of a Microdroid VM and analyzing it
+
+A Microdroid VM creates a RAM dump of itself when the kernel panics. This
+document explains how the dump can be obtained and analyzed.
+
+## Force triggering a RAM dump
+
+RAM dump is created automatically when there's a kernel panic. However, for
+debugging purpose, you can forcibly trigger it via magic SysRq key.
+
+```shell
+$ adb shell /apex/com.android.virt/bin/vm run-app ... // run a Microdroid VM
+$ m vm_shell; vm_shell // connect to the VM
+# echo c > /proc/sysrq-trigger // force trigger a crash
+```
+
+Then you will see following message showing that crash is detected and the
+crashdump kernel is executed.
+
+```
+[ 14.949892][ T148] sysrq: Trigger a crash
+[ 14.952133][ T148] Kernel panic - not syncing: sysrq triggered crash
+[ 14.955309][ T148] CPU: 0 PID: 148 Comm: sh Kdump: loaded Not tainted 5.15.60-android14-5-04357-gbac79d727aea-ab9013362 #1
+[ 14.957803][ T148] Hardware name: linux,dummy-virt (DT)
+[ 14.959053][ T148] Call trace:
+[ 14.959809][ T148] dump_backtrace.cfi_jt+0x0/0x8
+[ 14.961019][ T148] dump_stack_lvl+0x68/0x98
+[ 14.962137][ T148] panic+0x160/0x3f4
+
+----------snip----------
+
+[ 14.998693][ T148] Starting crashdump kernel...
+[ 14.999411][ T148] Bye!
+Booting Linux on physical CPU 0x0000000000 [0x412fd050]
+Linux version 5.15.44+ (build-user@build-host) (Android (8508608, based on r450784e) clang version 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6), LLD 14.0.7) #1 SMP PREEMPT Thu Jul 7 02:57:03 UTC 2022
+achine model: linux,dummy-virt
+earlycon: uart8250 at MMIO 0x00000000000003f8 (options '')
+printk: bootconsole [uart8250] enabled
+
+----------snip----------
+
+Run /bin/crashdump as init process
+Crashdump started
+Size is 98836480 bytes
+.....................................................................random: crng init done
+...............................done
+reboot: Restarting system with command 'kernel panic'
+```
+
+## Obtaining the RAM dump
+
+By default, RAM dumps are sent to tombstone. To see which tombstone file is for
+the RAM dump, look into the log.
+
+```shell
+$ adb logcat | grep SYSTEM_TOMBSTONE
+09-22 17:24:28.798 1335 1504 I BootReceiver: Copying /data/tombstones/tombstone_47 to DropBox (SYSTEM_TOMBSTONE)
+```
+
+In the above example, the RAM dump is saved as `/data/tombstones/tombstone_47`.
+You can download this using `adb pull`.
+
+```shell
+$ adb root && adb pull /data/tombstones/tombstone_47 ramdump && adb unroot
+```
+
+Alternatively, you can specify the path to where RAM dump is stored when
+launching the VM using the `--ramdump` option of the `vm` tool.
+
+```shell
+$ adb shelll /apex/com.android.virt/bin/vm run-app --ramdump /data/local/tmp/virt/ramdump ...
+```
+
+In the above example, the RAM dump is saved to `/data/local/tmp/virt/ramdump`.
+
+## Analyzing the RAM dump
+
+### Building the crash(8) tool
+
+You first need to build the crash(8) tool for the target architecture, which in most case is aarch64.
+
+Download the source code and build it as follows. This needs to be done only once.
+
+```shell
+$ wget https://github.com/crash-utility/crash/archive/refs/tags/8.0.1.tar.gz -O - | tar xzvf
+$ make -C crash-8.0.1 target=ARM64
+```
+
+### Obtaining vmlinux
+
+You also need the image of the kernel binary with debuggin enabled. The kernel
+binary should be the same as the actual kernel that you used in the Microdroid
+VM that crashed. To identify which kernel it was, look for the kernel version
+number in the logcat log.
+
+```
+[ 14.955309][ T148] CPU: 0 PID: 148 Comm: sh Kdump: loaded Not tainted 5.15.60-android14-5-04357-gbac79d727aea-ab9013362 #1
+```
+
+Here, the version number is
+`5.15.60-android14-5-04357-gbac79d727aea-ab9013362`. What is important here is
+the last component: `ab9013362`. The numbers after `ab` is the Android Build ID
+of the kernel.
+
+With the build ID, you can find the image from `ci.android.com` and download
+it. The direct link to the image is `https://ci.android.com/builds/submitted/9013362/kernel_microdroid_aarch64/latest/vmlinux`.
+
+DON'T forget to replace `9013362` with the actual build ID of the kernel you used.
+
+### Running crash(8) with the RAM dump and the kernel image
+
+```shell
+$ crash-8.0.1/crash ramdump vmlinux
+```
+
+You can now analyze the RAM dump using the various commands that crash(8) provides. For example, `bt <pid>` command shows the stack trace of a process.
+
+```
+crash> bt
+PID: 148 TASK: ffffff8001a2d880 CPU: 0 COMMAND: "sh"
+ #0 [ffffffc00926b9f0] machine_kexec at ffffffd48a852004
+ #1 [ffffffc00926bb90] __crash_kexec at ffffffd48a948008
+ #2 [ffffffc00926bc40] panic at ffffffd48a86e2a8
+ #3 [ffffffc00926bc90] sysrq_handle_crash.35db4764f472dc1c4a43f39b71f858ea at ffffffd48ad985c8
+ #4 [ffffffc00926bca0] __handle_sysrq at ffffffd48ad980e4
+ #5 [ffffffc00926bcf0] write_sysrq_trigger.35db4764f472dc1c4a43f39b71f858ea at ffffffd48ad994f0
+ #6 [ffffffc00926bd10] proc_reg_write.bc7c2a3e70d8726163739fbd131db16e at ffffffd48ab4d280
+ #7 [ffffffc00926bda0] vfs_write at ffffffd48aaaa1a4
+ #8 [ffffffc00926bdf0] ksys_write at ffffffd48aaaa5b0
+ #9 [ffffffc00926be30] __arm64_sys_write at ffffffd48aaaa644
+#10 [ffffffc00926be40] invoke_syscall at ffffffd48a84b55c
+#11 [ffffffc00926be60] do_el0_svc at ffffffd48a84b424
+#12 [ffffffc00926be80] el0_svc at ffffffd48b0a29e4
+#13 [ffffffc00926bea0] el0t_64_sync_handler at ffffffd48b0a2950
+#14 [ffffffc00926bfe0] el0t_64_sync at ffffffd48a811644
+ PC: 00000079d880b798 LR: 00000064b4afec8c SP: 0000007ff6ddb2e0
+ X29: 0000007ff6ddb360 X28: 0000007ff6ddb320 X27: 00000064b4b238e8
+ X26: 00000079d9c49000 X25: 0000000000000000 X24: b40000784870fda9
+ X23: 00000064b4b236f8 X22: 0000007ff6ddb340 X21: 0000007ff6ddb338
+ X20: b40000784870f618 X19: 0000000000000002 X18: 00000079daea4000
+ X17: 00000079d880b790 X16: 00000079d882dee0 X15: 0000000000000080
+ X14: 0000000000000000 X13: 0000008f00000160 X12: 000000004870f6ac
+ X11: 0000000000000008 X10: 000000000009c000 X9: b40000784870f618
+ X8: 0000000000000040 X7: 000000e70000000b X6: 0000020500000210
+ X5: 00000079d883a984 X4: ffffffffffffffff X3: ffffffffffffffff
+ X2: 0000000000000002 X1: b40000784870f618 X0: 0000000000000001
+ ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 00001000
+```
+
+Above shows that the shell process that executed `echo c > /proc/sysrq-trigger`
+actually triggered a crash in the kernel.
+
+For more commands of crash(8), refer to the man page, or embedded `help` command.
+
+
+
+
+
+