Merge "Add documentation for device assignment" into main am: c89f056468
Original change: https://android-review.googlesource.com/c/platform/packages/modules/Virtualization/+/2960349
Change-Id: Iad56b39305104941c68e2c556bb7281c704dd887
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
diff --git a/README.md b/README.md
index 3935f93..210f70e 100644
--- a/README.md
+++ b/README.md
@@ -29,3 +29,4 @@
* [Building and running a demo app in C++](demo_native/README.md)
* [Debugging](docs/debug)
* [Using custom VM](docs/custom_vm.md)
+* [Device assignment](docs/device_assignment.md)
\ No newline at end of file
diff --git a/docs/device_assignment.md b/docs/device_assignment.md
new file mode 100644
index 0000000..4c5b477
--- /dev/null
+++ b/docs/device_assignment.md
@@ -0,0 +1,239 @@
+# Getting started with device assignment
+
+Device assignment allows a VM to have direct access to HW without host/hyp
+intervention. AVF uses `vfio-platform` for device assignment, and host kernel
+support is required.
+
+This document explains how to setup and launch VM with device assignments.
+
+## VM device assignment DTBO (a.k.a. VM DTBO)
+
+For device assignment, a VM device assignment DTBO (a.k.a. VM DTBO) is required.
+VM DTBO is a device tree overlay which describes all assignable devices
+information. Information includes physical reg, IOMMU, device properties, and
+dependencies.
+
+VM DTBO allows to pass extra properties of assignable platform
+devices to the VM (which can't be discovered from the HW) while keeping the VMM
+device-agnostic.
+
+When the host boots, the bootloader provides VM DTBO to both Android and pvmfw.
+
+When a VM boots, the VMM selectively applies the DTBO based from provided
+labels, describing the assigned devices.
+
+## Prepare VM DTBO
+
+VM DTBO should be included in the dtbo partition. It should be in its own
+entry, and not together with any host OS's. See [DTB/DTBO Paritions] for
+partition format.
+
+[DTB/DTBO Paritions]: https://source.android.com/docs/core/architecture/dto/partitions
+
+### Write VM DTS for VM DTBO
+
+DTBO is compiled from device tree source (DTS) with `dtc` tool. [DTBO syntax]
+explains basic syntax of DTS.
+
+[DTBO syntax]: https://source.android.com/docs/core/architecture/dto/syntax
+
+Here are details and requirements:
+
+#### Describe assignable devices
+
+VM DTBO should describe assignable devices and their labels.
+
+* VM DTBO should have assignable devices in the `&{/}`, so it can be
+ overlaid onto VM DT. Assignable devices should be backed by physical device.
+ * We only support overlaying onto root node (i.e. `&{/}`) to prevent
+ unexpected modification of VM DT.
+* VM DTBO should have labels for assignable devices, so AVF can recognize
+ assignable device list. Labels should point to valid 'overlayable' nodes.
+ * Overlayable node is a node that would be applied to the base device tree
+ when DTBO is applied.
+
+#### Describe physical devices and physical IOMMUs
+
+VM DTBO should describe a `/host` node which describes physical devices and
+physical IOMMUs. The `/host` node only describes information for verification of
+assigned devices, and wouldn't be applied to VM DT. Here are details:
+
+* Physical IOMMU nodes
+ * IOMMU nodes must have a phandle to be referenced by a physical device node.
+ * IOMMU nodes must have `<android,pvmfw,token>` property. The property
+ describes the IOMMU token. An IOMMU token is a hypervisor-specific `<u64>`
+ which uniquely identifies a physical IOMMU. IOMMU token must be constant
+ across the VM boot for provisioning by pvmfw remains valid. The token must
+ be kept up-to-date across hypervisor updates.
+ * IOMMU nodes should be multi-master IOMMUs. (i.e. `#iommu-cells = <1>`)
+ * Other `#iommu-cells` values aren't supported for now.
+ * See: [Device tree binding for IOMMUs][IOMMU]
+* Physical device nodes
+ * Physical device nodes must have a `<android,pvmfw,target>` property that
+ references an overlayable node. The overlayable node contains the properties
+ that would be included in VM DT.
+ * Physical device nodes must have `<reg>` property to provide physical
+ regions.
+ * Physical device nodes can optionally contain `<iommus>` property. The
+ property is a prop-encoded-array and contains a number of
+ (iommu phandle, SID) pairs.
+ * IOMMU can be shared among devices, but should use distinct SIDs. Sharing
+ the same IOMMU-SID pair among multiple devices isn't supported for now.
+
+[IOMMU]: https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
+
+#### Describe dependencies
+
+VM DTBO may have dependencies via phandle references. When a device node is
+assigned, dependencies of the node are also applied to VM DT.
+
+When dependencies are applied, siblings or children nodes of dependencies are
+ignored unless explicitly referenced.
+
+#### VM DTBO example
+
+Here's a simple example device tree source with four assignable devices nodes.
+
+```dts
+/dts-v1/;
+/plugin/;
+
+/ {
+ // host node describes physical devices and IOMMUs, and wouldn't be applied to VM DT
+ host {
+ #address-cells = <0x2>;
+ #size-cells = <0x1>;
+ rng {
+ reg = <0x0 0x12f00000 0x1000>;
+ iommus = <&iommu0 0x3>;
+ android,pvmfw,target = <&rng>;
+ };
+ light {
+ reg = <0x0 0x00f00000 0x1000>, <0x0 0x00f10000 0x1000>;
+ iommus = <&iommu1 0x4>, <&iommu2 0x5>;
+ android,pvmfw,target = <&light>;
+ };
+ led {
+ reg = <0x0 0x12000000 0x1000>;
+ iommus = <&iommu1 0x3>;
+ android,pvmfw,target = <&led>;
+ };
+ bus0 {
+ #address-cells = <0x1>;
+ #size-cells = <0x1>;
+ backlight {
+ reg = <0x300 0x100>;
+ android,pvmfw,target = <&backlight>;
+ };
+ };
+ iommu0: iommu0 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x12e40000>;
+ };
+ iommu1: iommu1 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x40000>;
+ };
+ iommu2: iommu2 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x50000>;
+ };
+ };
+};
+
+// Beginning of the assignable devices. Assigned devices would be applied to VM DT
+&{/} { // We only allows to overlay to root node
+ rng: rng {
+ compatible = "android,rng";
+ android,rng,ignore-gctrl-reset;
+ };
+ light: light {
+ compatible = "android,light";
+ version = <0x1 0x2>;
+ };
+ led: led {
+ compatible = "android,led";
+ prop = <0x555>;
+ };
+ bus0 {
+ backlight: backlight {
+ compatible = "android,backlight";
+ android,backlight,ignore-gctrl-reset;
+ };
+ };
+};
+```
+
+If you compile the above with `dtc -@`, then you'll get `__symbols__` for free.
+The generated `__symbols__` indicates that there are four assignable devices.
+
+```dts
+ // generated __symbols__. AVF will ignore non-overlayable nodes.
+ __symbols__ {
+ iommu0 = "/host/iommu0";
+ iommu1 = "/host/iommu1";
+ iommu2 = "/host/iommu2";
+ rng = "/fragment@rng/__overlay__/rng";
+ light = "/fragment@sensor/__overlay__/light";
+ led = "/fragment@led/__overlay__/led";
+ backlight = "/fragment@backlight/__overlay__/bus0/backlight";
+ };
+```
+
+## Prepare AVF assignable devices XML
+
+AVF requires assignable device information to unbind from the host device driver
+and bind to VFIO driver. The information should be provided in an XML file at
+`/vendor/etc/avf/assignable_devices.xml`.
+
+Here's example.
+
+```xml
+<devices>
+ <device>
+ <kind>sensor</kind>
+ <dtbo_label>light</dtbo_label>
+ <sysfs_path>/sys/bus/platform/devices/16d00000.light</sysfs_path>
+ </device>
+</devices>
+```
+
+* `<kind>`: Device kind. Currently only used for debugging purposes and not used
+ for device assignment.
+* `<dtbo_label>`: Label in the VM DTBO (i.e. symbols in `__symbols__`). Must be
+ unique.
+* `<sysfs_path>`: Sysfs path of the device in host, used to bind to the VFIO
+ driver. Must be unique in the XML.
+
+## Boot with VM DTBO
+
+Bootloader should provide VM DTBO to both Android and pvmfw.
+
+### Provide VM DTBO index in dtbo.img
+
+Bootloader should provide the VM DTBO index with sysprop
+`ro.boot.hypervisor.vm_dtbo_idx.`. DTBO index represents DTBO location in
+dtbo.img.
+
+### Provide VM DTBO in the pvmfw config
+
+For protected VM, bootloader must provide VM DTBO to the pvmfw. pvmfw sanitizes
+incoming device tree with the VM DTBO.
+
+For more detail about providing VM DTBO in pvmfw,
+see: [pvmfw/README.md](../pvmfw/README.md#configuration-data-format)
+
+
+## Launch VM with device assignment
+
+We don't support client API yet in Android V, but you can use CLI to test device
+assignment. Note that host kernel support is required.
+
+Specify `--devices ${sysfs_path}` when booting VM. The parameter can be repeated
+multiple times for specifying multiple devices.
+
+Here's an example:
+
+```sh
+adb shell /apex/com.android.virt/bin/vm run-microdroid --devices /sys/bus/platform/devices/16d00000.light
+```
\ No newline at end of file
diff --git a/pvmfw/README.md b/pvmfw/README.md
index d7884fb..2758a5d 100644
--- a/pvmfw/README.md
+++ b/pvmfw/README.md
@@ -195,7 +195,8 @@
In version 1.1, a third blob is added.
-- entry 2 may point to a [DTBO] that describes VM DTBO for device assignment.
+- entry 2 may point to a [DTBO] that describes VM DA DTBO for
+ [device assignment][device_assignment].
pvmfw will provision assigned devices with the VM DTBO.
In version 1.2, a fourth blob is added.
@@ -225,6 +226,7 @@
[header]: src/config.rs
[DTBO]: https://android.googlesource.com/platform/external/dtc/+/refs/heads/main/Documentation/dt-object-internal.txt
[debug_policy]: ../docs/debug/README.md#debug-policy
+[device_assignment]: ../docs/device_assignment.md
[secretkeeper_key]: https://android.googlesource.com/platform/system/secretkeeper/+/refs/heads/main/README.md#secretkeeper-public-key
[vendor_hashtree_digest]: ../microdroid/README.md#verification-of-vendor-image