Merge "pvmfw: Expand the allowed numbers of opp tables" into main
diff --git a/README.md b/README.md
index 3935f93..210f70e 100644
--- a/README.md
+++ b/README.md
@@ -29,3 +29,4 @@
* [Building and running a demo app in C++](demo_native/README.md)
* [Debugging](docs/debug)
* [Using custom VM](docs/custom_vm.md)
+* [Device assignment](docs/device_assignment.md)
\ No newline at end of file
diff --git a/docs/device_assignment.md b/docs/device_assignment.md
new file mode 100644
index 0000000..4c5b477
--- /dev/null
+++ b/docs/device_assignment.md
@@ -0,0 +1,239 @@
+# Getting started with device assignment
+
+Device assignment allows a VM to have direct access to HW without host/hyp
+intervention. AVF uses `vfio-platform` for device assignment, and host kernel
+support is required.
+
+This document explains how to setup and launch VM with device assignments.
+
+## VM device assignment DTBO (a.k.a. VM DTBO)
+
+For device assignment, a VM device assignment DTBO (a.k.a. VM DTBO) is required.
+VM DTBO is a device tree overlay which describes all assignable devices
+information. Information includes physical reg, IOMMU, device properties, and
+dependencies.
+
+VM DTBO allows to pass extra properties of assignable platform
+devices to the VM (which can't be discovered from the HW) while keeping the VMM
+device-agnostic.
+
+When the host boots, the bootloader provides VM DTBO to both Android and pvmfw.
+
+When a VM boots, the VMM selectively applies the DTBO based from provided
+labels, describing the assigned devices.
+
+## Prepare VM DTBO
+
+VM DTBO should be included in the dtbo partition. It should be in its own
+entry, and not together with any host OS's. See [DTB/DTBO Paritions] for
+partition format.
+
+[DTB/DTBO Paritions]: https://source.android.com/docs/core/architecture/dto/partitions
+
+### Write VM DTS for VM DTBO
+
+DTBO is compiled from device tree source (DTS) with `dtc` tool. [DTBO syntax]
+explains basic syntax of DTS.
+
+[DTBO syntax]: https://source.android.com/docs/core/architecture/dto/syntax
+
+Here are details and requirements:
+
+#### Describe assignable devices
+
+VM DTBO should describe assignable devices and their labels.
+
+* VM DTBO should have assignable devices in the `&{/}`, so it can be
+ overlaid onto VM DT. Assignable devices should be backed by physical device.
+ * We only support overlaying onto root node (i.e. `&{/}`) to prevent
+ unexpected modification of VM DT.
+* VM DTBO should have labels for assignable devices, so AVF can recognize
+ assignable device list. Labels should point to valid 'overlayable' nodes.
+ * Overlayable node is a node that would be applied to the base device tree
+ when DTBO is applied.
+
+#### Describe physical devices and physical IOMMUs
+
+VM DTBO should describe a `/host` node which describes physical devices and
+physical IOMMUs. The `/host` node only describes information for verification of
+assigned devices, and wouldn't be applied to VM DT. Here are details:
+
+* Physical IOMMU nodes
+ * IOMMU nodes must have a phandle to be referenced by a physical device node.
+ * IOMMU nodes must have `<android,pvmfw,token>` property. The property
+ describes the IOMMU token. An IOMMU token is a hypervisor-specific `<u64>`
+ which uniquely identifies a physical IOMMU. IOMMU token must be constant
+ across the VM boot for provisioning by pvmfw remains valid. The token must
+ be kept up-to-date across hypervisor updates.
+ * IOMMU nodes should be multi-master IOMMUs. (i.e. `#iommu-cells = <1>`)
+ * Other `#iommu-cells` values aren't supported for now.
+ * See: [Device tree binding for IOMMUs][IOMMU]
+* Physical device nodes
+ * Physical device nodes must have a `<android,pvmfw,target>` property that
+ references an overlayable node. The overlayable node contains the properties
+ that would be included in VM DT.
+ * Physical device nodes must have `<reg>` property to provide physical
+ regions.
+ * Physical device nodes can optionally contain `<iommus>` property. The
+ property is a prop-encoded-array and contains a number of
+ (iommu phandle, SID) pairs.
+ * IOMMU can be shared among devices, but should use distinct SIDs. Sharing
+ the same IOMMU-SID pair among multiple devices isn't supported for now.
+
+[IOMMU]: https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
+
+#### Describe dependencies
+
+VM DTBO may have dependencies via phandle references. When a device node is
+assigned, dependencies of the node are also applied to VM DT.
+
+When dependencies are applied, siblings or children nodes of dependencies are
+ignored unless explicitly referenced.
+
+#### VM DTBO example
+
+Here's a simple example device tree source with four assignable devices nodes.
+
+```dts
+/dts-v1/;
+/plugin/;
+
+/ {
+ // host node describes physical devices and IOMMUs, and wouldn't be applied to VM DT
+ host {
+ #address-cells = <0x2>;
+ #size-cells = <0x1>;
+ rng {
+ reg = <0x0 0x12f00000 0x1000>;
+ iommus = <&iommu0 0x3>;
+ android,pvmfw,target = <&rng>;
+ };
+ light {
+ reg = <0x0 0x00f00000 0x1000>, <0x0 0x00f10000 0x1000>;
+ iommus = <&iommu1 0x4>, <&iommu2 0x5>;
+ android,pvmfw,target = <&light>;
+ };
+ led {
+ reg = <0x0 0x12000000 0x1000>;
+ iommus = <&iommu1 0x3>;
+ android,pvmfw,target = <&led>;
+ };
+ bus0 {
+ #address-cells = <0x1>;
+ #size-cells = <0x1>;
+ backlight {
+ reg = <0x300 0x100>;
+ android,pvmfw,target = <&backlight>;
+ };
+ };
+ iommu0: iommu0 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x12e40000>;
+ };
+ iommu1: iommu1 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x40000>;
+ };
+ iommu2: iommu2 {
+ #iommu-cells = <0x1>;
+ android,pvmfw,token = <0x0 0x50000>;
+ };
+ };
+};
+
+// Beginning of the assignable devices. Assigned devices would be applied to VM DT
+&{/} { // We only allows to overlay to root node
+ rng: rng {
+ compatible = "android,rng";
+ android,rng,ignore-gctrl-reset;
+ };
+ light: light {
+ compatible = "android,light";
+ version = <0x1 0x2>;
+ };
+ led: led {
+ compatible = "android,led";
+ prop = <0x555>;
+ };
+ bus0 {
+ backlight: backlight {
+ compatible = "android,backlight";
+ android,backlight,ignore-gctrl-reset;
+ };
+ };
+};
+```
+
+If you compile the above with `dtc -@`, then you'll get `__symbols__` for free.
+The generated `__symbols__` indicates that there are four assignable devices.
+
+```dts
+ // generated __symbols__. AVF will ignore non-overlayable nodes.
+ __symbols__ {
+ iommu0 = "/host/iommu0";
+ iommu1 = "/host/iommu1";
+ iommu2 = "/host/iommu2";
+ rng = "/fragment@rng/__overlay__/rng";
+ light = "/fragment@sensor/__overlay__/light";
+ led = "/fragment@led/__overlay__/led";
+ backlight = "/fragment@backlight/__overlay__/bus0/backlight";
+ };
+```
+
+## Prepare AVF assignable devices XML
+
+AVF requires assignable device information to unbind from the host device driver
+and bind to VFIO driver. The information should be provided in an XML file at
+`/vendor/etc/avf/assignable_devices.xml`.
+
+Here's example.
+
+```xml
+<devices>
+ <device>
+ <kind>sensor</kind>
+ <dtbo_label>light</dtbo_label>
+ <sysfs_path>/sys/bus/platform/devices/16d00000.light</sysfs_path>
+ </device>
+</devices>
+```
+
+* `<kind>`: Device kind. Currently only used for debugging purposes and not used
+ for device assignment.
+* `<dtbo_label>`: Label in the VM DTBO (i.e. symbols in `__symbols__`). Must be
+ unique.
+* `<sysfs_path>`: Sysfs path of the device in host, used to bind to the VFIO
+ driver. Must be unique in the XML.
+
+## Boot with VM DTBO
+
+Bootloader should provide VM DTBO to both Android and pvmfw.
+
+### Provide VM DTBO index in dtbo.img
+
+Bootloader should provide the VM DTBO index with sysprop
+`ro.boot.hypervisor.vm_dtbo_idx.`. DTBO index represents DTBO location in
+dtbo.img.
+
+### Provide VM DTBO in the pvmfw config
+
+For protected VM, bootloader must provide VM DTBO to the pvmfw. pvmfw sanitizes
+incoming device tree with the VM DTBO.
+
+For more detail about providing VM DTBO in pvmfw,
+see: [pvmfw/README.md](../pvmfw/README.md#configuration-data-format)
+
+
+## Launch VM with device assignment
+
+We don't support client API yet in Android V, but you can use CLI to test device
+assignment. Note that host kernel support is required.
+
+Specify `--devices ${sysfs_path}` when booting VM. The parameter can be repeated
+multiple times for specifying multiple devices.
+
+Here's an example:
+
+```sh
+adb shell /apex/com.android.virt/bin/vm run-microdroid --devices /sys/bus/platform/devices/16d00000.light
+```
\ No newline at end of file
diff --git a/pvmfw/README.md b/pvmfw/README.md
index d7884fb..2758a5d 100644
--- a/pvmfw/README.md
+++ b/pvmfw/README.md
@@ -195,7 +195,8 @@
In version 1.1, a third blob is added.
-- entry 2 may point to a [DTBO] that describes VM DTBO for device assignment.
+- entry 2 may point to a [DTBO] that describes VM DA DTBO for
+ [device assignment][device_assignment].
pvmfw will provision assigned devices with the VM DTBO.
In version 1.2, a fourth blob is added.
@@ -225,6 +226,7 @@
[header]: src/config.rs
[DTBO]: https://android.googlesource.com/platform/external/dtc/+/refs/heads/main/Documentation/dt-object-internal.txt
[debug_policy]: ../docs/debug/README.md#debug-policy
+[device_assignment]: ../docs/device_assignment.md
[secretkeeper_key]: https://android.googlesource.com/platform/system/secretkeeper/+/refs/heads/main/README.md#secretkeeper-public-key
[vendor_hashtree_digest]: ../microdroid/README.md#verification-of-vendor-image
diff --git a/service_vm/test_apk/src/native/main.rs b/service_vm/test_apk/src/native/main.rs
index 26b8062..d5d599d 100644
--- a/service_vm/test_apk/src/native/main.rs
+++ b/service_vm/test_apk/src/native/main.rs
@@ -45,7 +45,7 @@
android_logger::init_once(
android_logger::Config::default()
.with_tag("service_vm_client")
- .with_min_level(log::Level::Debug),
+ .with_max_level(log::LevelFilter::Debug),
);
// Redirect panic messages to logcat.
panic::set_hook(Box::new(|panic_info| {
diff --git a/virtualizationmanager/fsfdt/src/lib.rs b/virtualizationmanager/fsfdt/src/lib.rs
index 549df04..84e50c1 100644
--- a/virtualizationmanager/fsfdt/src/lib.rs
+++ b/virtualizationmanager/fsfdt/src/lib.rs
@@ -26,8 +26,8 @@
/// Creates a Fdt from /proc/device-tree style directory by wrapping a mutable slice
fn from_fs(fs_path: &Path, fdt_buffer: &'a mut [u8]) -> Result<&'a mut Self>;
- /// Appends a FDT from /proc/device-tree style directory at the given node path
- fn append(&mut self, fdt_node_path: &CStr, fs_path: &Path) -> Result<()>;
+ /// Overlay an FDT from /proc/device-tree style directory at the given node path
+ fn overlay_onto(&mut self, fdt_node_path: &CStr, fs_path: &Path) -> Result<()>;
}
impl<'a> FsFdt<'a> for Fdt {
@@ -35,12 +35,12 @@
let fdt = Fdt::create_empty_tree(fdt_buffer)
.map_err(|e| anyhow!("Failed to create FDT, {e:?}"))?;
- fdt.append(&CString::new("").unwrap(), fs_path)?;
+ fdt.overlay_onto(&CString::new("").unwrap(), fs_path)?;
Ok(fdt)
}
- fn append(&mut self, fdt_node_path: &CStr, fs_path: &Path) -> Result<()> {
+ fn overlay_onto(&mut self, fdt_node_path: &CStr, fs_path: &Path) -> Result<()> {
// Recursively traverse fs_path with DFS algorithm.
let mut stack = vec![fs_path.to_path_buf()];
while let Some(dir_path) = stack.pop() {
@@ -56,7 +56,7 @@
let mut node = self
.node_mut(&fdt_path)
.map_err(|e| anyhow!("Failed to write FDT, {e:?}"))?
- .ok_or_else(|| anyhow!("Failed to find {fdt_node_path:?} in FDT"))?;
+ .ok_or_else(|| anyhow!("Failed to find {fdt_path:?} in FDT"))?;
let mut subnode_names = vec![];
let entries =
@@ -90,9 +90,21 @@
// FDT library may omit address in node name when comparing their name, so sort to add
// node without address first.
subnode_names.sort();
- let subnode_names_c_str: Vec<_> = subnode_names.iter().map(|x| x.as_c_str()).collect();
- node.add_subnodes(&subnode_names_c_str)
- .map_err(|e| anyhow!("Failed to add node, {e:?}"))?;
+ let subnode_names: Vec<_> = subnode_names
+ .iter()
+ .filter_map(|name| {
+ // Filter out subnode names which are already present in the target parent node!
+ let name = name.as_c_str();
+ let is_present_res = node.as_node().subnode(name);
+ match is_present_res {
+ Ok(Some(_)) => None,
+ Ok(None) => Some(Ok(name)),
+ Err(e) => Some(Err(e)),
+ }
+ })
+ .collect::<Result<_, _>>()
+ .map_err(|e| anyhow!("Failed to filter subnodes, {e:?}"))?;
+ node.add_subnodes(&subnode_names).map_err(|e| anyhow!("Failed to add node, {e:?}"))?;
}
Ok(())
@@ -149,5 +161,8 @@
let actual = dts_from_dtb(file.path());
assert_eq!(&expected, &actual);
+ // Again append fdt from TEST_FS_FDT_ROOT_PATH at root & ensure it succeeds when some
+ // subnode are already present.
+ fdt.overlay_onto(&CString::new("/").unwrap(), fs_path).unwrap();
}
}
diff --git a/virtualizationmanager/src/reference_dt.rs b/virtualizationmanager/src/reference_dt.rs
index 797ee3c..39b8db8 100644
--- a/virtualizationmanager/src/reference_dt.rs
+++ b/virtualizationmanager/src/reference_dt.rs
@@ -55,7 +55,7 @@
.add_subnode(cstr!("__overlay__"))
.map_err(|e| anyhow!("Failed to create the __overlay__, {e:?}"))?;
- fdt.append(cstr!("/fragment@0/__overlay__"), dir_path)?;
+ fdt.overlay_onto(cstr!("/fragment@0/__overlay__"), dir_path)?;
fdt.pack().map_err(|e| anyhow!("Failed to pack VM reference DT, {e:?}"))?;
fs::write(fdt_path, fdt.as_slice())?;