Merge "Add LOCAL_SDK_VERSION := system_$(VER)"
diff --git a/Changes.md b/Changes.md
new file mode 100644
index 0000000..53ff007
--- /dev/null
+++ b/Changes.md
@@ -0,0 +1,106 @@
+# Build System Changes for Android.mk Writers
+
+## Deprecating / obsoleting envsetup.sh variables in Makefiles
+
+It is not required to source envsetup.sh before running a build. Many scripts,
+including a majority of our automated build systems, do not do so. Make will
+transparently make every environment variable available as a make variable.
+This means that relying on environment variables only set up in envsetup.sh will
+produce different output for local users and scripted users.
+
+Many of these variables also include absolute path names, which we'd like to
+keep out of the generated files, so that you don't need to do a full rebuild if
+you move the source tree.
+
+To fix this, we're marking the variables that are set in envsetup.sh as
+deprecated in the makefiles. This will trigger a warning every time one is read
+(or written) inside Kati. Once all the warnings have been removed for a
+particular variable, we'll switch it to obsolete, and any references will become
+errors.
+
+### envsetup.sh variables with make equivalents
+
+| instead of | use |
+|--------------------------------------------------------------|----------------------|
+| OUT {#OUT} | OUT_DIR |
+| ANDROID_HOST_OUT {#ANDROID_HOST_OUT} | HOST_OUT |
+| ANDROID_PRODUCT_OUT {#ANDROID_PRODUCT_OUT} | PRODUCT_OUT |
+| ANDROID_HOST_OUT_TESTCASES {#ANDROID_HOST_OUT_TESTCASES} | HOST_OUT_TESTCASES |
+| ANDROID_TARGET_OUT_TESTCASES {#ANDROID_TARGET_OUT_TESTCASES} | TARGET_OUT_TESTCASES |
+
+All of the make variables may be relative paths from the current directory, or
+absolute paths if the output directory was specified as an absolute path. If you
+need an absolute variable, convert it to absolute during a rule, so that it's
+not expanded into the generated ninja file:
+
+``` make
+$(PRODUCT_OUT)/gen.img: my/src/path/gen.sh
+ export PRODUCT_OUT=$$(cd $(PRODUCT_OUT); pwd); cd my/src/path; ./gen.sh -o $${PRODUCT_OUT}/gen.img
+```
+
+### ANDROID_BUILD_TOP {#ANDROID_BUILD_TOP}
+
+In Android.mk files, you can always assume that the current directory is the
+root of the source tree, so this can just be replaced with '.' (which is what
+$TOP is hardcoded to), or removed entirely. If you need an absolute path, see
+the instructions above.
+
+### Stop using PATH directly {#PATH}
+
+This isn't only set by envsetup.sh, but it is modified by it. Due to that it's
+rather easy for this to change between different shells, and it's not ideal to
+reread the makefiles every time this changes.
+
+In most cases, you shouldn't need to touch PATH at all. When you need to have a
+rule reference a particular binary that's part of the source tree or outputs,
+it's preferrable to just use the path to the file itself (since you should
+already be adding that as a dependency).
+
+Depending on the rule, passing the file path itself may not be feasible due to
+layers of unchangable scripts/binaries. In that case, be sure to add the
+dependency, but modify the PATH within the rule itself:
+
+``` make
+$(TARGET): myscript my/path/binary
+ PATH=my/path:$$PATH myscript -o $@
+```
+
+### Stop using PYTHONPATH directly {#PYTHONPATH}
+
+Like PATH, this isn't only set by envsetup.sh, but it is modified by it. Due to
+that it's rather easy for this to change between different shells, and it's not
+ideal to reread the makefiles every time.
+
+The best solution here is to start switching to Soong's python building support,
+which packages the python interpreter, libraries, and script all into one file
+that no longer needs PYTHONPATH. See fontchain_lint for examples of this:
+
+* [external/fonttools/Lib/fontTools/Android.bp] for python_library_host
+* [frameworks/base/Android.bp] for python_binary_host
+* [frameworks/base/data/fonts/Android.mk] to execute the python binary
+
+If you still need to use PYTHONPATH, do so within the rule itself, just like
+path:
+
+``` make
+$(TARGET): myscript.py $(sort $(shell find my/python/lib -name '*.py'))
+ PYTHONPATH=my/python/lib:$$PYTHONPATH myscript.py -o $@
+```
+
+### Other envsetup.sh variables {#other_envsetup_variables}
+
+* ANDROID_TOOLCHAIN
+* ANDROID_TOOLCHAIN_2ND_ARCH
+* ANDROID_DEV_SCRIPTS
+* ANDROID_EMULATOR_PREBUILTS
+* ANDROID_PRE_BUILD_PATHS
+
+These are all exported from envsetup.sh, but don't have clear equivalents within
+the makefile system. If you need one of them, you'll have to set up your own
+version.
+
+
+[build/soong/Changes.md]: https://android.googlesource.com/platform/build/soong/+/master/Changes.md
+[external/fonttools/Lib/fontTools/Android.bp]: https://android.googlesource.com/platform/external/fonttools/+/master/Lib/fontTools/Android.bp
+[frameworks/base/Android.bp]: https://android.googlesource.com/platform/frameworks/base/+/master/Android.bp
+[frameworks/base/data/fonts/Android.mk]: https://android.googlesource.com/platform/frameworks/base/+/master/data/fonts/Android.mk
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..47809a9
--- /dev/null
+++ b/README.md
@@ -0,0 +1,23 @@
+# Android Make Build System
+
+This is the Makefile-based portion of the Android Build System.
+
+For documentation on how to run a build, see [Usage.txt](Usage.txt)
+
+For a list of behavioral changes useful for Android.mk writers see
+[Changes.md](Changes.md)
+
+For an outdated reference on Android.mk files, see
+[build-system.html](/core/build-system.html). Our Android.mk files look similar,
+but are entirely different from the Android.mk files used by the NDK build
+system. When searching for documentation elsewhere, ensure that it is for the
+platform build system -- most are not.
+
+This Makefile-based system is in the process of being replaced with [Soong], a
+new build system written in Go. During the transition, all of these makefiles
+are read by [Kati], and generate a ninja file instead of being executed
+directly. That's combined with a ninja file read by Soong so that the build
+graph of the two systems can be combined and run as one.
+
+[Kati]: https://github.com/google/kati
+[Soong]: https://android.googlesource.com/platform/build/soong/+/master
diff --git a/README.txt b/Usage.txt
similarity index 100%
rename from README.txt
rename to Usage.txt
diff --git a/core/Makefile b/core/Makefile
index 06fdf62..2e863a7 100644
--- a/core/Makefile
+++ b/core/Makefile
@@ -58,6 +58,19 @@
unique_product_copy_files_destinations :=
# -----------------------------------------------------------------
+# Returns the max allowed size for an image suitable for hash verification
+# (e.g., boot.img, recovery.img, etc).
+# The value 69632 derives from MAX_VBMETA_SIZE + MAX_FOOTER_SIZE in $(AVBTOOL).
+# $(1): partition size to flash the image
+define get-hash-image-max-size
+$(if $(1), \
+ $(if $(filter true,$(BOARD_AVB_ENABLE)), \
+ $(eval _hash_meta_size := 69632), \
+ $(eval _hash_meta_size := 0)) \
+ $(1)-$(_hash_meta_size))
+endef
+
+# -----------------------------------------------------------------
# Define rules to copy headers defined in copy_headers.mk
# If more than one makefile declared a header, print a warning,
# then copy the last one defined. This matches the previous make
@@ -686,7 +699,7 @@
$(INSTALLED_BOOTIMAGE_TARGET): $(MKBOOTIMG) $(AVBTOOL) $(INTERNAL_BOOTIMAGE_FILES) $(BOARD_AVB_BOOT_KEY_PATH)
$(call pretty,"Target boot image: $@")
$(hide) $(MKBOOTIMG) $(INTERNAL_BOOTIMAGE_ARGS) $(INTERNAL_MKBOOTIMG_VERSION_ARGS) $(BOARD_MKBOOTIMG_ARGS) --output $@
- $(hide) $(call assert-max-image-size,$@,$(BOARD_BOOTIMAGE_PARTITION_SIZE))
+ $(hide) $(call assert-max-image-size,$@,$(call get-hash-image-max-size,$(BOARD_BOOTIMAGE_PARTITION_SIZE)))
$(hide) $(AVBTOOL) add_hash_footer \
--image $@ \
--partition_size $(BOARD_BOOTIMAGE_PARTITION_SIZE) \
@@ -697,9 +710,9 @@
bootimage-nodeps: $(MKBOOTIMG) $(AVBTOOL) $(BOARD_AVB_BOOT_KEY_PATH)
@echo "make $@: ignoring dependencies"
$(hide) $(MKBOOTIMG) $(INTERNAL_BOOTIMAGE_ARGS) $(INTERNAL_MKBOOTIMG_VERSION_ARGS) $(BOARD_MKBOOTIMG_ARGS) --output $(INSTALLED_BOOTIMAGE_TARGET)
- $(hide) $(call assert-max-image-size,$(INSTALLED_BOOTIMAGE_TARGET),$(BOARD_BOOTIMAGE_PARTITION_SIZE))
+ $(hide) $(call assert-max-image-size,$(INSTALLED_BOOTIMAGE_TARGET),$(call get-hash-image-max-size,$(BOARD_BOOTIMAGE_PARTITION_SIZE)))
$(hide) $(AVBTOOL) add_hash_footer \
- --image $@ \
+ --image $(INSTALLED_BOOTIMAGE_TARGET) \
--partition_size $(BOARD_BOOTIMAGE_PARTITION_SIZE) \
--partition_name boot $(INTERNAL_AVB_BOOT_SIGNING_ARGS) \
$(BOARD_AVB_BOOT_ADD_HASH_FOOTER_ARGS)
@@ -1288,15 +1301,15 @@
)
$(if $(filter true,$(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_SUPPORTS_VBOOT)), \
$(VBOOT_SIGNER) $(FUTILITY) $(1).unsigned $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_VBOOT_SIGNING_KEY).vbpubk $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_VBOOT_SIGNING_KEY).vbprivk $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_VBOOT_SIGNING_SUBKEY).vbprivk $(1).keyblock $(1))
+ $(if $(filter true,$(BOARD_USES_RECOVERY_AS_BOOT)), \
+ $(hide) $(call assert-max-image-size,$(1),$(call get-hash-image-max-size,$(BOARD_BOOTIMAGE_PARTITION_SIZE))), \
+ $(hide) $(call assert-max-image-size,$(1),$(call get-hash-image-max-size,$(BOARD_RECOVERYIMAGE_PARTITION_SIZE))))
$(if $(and $(filter true,$(BOARD_USES_RECOVERY_AS_BOOT)),$(filter true,$(BOARD_AVB_ENABLE))), \
$(hide) $(AVBTOOL) add_hash_footer \
--image $(1) \
--partition_size $(BOARD_BOOTIMAGE_PARTITION_SIZE) \
--partition_name boot $(INTERNAL_AVB_BOOT_SIGNING_ARGS) \
$(BOARD_AVB_BOOT_ADD_HASH_FOOTER_ARGS))
- $(if $(filter true,$(BOARD_USES_RECOVERY_AS_BOOT)), \
- $(hide) $(call assert-max-image-size,$(1),$(BOARD_BOOTIMAGE_PARTITION_SIZE)), \
- $(hide) $(call assert-max-image-size,$(1),$(BOARD_RECOVERYIMAGE_PARTITION_SIZE)))
endef
ADBD := $(TARGET_OUT_EXECUTABLES)/adbd
diff --git a/core/base_rules.mk b/core/base_rules.mk
index 3663d51..50436d1 100644
--- a/core/base_rules.mk
+++ b/core/base_rules.mk
@@ -612,6 +612,7 @@
ALL_MODULES.$(my_register_name).FOR_2ND_ARCH := true
endif
ALL_MODULES.$(my_register_name).FOR_HOST_CROSS := $(my_host_cross)
+ALL_MODULES.$(my_register_name).COMPATIBILITY_SUITES := $(LOCAL_COMPATIBILITY_SUITE)
INSTALLABLE_FILES.$(LOCAL_INSTALLED_MODULE).MODULE := $(my_register_name)
diff --git a/core/config.mk b/core/config.mk
index e0c79d7..2f022a5 100644
--- a/core/config.mk
+++ b/core/config.mk
@@ -59,7 +59,24 @@
.DELETE_ON_ERROR:
# Mark variables deprecated/obsolete
-$(KATI_deprecated_var PATH,Do not use PATH directly)
+CHANGES_URL := https://android.googlesource.com/platform/build/+/master/Changes.md
+$(KATI_deprecated_var PATH,Do not use PATH directly. See $(CHANGES_URL)#PATH)
+$(KATI_obsolete_var PYTHONPATH,Do not use PYTHONPATH directly. See $(CHANGES_URL)#PYTHONPATH)
+$(KATI_obsolete_var OUT,Use OUT_DIR instead. See $(CHANGES_URL)#OUT)
+$(KATI_obsolete_var ANDROID_HOST_OUT,Use HOST_OUT instead. See $(CHANGES_URL)#ANDROID_HOST_OUT)
+$(KATI_deprecated_var ANDROID_PRODUCT_OUT,Use PRODUCT_OUT instead. See $(CHANGES_URL)#ANDROID_PRODUCT_OUT)
+$(KATI_obsolete_var ANDROID_HOST_OUT_TESTCASES,Use HOST_OUT_TESTCASES instead. See $(CHANGES_URL)#ANDROID_HOST_OUT_TESTCASES)
+$(KATI_obsolete_var ANDROID_TARGET_OUT_TESTCASES,Use TARGET_OUT_TESTCASES instead. See $(CHANGES_URL)#ANDROID_TARGET_OUT_TESTCASES)
+$(KATI_deprecated_var ANDROID_BUILD_TOP,Use '.' instead. See $(CHANGES_URL)#ANDROID_BUILD_TOP)
+$(KATI_obsolete_var \
+ ANDROID_TOOLCHAIN \
+ ANDROID_TOOLCHAIN_2ND_ARCH \
+ ANDROID_DEV_SCRIPTS \
+ ANDROID_EMULATOR_PREBUILTS \
+ ANDROID_PRE_BUILD_PATHS \
+ ,See $(CHANGES_URL)#other_envsetup_variables)
+
+CHANGES_URL :=
# Used to force goals to build. Only use for conditionally defined goals.
.PHONY: FORCE
diff --git a/core/product.mk b/core/product.mk
index c01a856..f15f6b3 100644
--- a/core/product.mk
+++ b/core/product.mk
@@ -102,6 +102,7 @@
PRODUCT_SDK_ADDON_COPY_MODULES \
PRODUCT_SDK_ADDON_DOC_MODULES \
PRODUCT_SDK_ADDON_SYS_IMG_SOURCE_PROP \
+ PRODUCT_SOONG_NAMESPACES \
PRODUCT_DEFAULT_WIFI_CHANNELS \
PRODUCT_DEFAULT_DEV_CERTIFICATE \
PRODUCT_RESTRICT_VENDOR_FILES \
diff --git a/core/product_config.mk b/core/product_config.mk
index 4e2d5ae..5b0e257 100644
--- a/core/product_config.mk
+++ b/core/product_config.mk
@@ -480,3 +480,7 @@
# Whether any paths should have CFI enabled for components
PRODUCT_CFI_INCLUDE_PATHS := \
$(strip $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_CFI_INCLUDE_PATHS))
+
+# which Soong namespaces to export to Make
+PRODUCT_SOONG_NAMESPACES :=
+ $(strip $(PRODUCTS.$(INTERNAL_PRODUCT).PRODUCT_SOONG_NAMESPACES))
diff --git a/core/soong_app_prebuilt.mk b/core/soong_app_prebuilt.mk
new file mode 100644
index 0000000..ae0b982
--- /dev/null
+++ b/core/soong_app_prebuilt.mk
@@ -0,0 +1,68 @@
+# App prebuilt coming from Soong.
+# Extra inputs:
+# LOCAL_SOONG_RESOURCE_EXPORT_PACKAGE
+
+ifneq ($(LOCAL_MODULE_MAKEFILE),$(SOONG_ANDROID_MK))
+ $(call pretty-error,soong_app_prebuilt.mk may only be used from Soong)
+endif
+
+LOCAL_MODULE_SUFFIX := .apk
+LOCAL_BUILT_MODULE_STEM := package.apk
+
+#######################################
+include $(BUILD_SYSTEM)/base_rules.mk
+#######################################
+
+ifdef LOCAL_SOONG_JACOCO_REPORT_CLASSES_JAR
+ $(eval $(call copy-one-file,$(LOCAL_SOONG_JACOCO_REPORT_CLASSES_JAR),\
+ $(intermediates.COMMON)/jacoco-report-classes.jar))
+endif
+
+$(eval $(call copy-one-file,$(LOCAL_PREBUILT_MODULE_FILE),$(LOCAL_BUILT_MODULE)))
+
+ifdef LOCAL_SOONG_RESOURCE_EXPORT_PACKAGE
+resource_export_package := $(intermediates.COMMON)/package-export.apk
+resource_export_stamp := $(intermediates.COMMON)/src/R.stamp
+
+$(resource_export_package): PRIVATE_STAMP := $(resource_export_stamp)
+$(resource_export_package): .KATI_IMPLICIT_OUTPUTS := $(resource_export_stamp)
+$(resource_export_package): $(LOCAL_SOONG_RESOURCE_EXPORT_PACKAGE)
+ @echo "Copy: $$@"
+ $(copy-file-to-target)
+ touch $(PRIVATE_STAMP)
+
+endif # LOCAL_SOONG_RESOURCE_EXPORT_PACKAGE
+
+java-dex: $(LOCAL_SOONG_DEX_JAR)
+
+ifdef LOCAL_DEX_PREOPT
+# defines built_odex along with rule to install odex
+include $(BUILD_SYSTEM)/dex_preopt_odex_install.mk
+
+$(built_odex): $(LOCAL_SOONG_DEX_JAR)
+ $(call dexpreopt-one-file,$<,$@)
+endif
+
+ifndef LOCAL_IS_HOST_MODULE
+ifeq ($(LOCAL_SDK_VERSION),system_current)
+my_link_type := java:system
+my_warn_types := java:platform
+my_allowed_types := java:sdk java:system
+else ifneq ($(LOCAL_SDK_VERSION),)
+my_link_type := java:sdk
+my_warn_types := java:system java:platform
+my_allowed_types := java:sdk
+else
+my_link_type := java:platform
+my_warn_types :=
+my_allowed_types := java:sdk java:system java:platform
+endif
+
+my_link_deps :=
+my_2nd_arch_prefix := $(LOCAL_2ND_ARCH_VAR_PREFIX)
+my_common := COMMON
+include $(BUILD_SYSTEM)/link_type.mk
+endif # !LOCAL_IS_HOST_MODULE
+
+# Built in equivalent to include $(CLEAR_VARS)
+LOCAL_SOONG_RESOURCE_EXPORT_PACKAGE :=
diff --git a/core/soong_config.mk b/core/soong_config.mk
index a90e5af..32e9ae0 100644
--- a/core/soong_config.mk
+++ b/core/soong_config.mk
@@ -118,9 +118,12 @@
$(call add_json_bool, MinimizeJavaDebugInfo, $(filter true,$(PRODUCT_MINIMIZE_JAVA_DEBUG_INFO)))
$(call add_json_bool, UseGoma, $(filter-out false,$(USE_GOMA)))
+$(call add_json_bool, Arc, $(filter true,$(TARGET_ARC)))
$(call add_json_str, DistDir, $(if $(dist_goal), $(DIST_DIR)))
+$(call add_json_list, NamespacesToExport, $(PRODUCT_SOONG_NAMESPACES))
+
_contents := $(subst $(comma)$(newline)__SV_END,$(newline)}$(newline),$(_contents)__SV_END)
$(file >$(SOONG_VARIABLES).tmp,$(_contents))
diff --git a/core/soong_java_prebuilt.mk b/core/soong_java_prebuilt.mk
index 14fe73e..e7f5fa5 100644
--- a/core/soong_java_prebuilt.mk
+++ b/core/soong_java_prebuilt.mk
@@ -16,10 +16,12 @@
#######################################
full_classes_jar := $(intermediates.COMMON)/classes.jar
+full_classes_pre_proguard_jar := $(intermediates.COMMON)/classes-pre-proguard.jar
full_classes_header_jar := $(intermediates.COMMON)/classes-header.jar
common_javalib.jar := $(intermediates.COMMON)/javalib.jar
$(eval $(call copy-one-file,$(LOCAL_PREBUILT_MODULE_FILE),$(full_classes_jar)))
+$(eval $(call copy-one-file,$(LOCAL_PREBUILT_MODULE_FILE),$(full_classes_pre_proguard_jar)))
ifdef LOCAL_SOONG_JACOCO_REPORT_CLASSES_JAR
$(eval $(call copy-one-file,$(LOCAL_SOONG_JACOCO_REPORT_CLASSES_JAR),\
diff --git a/core/tasks/module-info.mk b/core/tasks/module-info.mk
index e9b2ac7..f6d688c 100644
--- a/core/tasks/module-info.mk
+++ b/core/tasks/module-info.mk
@@ -11,6 +11,7 @@
'"path": [$(foreach w,$(sort $(ALL_MODULES.$(m).PATH)),"$(w)", )], ' \
'"tags": [$(foreach w,$(sort $(ALL_MODULES.$(m).TAGS)),"$(w)", )], ' \
'"installed": [$(foreach w,$(sort $(ALL_MODULES.$(m).INSTALLED)),"$(w)", )], ' \
+ '"compatibility_suites": [$(foreach w,$(sort $(ALL_MODULES.$(m).COMPATIBILITY_SUITES)),"$(w)", )], ' \
'},\n' \
) | sed -e 's/, *\]/]/g' -e 's/, *\}/ }/g' -e '$$s/,$$//' >> $@
$(hide) echo '}' >> $@
diff --git a/core/tasks/tools/compatibility.mk b/core/tasks/tools/compatibility.mk
index 0fc2045..41ee76f 100644
--- a/core/tasks/tools/compatibility.mk
+++ b/core/tasks/tools/compatibility.mk
@@ -21,6 +21,8 @@
# test_suite_readme: the path to a README file for this test suite
# test_suite_prebuilt_tools: the set of prebuilt tools to be included directly
# in the 'tools' subdirectory of the test suite.
+# test_suite_tools: the set of tools for this test suite
+#
# Output variables:
# compatibility_zip: the path to the output zip file.
@@ -38,6 +40,8 @@
$(HOST_OUT_EXECUTABLES)/$(test_suite_tradefed) \
$(test_suite_readme)
+test_tools += $(test_suite_tools)
+
compatibility_zip := $(out_dir).zip
$(compatibility_zip): PRIVATE_NAME := android-$(test_suite_name)
$(compatibility_zip): PRIVATE_OUT_DIR := $(out_dir)
@@ -59,3 +63,4 @@
test_suite_dynamic_config :=
test_suite_readme :=
test_suite_prebuilt_tools :=
+test_suite_tools :=
diff --git a/help.sh b/help.sh
index c031dcc..3f39c77 100755
--- a/help.sh
+++ b/help.sh
@@ -15,7 +15,7 @@
m -j [<goals>] # Execute the configured build.
Usage of "m" imitates usage of the program "make".
-See '"${SCRIPT_DIR}"'/README.txt for more info about build usage and concepts.
+See '"${SCRIPT_DIR}"'/Usage.txt for more info about build usage and concepts.
Common goals are:
diff --git a/navbar.md b/navbar.md
new file mode 100644
index 0000000..e218a78
--- /dev/null
+++ b/navbar.md
@@ -0,0 +1,4 @@
+* [Home](/README.md)
+* [Usage](/Usage.txt)
+* [Changes](/Changes.md)
+* [Outdated Reference](/core/build-system.html)
diff --git a/target/board/treble_common.mk b/target/board/treble_common.mk
index 1ba6726..0296d71 100644
--- a/target/board/treble_common.mk
+++ b/target/board/treble_common.mk
@@ -19,7 +19,6 @@
# VNDK
BOARD_VNDK_VERSION := current
-BOARD_VNDK_RUNTIME_DISABLE := true
# Properties
TARGET_SYSTEM_PROP := build/make/target/board/treble_system.prop
diff --git a/tools/releasetools/blockimgdiff.py b/tools/releasetools/blockimgdiff.py
index 1ef55ff..8f06b95 100644
--- a/tools/releasetools/blockimgdiff.py
+++ b/tools/releasetools/blockimgdiff.py
@@ -1178,9 +1178,22 @@
def FindTransfers(self):
"""Parse the file_map to generate all the transfers."""
- def AddSplitTransfers(tgt_name, src_name, tgt_ranges, src_ranges, style,
- by_id):
- """Add one or multiple Transfer()s by splitting large files."""
+ def AddSplitTransfersWithFixedSizeChunks(tgt_name, src_name, tgt_ranges,
+ src_ranges, style, by_id):
+ """Add one or multiple Transfer()s by splitting large files.
+
+ For BBOTA v3, we need to stash source blocks for resumable feature.
+ However, with the growth of file size and the shrink of the cache
+ partition source blocks are too large to be stashed. If a file occupies
+ too many blocks, we split it into smaller pieces by getting multiple
+ Transfer()s.
+
+ The downside is that after splitting, we may increase the package size
+ since the split pieces don't align well. According to our experiments,
+ 1/8 of the cache size as the per-piece limit appears to be optimal.
+ Compared to the fixed 1024-block limit, it reduces the overall package
+ size by 30% for volantis, and 20% for angler and bullhead."""
+
pieces = 0
while (tgt_ranges.size() > max_blocks_per_transfer and
src_ranges.size() > max_blocks_per_transfer):
@@ -1207,21 +1220,15 @@
self.tgt.RangeSha1(tgt_ranges), self.src.RangeSha1(src_ranges),
style, by_id)
- def FindZipsAndAddSplitTransfers(tgt_name, src_name, tgt_ranges,
- src_ranges, style, by_id):
- """Find all the zip archives and add split transfers for the other files.
+ def AddSplitTransfers(tgt_name, src_name, tgt_ranges, src_ranges, style,
+ by_id):
+ """Find all the zip files and split the others with a fixed chunk size.
- For BBOTA v3, we need to stash source blocks for resumable feature.
- However, with the growth of file size and the shrink of the cache
- partition source blocks are too large to be stashed. If a file occupies
- too many blocks, we split it into smaller pieces by getting multiple
- Transfer()s.
-
- The downside is that after splitting, we may increase the package size
- since the split pieces don't align well. According to our experiments,
- 1/8 of the cache size as the per-piece limit appears to be optimal.
- Compared to the fixed 1024-block limit, it reduces the overall package
- size by 30% for volantis, and 20% for angler and bullhead."""
+ This function will construct a list of zip archives, which will later be
+ split by imgdiff to reduce the final patch size. For the other files,
+ we will plainly split them based on a fixed chunk size with the potential
+ patch size penalty.
+ """
assert style == "diff"
@@ -1241,8 +1248,8 @@
large_apks.append((tgt_name, src_name, tgt_ranges, src_ranges))
return
- AddSplitTransfers(tgt_name, src_name, tgt_ranges, src_ranges,
- style, by_id)
+ AddSplitTransfersWithFixedSizeChunks(tgt_name, src_name, tgt_ranges,
+ src_ranges, style, by_id)
def AddTransfer(tgt_name, src_name, tgt_ranges, src_ranges, style, by_id,
split=False):
@@ -1292,7 +1299,7 @@
assert tgt_changed + tgt_skipped.size() == tgt_size
print('%10d %10d (%6.2f%%) %s' % (tgt_skipped.size(), tgt_size,
tgt_skipped.size() * 100.0 / tgt_size, tgt_name))
- FindZipsAndAddSplitTransfers(
+ AddSplitTransfers(
"%s-skipped" % (tgt_name,),
"%s-skipped" % (src_name,),
tgt_skipped, src_skipped, style, by_id)
@@ -1309,7 +1316,7 @@
return
# Add the transfer(s).
- FindZipsAndAddSplitTransfers(
+ AddSplitTransfers(
tgt_name, src_name, tgt_ranges, src_ranges, style, by_id)
def ParseAndValidateSplitInfo(patch_size, tgt_ranges, src_ranges,
@@ -1384,6 +1391,11 @@
be valid because the block ranges of src-X & tgt-X will always stay the
same afterwards; but there's a chance we don't use the patch if we
convert the "diff" command into "new" or "move" later.
+
+ The split will be attempted by calling imgdiff, which expects the input
+ files to be valid zip archives. If imgdiff fails for some reason (i.e.
+ holes in the APK file), we will fall back to split the failed APKs into
+ fixed size chunks.
"""
while True:
@@ -1412,8 +1424,9 @@
print("Failed to create patch between {} and {},"
" falling back to bsdiff".format(src_name, tgt_name))
with transfer_lock:
- AddSplitTransfers(tgt_name, src_name, tgt_ranges, src_ranges,
- "diff", self.transfers)
+ AddSplitTransfersWithFixedSizeChunks(tgt_name, src_name,
+ tgt_ranges, src_ranges,
+ "diff", self.transfers)
continue
with open(patch_info_file) as patch_info: