blob: 8f93688791914719f741e79cafd361966aead26e [file] [log] [blame]
Bill Peckhame9eb5f92019-02-01 15:52:10 -08001#!/usr/bin/env python
2#
Daniel Norman03747412022-02-25 10:38:37 -08003# Copyright (C) 2022 The Android Open Source Project
Bill Peckhame9eb5f92019-02-01 15:52:10 -08004#
5# Licensed under the Apache License, Version 2.0 (the "License"); you may not
6# use this file except in compliance with the License. You may obtain a copy of
7# the License at
8#
9# http://www.apache.org/licenses/LICENSE-2.0
10#
11# Unless required by applicable law or agreed to in writing, software
12# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
13# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
14# License for the specific language governing permissions and limitations under
15# the License.
Daniel Norman4cc9df62019-07-18 10:11:07 -070016#
17"""This script merges two partial target files packages.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080018
Daniel Normandbbf5a32020-10-22 16:03:32 -070019One input package contains framework files, and the other contains vendor files.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080020
Daniel Normandbbf5a32020-10-22 16:03:32 -070021This script produces a complete, merged target files package:
22 - This package can be used to generate a flashable IMG package.
23 See --output-img.
24 - This package can be used to generate an OTA package. See --output-ota.
25 - The merged package is checked for compatibility between the two inputs.
26
27Usage: merge_target_files [args]
Bill Peckhame9eb5f92019-02-01 15:52:10 -080028
Dennis Song5bfa43e2023-03-30 18:28:00 +080029 --framework-target-files framework-target-files-package
Daniel Normand5d70ea2019-06-05 15:13:43 -070030 The input target files package containing framework bits. This is a zip
Dennis Song5bfa43e2023-03-30 18:28:00 +080031 archive or a directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080032
Daniel Normand5d70ea2019-06-05 15:13:43 -070033 --framework-item-list framework-item-list-file
Daniel Norman5f476772022-03-02 15:46:34 -080034 The optional path to a newline-separated config file of items that
35 are extracted as-is from the framework target files package.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080036
Daniel Normand5d70ea2019-06-05 15:13:43 -070037 --framework-misc-info-keys framework-misc-info-keys-file
Daniel Norman5f476772022-03-02 15:46:34 -080038 The optional path to a newline-separated config file of keys to
39 extract from the framework META/misc_info.txt file.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080040
Dennis Song5bfa43e2023-03-30 18:28:00 +080041 --vendor-target-files vendor-target-files-package
Daniel Normand5d70ea2019-06-05 15:13:43 -070042 The input target files package containing vendor bits. This is a zip
Dennis Song5bfa43e2023-03-30 18:28:00 +080043 archive or a directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080044
Daniel Normand5d70ea2019-06-05 15:13:43 -070045 --vendor-item-list vendor-item-list-file
Daniel Norman5f476772022-03-02 15:46:34 -080046 The optional path to a newline-separated config file of items that
47 are extracted as-is from the vendor target files package.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080048
Bill Peckhame9eb5f92019-02-01 15:52:10 -080049 --output-target-files output-target-files-package
Daniel Normanfdb38812019-04-15 09:47:24 -070050 If provided, the output merged target files package. Also a zip archive.
51
52 --output-dir output-directory
53 If provided, the destination directory for saving merged files. Requires
54 the --output-item-list flag.
55 Can be provided alongside --output-target-files, or by itself.
56
57 --output-item-list output-item-list-file.
58 The optional path to a newline-separated config file that specifies the
59 file patterns to copy into the --output-dir. Required if providing
60 the --output-dir flag.
Daniel Normana4911da2019-03-15 14:36:21 -070061
Daniel Norman3b64ce12019-04-16 16:11:35 -070062 --output-ota output-ota-package
63 The output ota package. This is a zip archive. Use of this flag may
64 require passing the --path common flag; see common.py.
65
Daniel Norman1bd2a1d2019-04-18 12:32:18 -070066 --output-img output-img-package
67 The output img package, suitable for use with 'fastboot update'. Use of
68 this flag may require passing the --path common flag; see common.py.
69
Daniel Normanf0318252019-04-15 11:34:56 -070070 --output-super-empty output-super-empty-image
71 If provided, creates a super_empty.img file from the merged target
72 files package and saves it at this path.
73
Daniel Normana4911da2019-03-15 14:36:21 -070074 --rebuild_recovery
Po Hua6c59122022-02-16 08:41:29 +000075 Copy the recovery image used by non-A/B devices, used when
76 regenerating vendor images with --rebuild-sepolicy.
Bill Peckham364c1cc2019-03-29 18:27:23 -070077
Daniel Normanb0c75912020-09-24 14:30:21 -070078 --allow-duplicate-apkapex-keys
79 If provided, duplicate APK/APEX keys are ignored and the value from the
80 framework is used.
81
Daniel Norman571e1822021-06-25 17:18:25 -070082 --rebuild-sepolicy
83 If provided, rebuilds odm.img or vendor.img to include merged sepolicy
84 files. If odm is present then odm is preferred.
85
86 --vendor-otatools otatools.zip
87 If provided, use this otatools.zip when recompiling the odm or vendor
88 image to include sepolicy.
89
Bill Peckham364c1cc2019-03-29 18:27:23 -070090 --keep-tmp
91 Keep tempoary files for debugging purposes.
Jose Galmes9c8f6eb2021-07-21 09:34:08 -070092
93 The following only apply when using the VSDK to perform dexopt on vendor apps:
94
95 --framework-dexpreopt-config
96 If provided, the location of framwework's dexpreopt_config.zip.
97
98 --framework-dexpreopt-tools
99 if provided, the location of framework's dexpreopt_tools.zip.
100
101 --vendor-dexpreopt-config
102 If provided, the location of vendor's dexpreopt_config.zip.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800103"""
104
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800105import logging
106import os
Daniel Normanfdb38812019-04-15 09:47:24 -0700107import shutil
Bill Peckham540d91a2019-04-25 14:18:16 -0700108import subprocess
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800109import sys
Daniel Norman5f476772022-03-02 15:46:34 -0800110import zipfile
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800111
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800112import add_img_to_target_files
Daniel Normandb8cacc2021-04-09 15:34:43 -0700113import build_image
Daniel Normanf0318252019-04-15 11:34:56 -0700114import build_super_image
115import common
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700116import img_from_target_files
Daniel Norman2465fc82022-03-02 12:01:20 -0800117import merge_compatibility_checks
118import merge_dexopt
119import merge_meta
120import merge_utils
Daniel Norman3b64ce12019-04-16 16:11:35 -0700121import ota_from_target_files
Daniel Normandb8cacc2021-04-09 15:34:43 -0700122
Daniel Normana84d13b2022-02-17 14:16:40 -0800123from common import ExternalError
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800124
125logger = logging.getLogger(__name__)
Tao Bao2ad4b822019-06-27 16:52:12 -0700126
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800127OPTIONS = common.OPTIONS
Bill Peckhamcb848172020-04-03 12:50:47 -0700128# Always turn on verbose logging.
129OPTIONS.verbose = True
Daniel Normand5d70ea2019-06-05 15:13:43 -0700130OPTIONS.framework_target_files = None
Daniel Norman5f476772022-03-02 15:46:34 -0800131OPTIONS.framework_item_list = []
132OPTIONS.framework_misc_info_keys = []
Daniel Normand5d70ea2019-06-05 15:13:43 -0700133OPTIONS.vendor_target_files = None
Daniel Norman5f476772022-03-02 15:46:34 -0800134OPTIONS.vendor_item_list = []
Bill Peckhamf753e152019-02-19 18:02:46 -0800135OPTIONS.output_target_files = None
Daniel Normanfdb38812019-04-15 09:47:24 -0700136OPTIONS.output_dir = None
Daniel Norman5f476772022-03-02 15:46:34 -0800137OPTIONS.output_item_list = []
Daniel Norman3b64ce12019-04-16 16:11:35 -0700138OPTIONS.output_ota = None
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700139OPTIONS.output_img = None
Daniel Normanf0318252019-04-15 11:34:56 -0700140OPTIONS.output_super_empty = None
Daniel Normana4911da2019-03-15 14:36:21 -0700141OPTIONS.rebuild_recovery = False
Daniel Normanb0c75912020-09-24 14:30:21 -0700142# TODO(b/150582573): Remove this option.
143OPTIONS.allow_duplicate_apkapex_keys = False
Daniel Norman571e1822021-06-25 17:18:25 -0700144OPTIONS.vendor_otatools = None
145OPTIONS.rebuild_sepolicy = False
Bill Peckhamf753e152019-02-19 18:02:46 -0800146OPTIONS.keep_tmp = False
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700147OPTIONS.framework_dexpreopt_config = None
148OPTIONS.framework_dexpreopt_tools = None
149OPTIONS.vendor_dexpreopt_config = None
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800150
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800151
jiangxu52bd42732022-04-11 16:02:48 +0800152def move_only_exists(source, destination):
153 """Judge whether the file exists and then move the file."""
154
155 if os.path.exists(source):
156 shutil.move(source, destination)
157
158
Dennis Song16d6fe02023-02-23 23:37:42 +0800159def remove_file_if_exists(file_name):
160 """Remove the file if it exists and skip otherwise."""
161
162 try:
163 os.remove(file_name)
164 except FileNotFoundError:
165 pass
166
167
Daniel Norman03747412022-02-25 10:38:37 -0800168def create_merged_package(temp_dir):
Tao Bao2ad4b822019-06-27 16:52:12 -0700169 """Merges two target files packages into one target files structure.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800170
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900171 Returns:
172 Path to merged package under temp directory.
173 """
Daniel Normandbbf5a32020-10-22 16:03:32 -0700174 # Extract "as is" items from the input framework and vendor partial target
Dennis Song5bfa43e2023-03-30 18:28:00 +0800175 # files packages directly into the output temporary directory, since these
176 # items do not need special case processing.
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800177
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800178 output_target_files_temp_dir = os.path.join(temp_dir, 'output')
Dennis Song5bfa43e2023-03-30 18:28:00 +0800179 merge_utils.CollectTargetFiles(
180 input_zipfile_or_dir=OPTIONS.framework_target_files,
Daniel Norman03747412022-02-25 10:38:37 -0800181 output_dir=output_target_files_temp_dir,
Dennis Song5bfa43e2023-03-30 18:28:00 +0800182 item_list=OPTIONS.framework_item_list)
183 merge_utils.CollectTargetFiles(
184 input_zipfile_or_dir=OPTIONS.vendor_target_files,
Daniel Norman03747412022-02-25 10:38:37 -0800185 output_dir=output_target_files_temp_dir,
Dennis Song5bfa43e2023-03-30 18:28:00 +0800186 item_list=OPTIONS.vendor_item_list)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800187
Daniel Normandbbf5a32020-10-22 16:03:32 -0700188 # Perform special case processing on META/* items.
189 # After this function completes successfully, all the files we need to create
190 # the output target files package are in place.
Daniel Norman2465fc82022-03-02 12:01:20 -0800191 merge_meta.MergeMetaFiles(
192 temp_dir=temp_dir, merged_dir=output_target_files_temp_dir)
Daniel Norman03747412022-02-25 10:38:37 -0800193
Daniel Norman2465fc82022-03-02 12:01:20 -0800194 merge_dexopt.MergeDexopt(
Daniel Norman03747412022-02-25 10:38:37 -0800195 temp_dir=temp_dir, output_target_files_dir=output_target_files_temp_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800196
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900197 return output_target_files_temp_dir
198
199
Daniel Norman03747412022-02-25 10:38:37 -0800200def generate_missing_images(target_files_dir):
201 """Generate any missing images from target files."""
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900202
203 # Regenerate IMAGES in the target directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800204
Daniel Normandbbf5a32020-10-22 16:03:32 -0700205 add_img_args = [
206 '--verbose',
207 '--add_missing',
208 ]
Daniel Norman03747412022-02-25 10:38:37 -0800209 if OPTIONS.rebuild_recovery:
Daniel Normana4911da2019-03-15 14:36:21 -0700210 add_img_args.append('--rebuild_recovery')
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900211 add_img_args.append(target_files_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800212
213 add_img_to_target_files.main(add_img_args)
214
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900215
Daniel Norman03747412022-02-25 10:38:37 -0800216def rebuild_image_with_sepolicy(target_files_dir):
Daniel Norman571e1822021-06-25 17:18:25 -0700217 """Rebuilds odm.img or vendor.img to include merged sepolicy files.
218
219 If odm is present then odm is preferred -- otherwise vendor is used.
Daniel Norman571e1822021-06-25 17:18:25 -0700220 """
221 partition = 'vendor'
Dennis Song16d6fe02023-02-23 23:37:42 +0800222 if os.path.exists(os.path.join(target_files_dir, 'ODM')):
Daniel Norman571e1822021-06-25 17:18:25 -0700223 partition = 'odm'
224 partition_img = '{}.img'.format(partition)
Po Hua6c59122022-02-16 08:41:29 +0000225 partition_map = '{}.map'.format(partition)
Daniel Norman571e1822021-06-25 17:18:25 -0700226
227 logger.info('Recompiling %s using the merged sepolicy files.', partition_img)
228
229 # Copy the combined SEPolicy file and framework hashes to the image that is
230 # being rebuilt.
231 def copy_selinux_file(input_path, output_filename):
Po Hu0e4403e2021-07-06 17:05:56 +0800232 input_filename = os.path.join(target_files_dir, input_path)
233 if not os.path.exists(input_filename):
Dennis Song5bfa43e2023-03-30 18:28:00 +0800234 input_filename = input_filename.replace('SYSTEM_EXT/',
235 'SYSTEM/system_ext/') \
Po Hu0e4403e2021-07-06 17:05:56 +0800236 .replace('PRODUCT/', 'SYSTEM/product/')
237 if not os.path.exists(input_filename):
238 logger.info('Skipping copy_selinux_file for %s', input_filename)
239 return
Daniel Norman571e1822021-06-25 17:18:25 -0700240 shutil.copy(
Po Hu0e4403e2021-07-06 17:05:56 +0800241 input_filename,
Daniel Norman571e1822021-06-25 17:18:25 -0700242 os.path.join(target_files_dir, partition.upper(), 'etc/selinux',
243 output_filename))
244
245 copy_selinux_file('META/combined_sepolicy', 'precompiled_sepolicy')
246 copy_selinux_file('SYSTEM/etc/selinux/plat_sepolicy_and_mapping.sha256',
247 'precompiled_sepolicy.plat_sepolicy_and_mapping.sha256')
248 copy_selinux_file(
249 'SYSTEM_EXT/etc/selinux/system_ext_sepolicy_and_mapping.sha256',
250 'precompiled_sepolicy.system_ext_sepolicy_and_mapping.sha256')
251 copy_selinux_file('PRODUCT/etc/selinux/product_sepolicy_and_mapping.sha256',
252 'precompiled_sepolicy.product_sepolicy_and_mapping.sha256')
253
Daniel Norman03747412022-02-25 10:38:37 -0800254 if not OPTIONS.vendor_otatools:
Daniel Norman571e1822021-06-25 17:18:25 -0700255 # Remove the partition from the merged target-files archive. It will be
Daniel Norman03747412022-02-25 10:38:37 -0800256 # rebuilt later automatically by generate_missing_images().
Dennis Song16d6fe02023-02-23 23:37:42 +0800257 remove_file_if_exists(
258 os.path.join(target_files_dir, 'IMAGES', partition_img))
Daniel Norman03747412022-02-25 10:38:37 -0800259 return
Daniel Norman571e1822021-06-25 17:18:25 -0700260
Daniel Norman03747412022-02-25 10:38:37 -0800261 # TODO(b/192253131): Remove the need for vendor_otatools by fixing
262 # backwards-compatibility issues when compiling images across releases.
263 if not OPTIONS.vendor_target_files:
264 raise ValueError(
265 'Expected vendor_target_files if vendor_otatools is not None.')
266 logger.info(
267 '%s recompilation will be performed using the vendor otatools.zip',
268 partition_img)
Daniel Norman571e1822021-06-25 17:18:25 -0700269
Daniel Norman03747412022-02-25 10:38:37 -0800270 # Unzip the vendor build's otatools.zip and target-files archive.
271 vendor_otatools_dir = common.MakeTempDir(
272 prefix='merge_target_files_vendor_otatools_')
273 vendor_target_files_dir = common.MakeTempDir(
274 prefix='merge_target_files_vendor_target_files_')
275 common.UnzipToDir(OPTIONS.vendor_otatools, vendor_otatools_dir)
Dennis Song5bfa43e2023-03-30 18:28:00 +0800276 merge_utils.CollectTargetFiles(
277 input_zipfile_or_dir=OPTIONS.vendor_target_files,
278 output_dir=vendor_target_files_dir,
279 item_list=OPTIONS.vendor_item_list)
Daniel Norman571e1822021-06-25 17:18:25 -0700280
Daniel Norman03747412022-02-25 10:38:37 -0800281 # Copy the partition contents from the merged target-files archive to the
282 # vendor target-files archive.
283 shutil.rmtree(os.path.join(vendor_target_files_dir, partition.upper()))
284 shutil.copytree(
285 os.path.join(target_files_dir, partition.upper()),
286 os.path.join(vendor_target_files_dir, partition.upper()),
287 symlinks=True)
Daniel Norman571e1822021-06-25 17:18:25 -0700288
Daniel Norman03747412022-02-25 10:38:37 -0800289 # Delete then rebuild the partition.
Dennis Song16d6fe02023-02-23 23:37:42 +0800290 remove_file_if_exists(
291 os.path.join(vendor_target_files_dir, 'IMAGES', partition_img))
Daniel Norman03747412022-02-25 10:38:37 -0800292 rebuild_partition_command = [
293 os.path.join(vendor_otatools_dir, 'bin', 'add_img_to_target_files'),
294 '--verbose',
295 '--add_missing',
296 ]
297 if OPTIONS.rebuild_recovery:
298 rebuild_partition_command.append('--rebuild_recovery')
299 rebuild_partition_command.append(vendor_target_files_dir)
300 logger.info('Recompiling %s: %s', partition_img,
301 ' '.join(rebuild_partition_command))
302 common.RunAndCheckOutput(rebuild_partition_command, verbose=True)
Po Hua6c59122022-02-16 08:41:29 +0000303
Daniel Norman03747412022-02-25 10:38:37 -0800304 # Move the newly-created image to the merged target files dir.
305 if not os.path.exists(os.path.join(target_files_dir, 'IMAGES')):
306 os.makedirs(os.path.join(target_files_dir, 'IMAGES'))
307 shutil.move(
308 os.path.join(vendor_target_files_dir, 'IMAGES', partition_img),
309 os.path.join(target_files_dir, 'IMAGES', partition_img))
Dennis Song5bfa43e2023-03-30 18:28:00 +0800310 move_only_exists(
311 os.path.join(vendor_target_files_dir, 'IMAGES', partition_map),
312 os.path.join(target_files_dir, 'IMAGES', partition_map))
Po Hua6c59122022-02-16 08:41:29 +0000313
Daniel Norman03747412022-02-25 10:38:37 -0800314 def copy_recovery_file(filename):
315 for subdir in ('VENDOR', 'SYSTEM/vendor'):
316 source = os.path.join(vendor_target_files_dir, subdir, filename)
317 if os.path.exists(source):
318 dest = os.path.join(target_files_dir, subdir, filename)
319 shutil.copy(source, dest)
320 return
321 logger.info('Skipping copy_recovery_file for %s, file not found', filename)
322
323 if OPTIONS.rebuild_recovery:
324 copy_recovery_file('etc/recovery.img')
325 copy_recovery_file('bin/install-recovery.sh')
326 copy_recovery_file('recovery-from-boot.p')
Daniel Norman571e1822021-06-25 17:18:25 -0700327
328
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900329def generate_super_empty_image(target_dir, output_super_empty):
Tao Bao2ad4b822019-06-27 16:52:12 -0700330 """Generates super_empty image from target package.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900331
332 Args:
333 target_dir: Path to the target file package which contains misc_info.txt for
334 detailed information for super image.
335 output_super_empty: If provided, copies a super_empty.img file from the
336 target files package to this path.
337 """
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700338 # Create super_empty.img using the merged misc_info.txt.
339
Daniel Norman4cc9df62019-07-18 10:11:07 -0700340 misc_info_txt = os.path.join(target_dir, 'META', 'misc_info.txt')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700341
Kiyoung Kimebe7c9c2019-06-25 17:09:55 +0900342 use_dynamic_partitions = common.LoadDictionaryFromFile(misc_info_txt).get(
343 'use_dynamic_partitions')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700344
345 if use_dynamic_partitions != 'true' and output_super_empty:
346 raise ValueError(
347 'Building super_empty.img requires use_dynamic_partitions=true.')
348 elif use_dynamic_partitions == 'true':
Daniel Norman4cc9df62019-07-18 10:11:07 -0700349 super_empty_img = os.path.join(target_dir, 'IMAGES', 'super_empty.img')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700350 build_super_image_args = [
351 misc_info_txt,
352 super_empty_img,
353 ]
354 build_super_image.main(build_super_image_args)
355
356 # Copy super_empty.img to the user-provided output_super_empty location.
357 if output_super_empty:
358 shutil.copyfile(super_empty_img, output_super_empty)
359
Daniel Normanb8a2f9d2019-04-24 12:55:51 -0700360
Daniel Norman03747412022-02-25 10:38:37 -0800361def create_target_files_archive(output_zip, source_dir, temp_dir):
362 """Creates a target_files zip archive from the input source dir.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900363
364 Args:
Daniel Norman03747412022-02-25 10:38:37 -0800365 output_zip: The name of the zip archive target files package.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900366 source_dir: The target directory contains package to be archived.
367 temp_dir: Path to temporary directory for any intermediate files.
368 """
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800369 output_target_files_list = os.path.join(temp_dir, 'output.list')
Daniel Norman4cc9df62019-07-18 10:11:07 -0700370 output_target_files_meta_dir = os.path.join(source_dir, 'META')
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800371
Daniel Normandbbf5a32020-10-22 16:03:32 -0700372 def files_from_path(target_path, extra_args=None):
373 """Gets files under the given path and return a sorted list."""
374 find_command = ['find', target_path] + (extra_args or [])
375 find_process = common.Run(
376 find_command, stdout=subprocess.PIPE, verbose=False)
377 return common.RunAndCheckOutput(['sort'],
378 stdin=find_process.stdout,
379 verbose=False)
380
Daniel Norman03747412022-02-25 10:38:37 -0800381 # META content appears first in the zip. This is done by the
382 # standard build system for optimized extraction of those files,
383 # so we do the same step for merged target_files.zips here too.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900384 meta_content = files_from_path(output_target_files_meta_dir)
Daniel Norman4cc9df62019-07-18 10:11:07 -0700385 other_content = files_from_path(
386 source_dir,
387 ['-path', output_target_files_meta_dir, '-prune', '-o', '-print'])
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800388
Tao Bao2ad4b822019-06-27 16:52:12 -0700389 with open(output_target_files_list, 'w') as f:
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800390 f.write(meta_content)
391 f.write(other_content)
392
393 command = [
Bill Peckhamf753e152019-02-19 18:02:46 -0800394 'soong_zip',
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800395 '-d',
Daniel Normane5b134a2019-04-17 14:54:06 -0700396 '-o',
Daniel Norman03747412022-02-25 10:38:37 -0800397 os.path.abspath(output_zip),
Daniel Normane5b134a2019-04-17 14:54:06 -0700398 '-C',
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900399 source_dir,
Daniel Normaneaf5c1d2021-02-09 11:01:42 -0800400 '-r',
Daniel Normane5b134a2019-04-17 14:54:06 -0700401 output_target_files_list,
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800402 ]
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900403
Daniel Norman03747412022-02-25 10:38:37 -0800404 logger.info('creating %s', output_zip)
Daniel Normaneaf5c1d2021-02-09 11:01:42 -0800405 common.RunAndCheckOutput(command, verbose=True)
Daniel Norman03747412022-02-25 10:38:37 -0800406 logger.info('finished creating %s', output_zip)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900407
408
Daniel Norman03747412022-02-25 10:38:37 -0800409def merge_target_files(temp_dir):
Tao Bao2ad4b822019-06-27 16:52:12 -0700410 """Merges two target files packages together.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900411
Daniel Norman03747412022-02-25 10:38:37 -0800412 This function uses framework and vendor target files packages as input,
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900413 performs various file extractions, special case processing, and finally
414 creates a merged zip archive as output.
415
416 Args:
417 temp_dir: The name of a directory we use when we extract items from the
418 input target files packages, and also a scratch directory that we use for
419 temporary files.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900420 """
421
422 logger.info('starting: merge framework %s and vendor %s into output %s',
Daniel Norman03747412022-02-25 10:38:37 -0800423 OPTIONS.framework_target_files, OPTIONS.vendor_target_files,
424 OPTIONS.output_target_files)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900425
Daniel Norman03747412022-02-25 10:38:37 -0800426 output_target_files_temp_dir = create_merged_package(temp_dir)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900427
Daniel Norman21c34f72020-11-11 17:25:50 -0800428 partition_map = common.PartitionMapFromTargetFiles(
429 output_target_files_temp_dir)
430
Daniel Norman2465fc82022-03-02 12:01:20 -0800431 compatibility_errors = merge_compatibility_checks.CheckCompatibility(
432 target_files_dir=output_target_files_temp_dir,
433 partition_map=partition_map)
434 if compatibility_errors:
435 for error in compatibility_errors:
436 logger.error(error)
437 raise ExternalError(
438 'Found incompatibilities in the merged target files package.')
Daniel Normand3351562020-10-29 12:33:11 -0700439
Daniel Norman571e1822021-06-25 17:18:25 -0700440 # Include the compiled policy in an image if requested.
Daniel Norman03747412022-02-25 10:38:37 -0800441 if OPTIONS.rebuild_sepolicy:
442 rebuild_image_with_sepolicy(output_target_files_temp_dir)
Daniel Norman48603ff2021-02-22 15:15:24 -0800443
Daniel Norman03747412022-02-25 10:38:37 -0800444 generate_missing_images(output_target_files_temp_dir)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900445
Daniel Norman03747412022-02-25 10:38:37 -0800446 generate_super_empty_image(output_target_files_temp_dir,
447 OPTIONS.output_super_empty)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900448
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900449 # Finally, create the output target files zip archive and/or copy the
450 # output items to the output target files directory.
451
Daniel Norman03747412022-02-25 10:38:37 -0800452 if OPTIONS.output_dir:
Daniel Norman2465fc82022-03-02 12:01:20 -0800453 merge_utils.CopyItems(output_target_files_temp_dir, OPTIONS.output_dir,
454 OPTIONS.output_item_list)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900455
Daniel Norman03747412022-02-25 10:38:37 -0800456 if not OPTIONS.output_target_files:
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900457 return
458
Daniel Norman03747412022-02-25 10:38:37 -0800459 create_target_files_archive(OPTIONS.output_target_files,
460 output_target_files_temp_dir, temp_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800461
Daniel Norman74eb74b2019-09-18 14:01:48 -0700462 # Create the IMG package from the merged target files package.
Daniel Norman03747412022-02-25 10:38:37 -0800463 if OPTIONS.output_img:
464 img_from_target_files.main(
465 [OPTIONS.output_target_files, OPTIONS.output_img])
Daniel Norman74eb74b2019-09-18 14:01:48 -0700466
Daniel Norman3b64ce12019-04-16 16:11:35 -0700467 # Create the OTA package from the merged target files package.
468
Daniel Norman03747412022-02-25 10:38:37 -0800469 if OPTIONS.output_ota:
470 ota_from_target_files.main(
471 [OPTIONS.output_target_files, OPTIONS.output_ota])
Daniel Norman3b64ce12019-04-16 16:11:35 -0700472
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700473
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800474def main():
475 """The main function.
476
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800477 Process command line arguments, then call merge_target_files to
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800478 perform the heavy lifting.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800479 """
480
481 common.InitLogging()
482
Bill Peckhamf753e152019-02-19 18:02:46 -0800483 def option_handler(o, a):
484 if o == '--system-target-files':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700485 logger.warning(
486 '--system-target-files has been renamed to --framework-target-files')
487 OPTIONS.framework_target_files = a
488 elif o == '--framework-target-files':
489 OPTIONS.framework_target_files = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800490 elif o == '--system-item-list':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700491 logger.warning(
492 '--system-item-list has been renamed to --framework-item-list')
493 OPTIONS.framework_item_list = a
494 elif o == '--framework-item-list':
495 OPTIONS.framework_item_list = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800496 elif o == '--system-misc-info-keys':
Daniel Norman4cc9df62019-07-18 10:11:07 -0700497 logger.warning('--system-misc-info-keys has been renamed to '
498 '--framework-misc-info-keys')
Daniel Normand5d70ea2019-06-05 15:13:43 -0700499 OPTIONS.framework_misc_info_keys = a
500 elif o == '--framework-misc-info-keys':
501 OPTIONS.framework_misc_info_keys = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800502 elif o == '--other-target-files':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700503 logger.warning(
504 '--other-target-files has been renamed to --vendor-target-files')
505 OPTIONS.vendor_target_files = a
506 elif o == '--vendor-target-files':
507 OPTIONS.vendor_target_files = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800508 elif o == '--other-item-list':
Daniel Norman2d7989a2021-04-05 17:40:47 +0000509 logger.warning('--other-item-list has been renamed to --vendor-item-list')
Daniel Normand5d70ea2019-06-05 15:13:43 -0700510 OPTIONS.vendor_item_list = a
511 elif o == '--vendor-item-list':
512 OPTIONS.vendor_item_list = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800513 elif o == '--output-target-files':
514 OPTIONS.output_target_files = a
Daniel Normanfdb38812019-04-15 09:47:24 -0700515 elif o == '--output-dir':
516 OPTIONS.output_dir = a
517 elif o == '--output-item-list':
518 OPTIONS.output_item_list = a
Daniel Norman3b64ce12019-04-16 16:11:35 -0700519 elif o == '--output-ota':
520 OPTIONS.output_ota = a
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700521 elif o == '--output-img':
522 OPTIONS.output_img = a
Daniel Normanf0318252019-04-15 11:34:56 -0700523 elif o == '--output-super-empty':
524 OPTIONS.output_super_empty = a
Daniel Norman2465fc82022-03-02 12:01:20 -0800525 elif o == '--rebuild_recovery' or o == '--rebuild-recovery':
Daniel Normana4911da2019-03-15 14:36:21 -0700526 OPTIONS.rebuild_recovery = True
Daniel Normanb0c75912020-09-24 14:30:21 -0700527 elif o == '--allow-duplicate-apkapex-keys':
528 OPTIONS.allow_duplicate_apkapex_keys = True
Daniel Norman571e1822021-06-25 17:18:25 -0700529 elif o == '--vendor-otatools':
530 OPTIONS.vendor_otatools = a
531 elif o == '--rebuild-sepolicy':
532 OPTIONS.rebuild_sepolicy = True
Bill Peckham364c1cc2019-03-29 18:27:23 -0700533 elif o == '--keep-tmp':
Bill Peckhamf753e152019-02-19 18:02:46 -0800534 OPTIONS.keep_tmp = True
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700535 elif o == '--framework-dexpreopt-config':
536 OPTIONS.framework_dexpreopt_config = a
537 elif o == '--framework-dexpreopt-tools':
538 OPTIONS.framework_dexpreopt_tools = a
539 elif o == '--vendor-dexpreopt-config':
540 OPTIONS.vendor_dexpreopt_config = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800541 else:
542 return False
543 return True
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800544
Bill Peckhamf753e152019-02-19 18:02:46 -0800545 args = common.ParseOptions(
Daniel Normane5b134a2019-04-17 14:54:06 -0700546 sys.argv[1:],
547 __doc__,
Bill Peckhamf753e152019-02-19 18:02:46 -0800548 extra_long_opts=[
549 'system-target-files=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700550 'framework-target-files=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800551 'system-item-list=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700552 'framework-item-list=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800553 'system-misc-info-keys=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700554 'framework-misc-info-keys=',
Bill Peckhamf753e152019-02-19 18:02:46 -0800555 'other-target-files=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700556 'vendor-target-files=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800557 'other-item-list=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700558 'vendor-item-list=',
Bill Peckhamf753e152019-02-19 18:02:46 -0800559 'output-target-files=',
Daniel Normanfdb38812019-04-15 09:47:24 -0700560 'output-dir=',
561 'output-item-list=',
Daniel Norman3b64ce12019-04-16 16:11:35 -0700562 'output-ota=',
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700563 'output-img=',
Daniel Normanf0318252019-04-15 11:34:56 -0700564 'output-super-empty=',
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700565 'framework-dexpreopt-config=',
566 'framework-dexpreopt-tools=',
567 'vendor-dexpreopt-config=',
Daniel Normana4911da2019-03-15 14:36:21 -0700568 'rebuild_recovery',
Daniel Norman2465fc82022-03-02 12:01:20 -0800569 'rebuild-recovery',
Daniel Normanb0c75912020-09-24 14:30:21 -0700570 'allow-duplicate-apkapex-keys',
Daniel Norman571e1822021-06-25 17:18:25 -0700571 'vendor-otatools=',
572 'rebuild-sepolicy',
Bill Peckham364c1cc2019-03-29 18:27:23 -0700573 'keep-tmp',
Bill Peckhamf753e152019-02-19 18:02:46 -0800574 ],
575 extra_option_handler=option_handler)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800576
Tao Bao2ad4b822019-06-27 16:52:12 -0700577 # pylint: disable=too-many-boolean-expressions
Daniel Normand5d70ea2019-06-05 15:13:43 -0700578 if (args or OPTIONS.framework_target_files is None or
579 OPTIONS.vendor_target_files is None or
Daniel Normane5b134a2019-04-17 14:54:06 -0700580 (OPTIONS.output_target_files is None and OPTIONS.output_dir is None) or
Daniel Norman5f476772022-03-02 15:46:34 -0800581 (OPTIONS.output_dir is not None and not OPTIONS.output_item_list) or
Po Hua6c59122022-02-16 08:41:29 +0000582 (OPTIONS.rebuild_recovery and not OPTIONS.rebuild_sepolicy)):
Bill Peckhamf753e152019-02-19 18:02:46 -0800583 common.Usage(__doc__)
Bill Peckham889b0c62019-02-21 18:53:37 -0800584 sys.exit(1)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800585
Dennis Song5bfa43e2023-03-30 18:28:00 +0800586 framework_namelist = merge_utils.GetTargetFilesItems(
587 OPTIONS.framework_target_files)
588 vendor_namelist = merge_utils.GetTargetFilesItems(
589 OPTIONS.vendor_target_files)
Daniel Norman5f476772022-03-02 15:46:34 -0800590
Daniel Normand5d70ea2019-06-05 15:13:43 -0700591 if OPTIONS.framework_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800592 OPTIONS.framework_item_list = common.LoadListFromFile(
593 OPTIONS.framework_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800594 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800595 OPTIONS.framework_item_list = merge_utils.InferItemList(
596 input_namelist=framework_namelist, framework=True)
Daniel Norman2465fc82022-03-02 12:01:20 -0800597 OPTIONS.framework_partition_set = merge_utils.ItemListToPartitionSet(
Daniel Norman03747412022-02-25 10:38:37 -0800598 OPTIONS.framework_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800599
Daniel Normand5d70ea2019-06-05 15:13:43 -0700600 if OPTIONS.framework_misc_info_keys:
Daniel Norman03747412022-02-25 10:38:37 -0800601 OPTIONS.framework_misc_info_keys = common.LoadListFromFile(
Daniel Normand5d70ea2019-06-05 15:13:43 -0700602 OPTIONS.framework_misc_info_keys)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800603 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800604 OPTIONS.framework_misc_info_keys = merge_utils.InferFrameworkMiscInfoKeys(
605 input_namelist=framework_namelist)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800606
Daniel Normand5d70ea2019-06-05 15:13:43 -0700607 if OPTIONS.vendor_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800608 OPTIONS.vendor_item_list = common.LoadListFromFile(OPTIONS.vendor_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800609 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800610 OPTIONS.vendor_item_list = merge_utils.InferItemList(
611 input_namelist=vendor_namelist, framework=False)
Daniel Norman2465fc82022-03-02 12:01:20 -0800612 OPTIONS.vendor_partition_set = merge_utils.ItemListToPartitionSet(
Daniel Norman03747412022-02-25 10:38:37 -0800613 OPTIONS.vendor_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800614
Daniel Normanfdb38812019-04-15 09:47:24 -0700615 if OPTIONS.output_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800616 OPTIONS.output_item_list = common.LoadListFromFile(OPTIONS.output_item_list)
Daniel Normanfdb38812019-04-15 09:47:24 -0700617
Daniel Norman2465fc82022-03-02 12:01:20 -0800618 if not merge_utils.ValidateConfigLists():
Daniel Normane5964522019-03-19 10:32:03 -0700619 sys.exit(1)
620
Daniel Norman2465fc82022-03-02 12:01:20 -0800621 temp_dir = common.MakeTempDir(prefix='merge_target_files_')
622 try:
623 merge_target_files(temp_dir)
624 finally:
625 if OPTIONS.keep_tmp:
626 logger.info('Keeping temp_dir %s', temp_dir)
627 else:
628 common.Cleanup()
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800629
630
631if __name__ == '__main__':
Bill Peckham889b0c62019-02-21 18:53:37 -0800632 main()