blob: ba2b14f0140fa3ba06e3897172721252a43a393b [file] [log] [blame]
Bill Peckhame9eb5f92019-02-01 15:52:10 -08001#!/usr/bin/env python
2#
Daniel Norman03747412022-02-25 10:38:37 -08003# Copyright (C) 2022 The Android Open Source Project
Bill Peckhame9eb5f92019-02-01 15:52:10 -08004#
5# Licensed under the Apache License, Version 2.0 (the "License"); you may not
6# use this file except in compliance with the License. You may obtain a copy of
7# the License at
8#
9# http://www.apache.org/licenses/LICENSE-2.0
10#
11# Unless required by applicable law or agreed to in writing, software
12# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
13# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
14# License for the specific language governing permissions and limitations under
15# the License.
Daniel Norman4cc9df62019-07-18 10:11:07 -070016#
17"""This script merges two partial target files packages.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080018
Daniel Normandbbf5a32020-10-22 16:03:32 -070019One input package contains framework files, and the other contains vendor files.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080020
Daniel Normandbbf5a32020-10-22 16:03:32 -070021This script produces a complete, merged target files package:
22 - This package can be used to generate a flashable IMG package.
23 See --output-img.
24 - This package can be used to generate an OTA package. See --output-ota.
25 - The merged package is checked for compatibility between the two inputs.
26
27Usage: merge_target_files [args]
Bill Peckhame9eb5f92019-02-01 15:52:10 -080028
Dennis Song5bfa43e2023-03-30 18:28:00 +080029 --framework-target-files framework-target-files-package
Daniel Normand5d70ea2019-06-05 15:13:43 -070030 The input target files package containing framework bits. This is a zip
Dennis Song5bfa43e2023-03-30 18:28:00 +080031 archive or a directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080032
Daniel Normand5d70ea2019-06-05 15:13:43 -070033 --framework-item-list framework-item-list-file
Daniel Norman5f476772022-03-02 15:46:34 -080034 The optional path to a newline-separated config file of items that
35 are extracted as-is from the framework target files package.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080036
Daniel Normand5d70ea2019-06-05 15:13:43 -070037 --framework-misc-info-keys framework-misc-info-keys-file
Daniel Norman5f476772022-03-02 15:46:34 -080038 The optional path to a newline-separated config file of keys to
39 extract from the framework META/misc_info.txt file.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080040
Dennis Song5bfa43e2023-03-30 18:28:00 +080041 --vendor-target-files vendor-target-files-package
Daniel Normand5d70ea2019-06-05 15:13:43 -070042 The input target files package containing vendor bits. This is a zip
Dennis Song5bfa43e2023-03-30 18:28:00 +080043 archive or a directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -080044
Daniel Normand5d70ea2019-06-05 15:13:43 -070045 --vendor-item-list vendor-item-list-file
Daniel Norman5f476772022-03-02 15:46:34 -080046 The optional path to a newline-separated config file of items that
47 are extracted as-is from the vendor target files package.
Daniel Norman2c99c5b2019-03-07 13:01:48 -080048
Bill Peckhame9eb5f92019-02-01 15:52:10 -080049 --output-target-files output-target-files-package
Daniel Normanfdb38812019-04-15 09:47:24 -070050 If provided, the output merged target files package. Also a zip archive.
51
52 --output-dir output-directory
53 If provided, the destination directory for saving merged files. Requires
54 the --output-item-list flag.
55 Can be provided alongside --output-target-files, or by itself.
56
57 --output-item-list output-item-list-file.
58 The optional path to a newline-separated config file that specifies the
59 file patterns to copy into the --output-dir. Required if providing
60 the --output-dir flag.
Daniel Normana4911da2019-03-15 14:36:21 -070061
Daniel Norman3b64ce12019-04-16 16:11:35 -070062 --output-ota output-ota-package
63 The output ota package. This is a zip archive. Use of this flag may
64 require passing the --path common flag; see common.py.
65
Daniel Norman1bd2a1d2019-04-18 12:32:18 -070066 --output-img output-img-package
67 The output img package, suitable for use with 'fastboot update'. Use of
68 this flag may require passing the --path common flag; see common.py.
69
Daniel Normanf0318252019-04-15 11:34:56 -070070 --output-super-empty output-super-empty-image
71 If provided, creates a super_empty.img file from the merged target
72 files package and saves it at this path.
73
Daniel Normana4911da2019-03-15 14:36:21 -070074 --rebuild_recovery
Po Hua6c59122022-02-16 08:41:29 +000075 Copy the recovery image used by non-A/B devices, used when
76 regenerating vendor images with --rebuild-sepolicy.
Bill Peckham364c1cc2019-03-29 18:27:23 -070077
Daniel Normanb0c75912020-09-24 14:30:21 -070078 --allow-duplicate-apkapex-keys
79 If provided, duplicate APK/APEX keys are ignored and the value from the
80 framework is used.
81
Daniel Norman571e1822021-06-25 17:18:25 -070082 --rebuild-sepolicy
83 If provided, rebuilds odm.img or vendor.img to include merged sepolicy
84 files. If odm is present then odm is preferred.
85
86 --vendor-otatools otatools.zip
87 If provided, use this otatools.zip when recompiling the odm or vendor
88 image to include sepolicy.
89
Bill Peckham364c1cc2019-03-29 18:27:23 -070090 --keep-tmp
91 Keep tempoary files for debugging purposes.
Jose Galmes9c8f6eb2021-07-21 09:34:08 -070092
93 The following only apply when using the VSDK to perform dexopt on vendor apps:
94
95 --framework-dexpreopt-config
96 If provided, the location of framwework's dexpreopt_config.zip.
97
98 --framework-dexpreopt-tools
99 if provided, the location of framework's dexpreopt_tools.zip.
100
101 --vendor-dexpreopt-config
102 If provided, the location of vendor's dexpreopt_config.zip.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800103"""
104
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800105import logging
106import os
Daniel Normanfdb38812019-04-15 09:47:24 -0700107import shutil
Bill Peckham540d91a2019-04-25 14:18:16 -0700108import subprocess
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800109import sys
Daniel Norman5f476772022-03-02 15:46:34 -0800110import zipfile
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800111
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800112import add_img_to_target_files
Daniel Normandb8cacc2021-04-09 15:34:43 -0700113import build_image
Daniel Normanf0318252019-04-15 11:34:56 -0700114import build_super_image
115import common
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700116import img_from_target_files
Daniel Norman2465fc82022-03-02 12:01:20 -0800117import merge_compatibility_checks
118import merge_dexopt
119import merge_meta
120import merge_utils
Daniel Norman3b64ce12019-04-16 16:11:35 -0700121import ota_from_target_files
Daniel Normandb8cacc2021-04-09 15:34:43 -0700122
Daniel Normana84d13b2022-02-17 14:16:40 -0800123from common import ExternalError
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800124
125logger = logging.getLogger(__name__)
Tao Bao2ad4b822019-06-27 16:52:12 -0700126
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800127OPTIONS = common.OPTIONS
Bill Peckhamcb848172020-04-03 12:50:47 -0700128# Always turn on verbose logging.
129OPTIONS.verbose = True
Daniel Normand5d70ea2019-06-05 15:13:43 -0700130OPTIONS.framework_target_files = None
Daniel Norman5f476772022-03-02 15:46:34 -0800131OPTIONS.framework_item_list = []
132OPTIONS.framework_misc_info_keys = []
Daniel Normand5d70ea2019-06-05 15:13:43 -0700133OPTIONS.vendor_target_files = None
Daniel Norman5f476772022-03-02 15:46:34 -0800134OPTIONS.vendor_item_list = []
Bill Peckhamf753e152019-02-19 18:02:46 -0800135OPTIONS.output_target_files = None
Daniel Normanfdb38812019-04-15 09:47:24 -0700136OPTIONS.output_dir = None
Daniel Norman5f476772022-03-02 15:46:34 -0800137OPTIONS.output_item_list = []
Daniel Norman3b64ce12019-04-16 16:11:35 -0700138OPTIONS.output_ota = None
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700139OPTIONS.output_img = None
Daniel Normanf0318252019-04-15 11:34:56 -0700140OPTIONS.output_super_empty = None
Daniel Normana4911da2019-03-15 14:36:21 -0700141OPTIONS.rebuild_recovery = False
Daniel Normanb0c75912020-09-24 14:30:21 -0700142# TODO(b/150582573): Remove this option.
143OPTIONS.allow_duplicate_apkapex_keys = False
Daniel Norman571e1822021-06-25 17:18:25 -0700144OPTIONS.vendor_otatools = None
145OPTIONS.rebuild_sepolicy = False
Bill Peckhamf753e152019-02-19 18:02:46 -0800146OPTIONS.keep_tmp = False
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700147OPTIONS.framework_dexpreopt_config = None
148OPTIONS.framework_dexpreopt_tools = None
149OPTIONS.vendor_dexpreopt_config = None
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800150
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800151
jiangxu52bd42732022-04-11 16:02:48 +0800152def move_only_exists(source, destination):
153 """Judge whether the file exists and then move the file."""
154
155 if os.path.exists(source):
156 shutil.move(source, destination)
157
158
Dennis Song16d6fe02023-02-23 23:37:42 +0800159def remove_file_if_exists(file_name):
160 """Remove the file if it exists and skip otherwise."""
161
162 try:
163 os.remove(file_name)
164 except FileNotFoundError:
165 pass
166
167
Dennis Song09f5c5d2023-04-14 11:43:04 +0800168def include_meta_in_list(item_list):
169 """Include all `META/*` files in the item list.
170
171 To ensure that `AddImagesToTargetFiles` can still be used with vendor item
172 list that do not specify all of the required META/ files, those files should
173 be included by default. This preserves the backward compatibility of
174 `rebuild_image_with_sepolicy`.
175 """
176 if not item_list:
177 return None
178 return list(item_list) + ['META/*']
179
180
Daniel Norman03747412022-02-25 10:38:37 -0800181def create_merged_package(temp_dir):
Tao Bao2ad4b822019-06-27 16:52:12 -0700182 """Merges two target files packages into one target files structure.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800183
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900184 Returns:
185 Path to merged package under temp directory.
186 """
Daniel Normandbbf5a32020-10-22 16:03:32 -0700187 # Extract "as is" items from the input framework and vendor partial target
Dennis Song5bfa43e2023-03-30 18:28:00 +0800188 # files packages directly into the output temporary directory, since these
189 # items do not need special case processing.
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800190
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800191 output_target_files_temp_dir = os.path.join(temp_dir, 'output')
Dennis Song5bfa43e2023-03-30 18:28:00 +0800192 merge_utils.CollectTargetFiles(
193 input_zipfile_or_dir=OPTIONS.framework_target_files,
Daniel Norman03747412022-02-25 10:38:37 -0800194 output_dir=output_target_files_temp_dir,
Dennis Song5bfa43e2023-03-30 18:28:00 +0800195 item_list=OPTIONS.framework_item_list)
196 merge_utils.CollectTargetFiles(
197 input_zipfile_or_dir=OPTIONS.vendor_target_files,
Daniel Norman03747412022-02-25 10:38:37 -0800198 output_dir=output_target_files_temp_dir,
Dennis Song5bfa43e2023-03-30 18:28:00 +0800199 item_list=OPTIONS.vendor_item_list)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800200
Daniel Normandbbf5a32020-10-22 16:03:32 -0700201 # Perform special case processing on META/* items.
202 # After this function completes successfully, all the files we need to create
203 # the output target files package are in place.
Daniel Norman2465fc82022-03-02 12:01:20 -0800204 merge_meta.MergeMetaFiles(
205 temp_dir=temp_dir, merged_dir=output_target_files_temp_dir)
Daniel Norman03747412022-02-25 10:38:37 -0800206
Daniel Norman2465fc82022-03-02 12:01:20 -0800207 merge_dexopt.MergeDexopt(
Daniel Norman03747412022-02-25 10:38:37 -0800208 temp_dir=temp_dir, output_target_files_dir=output_target_files_temp_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800209
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900210 return output_target_files_temp_dir
211
212
Daniel Norman03747412022-02-25 10:38:37 -0800213def generate_missing_images(target_files_dir):
214 """Generate any missing images from target files."""
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900215
216 # Regenerate IMAGES in the target directory.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800217
Daniel Normandbbf5a32020-10-22 16:03:32 -0700218 add_img_args = [
219 '--verbose',
220 '--add_missing',
221 ]
Daniel Norman03747412022-02-25 10:38:37 -0800222 if OPTIONS.rebuild_recovery:
Daniel Normana4911da2019-03-15 14:36:21 -0700223 add_img_args.append('--rebuild_recovery')
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900224 add_img_args.append(target_files_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800225
226 add_img_to_target_files.main(add_img_args)
227
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900228
Daniel Norman03747412022-02-25 10:38:37 -0800229def rebuild_image_with_sepolicy(target_files_dir):
Daniel Norman571e1822021-06-25 17:18:25 -0700230 """Rebuilds odm.img or vendor.img to include merged sepolicy files.
231
232 If odm is present then odm is preferred -- otherwise vendor is used.
Daniel Norman571e1822021-06-25 17:18:25 -0700233 """
234 partition = 'vendor'
Dennis Song16d6fe02023-02-23 23:37:42 +0800235 if os.path.exists(os.path.join(target_files_dir, 'ODM')):
Daniel Norman571e1822021-06-25 17:18:25 -0700236 partition = 'odm'
237 partition_img = '{}.img'.format(partition)
Po Hua6c59122022-02-16 08:41:29 +0000238 partition_map = '{}.map'.format(partition)
Daniel Norman571e1822021-06-25 17:18:25 -0700239
240 logger.info('Recompiling %s using the merged sepolicy files.', partition_img)
241
242 # Copy the combined SEPolicy file and framework hashes to the image that is
243 # being rebuilt.
244 def copy_selinux_file(input_path, output_filename):
Po Hu0e4403e2021-07-06 17:05:56 +0800245 input_filename = os.path.join(target_files_dir, input_path)
246 if not os.path.exists(input_filename):
Dennis Song5bfa43e2023-03-30 18:28:00 +0800247 input_filename = input_filename.replace('SYSTEM_EXT/',
248 'SYSTEM/system_ext/') \
Po Hu0e4403e2021-07-06 17:05:56 +0800249 .replace('PRODUCT/', 'SYSTEM/product/')
250 if not os.path.exists(input_filename):
251 logger.info('Skipping copy_selinux_file for %s', input_filename)
252 return
Daniel Norman571e1822021-06-25 17:18:25 -0700253 shutil.copy(
Po Hu0e4403e2021-07-06 17:05:56 +0800254 input_filename,
Daniel Norman571e1822021-06-25 17:18:25 -0700255 os.path.join(target_files_dir, partition.upper(), 'etc/selinux',
256 output_filename))
257
258 copy_selinux_file('META/combined_sepolicy', 'precompiled_sepolicy')
259 copy_selinux_file('SYSTEM/etc/selinux/plat_sepolicy_and_mapping.sha256',
260 'precompiled_sepolicy.plat_sepolicy_and_mapping.sha256')
261 copy_selinux_file(
262 'SYSTEM_EXT/etc/selinux/system_ext_sepolicy_and_mapping.sha256',
263 'precompiled_sepolicy.system_ext_sepolicy_and_mapping.sha256')
264 copy_selinux_file('PRODUCT/etc/selinux/product_sepolicy_and_mapping.sha256',
265 'precompiled_sepolicy.product_sepolicy_and_mapping.sha256')
266
Daniel Norman03747412022-02-25 10:38:37 -0800267 if not OPTIONS.vendor_otatools:
Daniel Norman571e1822021-06-25 17:18:25 -0700268 # Remove the partition from the merged target-files archive. It will be
Daniel Norman03747412022-02-25 10:38:37 -0800269 # rebuilt later automatically by generate_missing_images().
Dennis Song16d6fe02023-02-23 23:37:42 +0800270 remove_file_if_exists(
271 os.path.join(target_files_dir, 'IMAGES', partition_img))
Daniel Norman03747412022-02-25 10:38:37 -0800272 return
Daniel Norman571e1822021-06-25 17:18:25 -0700273
Daniel Norman03747412022-02-25 10:38:37 -0800274 # TODO(b/192253131): Remove the need for vendor_otatools by fixing
275 # backwards-compatibility issues when compiling images across releases.
276 if not OPTIONS.vendor_target_files:
277 raise ValueError(
278 'Expected vendor_target_files if vendor_otatools is not None.')
279 logger.info(
280 '%s recompilation will be performed using the vendor otatools.zip',
281 partition_img)
Daniel Norman571e1822021-06-25 17:18:25 -0700282
Daniel Norman03747412022-02-25 10:38:37 -0800283 # Unzip the vendor build's otatools.zip and target-files archive.
284 vendor_otatools_dir = common.MakeTempDir(
285 prefix='merge_target_files_vendor_otatools_')
286 vendor_target_files_dir = common.MakeTempDir(
287 prefix='merge_target_files_vendor_target_files_')
288 common.UnzipToDir(OPTIONS.vendor_otatools, vendor_otatools_dir)
Dennis Song5bfa43e2023-03-30 18:28:00 +0800289 merge_utils.CollectTargetFiles(
290 input_zipfile_or_dir=OPTIONS.vendor_target_files,
291 output_dir=vendor_target_files_dir,
Dennis Song09f5c5d2023-04-14 11:43:04 +0800292 item_list=include_meta_in_list(OPTIONS.vendor_item_list))
Daniel Norman571e1822021-06-25 17:18:25 -0700293
Daniel Norman03747412022-02-25 10:38:37 -0800294 # Copy the partition contents from the merged target-files archive to the
295 # vendor target-files archive.
296 shutil.rmtree(os.path.join(vendor_target_files_dir, partition.upper()))
297 shutil.copytree(
298 os.path.join(target_files_dir, partition.upper()),
299 os.path.join(vendor_target_files_dir, partition.upper()),
300 symlinks=True)
Daniel Norman571e1822021-06-25 17:18:25 -0700301
Daniel Norman03747412022-02-25 10:38:37 -0800302 # Delete then rebuild the partition.
Dennis Song16d6fe02023-02-23 23:37:42 +0800303 remove_file_if_exists(
304 os.path.join(vendor_target_files_dir, 'IMAGES', partition_img))
Daniel Norman03747412022-02-25 10:38:37 -0800305 rebuild_partition_command = [
306 os.path.join(vendor_otatools_dir, 'bin', 'add_img_to_target_files'),
307 '--verbose',
308 '--add_missing',
309 ]
310 if OPTIONS.rebuild_recovery:
311 rebuild_partition_command.append('--rebuild_recovery')
312 rebuild_partition_command.append(vendor_target_files_dir)
313 logger.info('Recompiling %s: %s', partition_img,
314 ' '.join(rebuild_partition_command))
315 common.RunAndCheckOutput(rebuild_partition_command, verbose=True)
Po Hua6c59122022-02-16 08:41:29 +0000316
Daniel Norman03747412022-02-25 10:38:37 -0800317 # Move the newly-created image to the merged target files dir.
318 if not os.path.exists(os.path.join(target_files_dir, 'IMAGES')):
319 os.makedirs(os.path.join(target_files_dir, 'IMAGES'))
320 shutil.move(
321 os.path.join(vendor_target_files_dir, 'IMAGES', partition_img),
322 os.path.join(target_files_dir, 'IMAGES', partition_img))
Dennis Song5bfa43e2023-03-30 18:28:00 +0800323 move_only_exists(
324 os.path.join(vendor_target_files_dir, 'IMAGES', partition_map),
325 os.path.join(target_files_dir, 'IMAGES', partition_map))
Po Hua6c59122022-02-16 08:41:29 +0000326
Daniel Norman03747412022-02-25 10:38:37 -0800327 def copy_recovery_file(filename):
328 for subdir in ('VENDOR', 'SYSTEM/vendor'):
329 source = os.path.join(vendor_target_files_dir, subdir, filename)
330 if os.path.exists(source):
331 dest = os.path.join(target_files_dir, subdir, filename)
332 shutil.copy(source, dest)
333 return
334 logger.info('Skipping copy_recovery_file for %s, file not found', filename)
335
336 if OPTIONS.rebuild_recovery:
337 copy_recovery_file('etc/recovery.img')
338 copy_recovery_file('bin/install-recovery.sh')
339 copy_recovery_file('recovery-from-boot.p')
Daniel Norman571e1822021-06-25 17:18:25 -0700340
341
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900342def generate_super_empty_image(target_dir, output_super_empty):
Tao Bao2ad4b822019-06-27 16:52:12 -0700343 """Generates super_empty image from target package.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900344
345 Args:
346 target_dir: Path to the target file package which contains misc_info.txt for
347 detailed information for super image.
348 output_super_empty: If provided, copies a super_empty.img file from the
349 target files package to this path.
350 """
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700351 # Create super_empty.img using the merged misc_info.txt.
352
Daniel Norman4cc9df62019-07-18 10:11:07 -0700353 misc_info_txt = os.path.join(target_dir, 'META', 'misc_info.txt')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700354
Kiyoung Kimebe7c9c2019-06-25 17:09:55 +0900355 use_dynamic_partitions = common.LoadDictionaryFromFile(misc_info_txt).get(
356 'use_dynamic_partitions')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700357
358 if use_dynamic_partitions != 'true' and output_super_empty:
359 raise ValueError(
360 'Building super_empty.img requires use_dynamic_partitions=true.')
361 elif use_dynamic_partitions == 'true':
Daniel Norman4cc9df62019-07-18 10:11:07 -0700362 super_empty_img = os.path.join(target_dir, 'IMAGES', 'super_empty.img')
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700363 build_super_image_args = [
364 misc_info_txt,
365 super_empty_img,
366 ]
367 build_super_image.main(build_super_image_args)
368
369 # Copy super_empty.img to the user-provided output_super_empty location.
370 if output_super_empty:
371 shutil.copyfile(super_empty_img, output_super_empty)
372
Daniel Normanb8a2f9d2019-04-24 12:55:51 -0700373
Daniel Norman03747412022-02-25 10:38:37 -0800374def create_target_files_archive(output_zip, source_dir, temp_dir):
375 """Creates a target_files zip archive from the input source dir.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900376
377 Args:
Daniel Norman03747412022-02-25 10:38:37 -0800378 output_zip: The name of the zip archive target files package.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900379 source_dir: The target directory contains package to be archived.
380 temp_dir: Path to temporary directory for any intermediate files.
381 """
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800382 output_target_files_list = os.path.join(temp_dir, 'output.list')
Daniel Norman4cc9df62019-07-18 10:11:07 -0700383 output_target_files_meta_dir = os.path.join(source_dir, 'META')
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800384
Daniel Normandbbf5a32020-10-22 16:03:32 -0700385 def files_from_path(target_path, extra_args=None):
386 """Gets files under the given path and return a sorted list."""
387 find_command = ['find', target_path] + (extra_args or [])
388 find_process = common.Run(
389 find_command, stdout=subprocess.PIPE, verbose=False)
390 return common.RunAndCheckOutput(['sort'],
391 stdin=find_process.stdout,
392 verbose=False)
393
Daniel Norman03747412022-02-25 10:38:37 -0800394 # META content appears first in the zip. This is done by the
395 # standard build system for optimized extraction of those files,
396 # so we do the same step for merged target_files.zips here too.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900397 meta_content = files_from_path(output_target_files_meta_dir)
Daniel Norman4cc9df62019-07-18 10:11:07 -0700398 other_content = files_from_path(
399 source_dir,
400 ['-path', output_target_files_meta_dir, '-prune', '-o', '-print'])
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800401
Tao Bao2ad4b822019-06-27 16:52:12 -0700402 with open(output_target_files_list, 'w') as f:
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800403 f.write(meta_content)
404 f.write(other_content)
405
406 command = [
Bill Peckhamf753e152019-02-19 18:02:46 -0800407 'soong_zip',
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800408 '-d',
Daniel Normane5b134a2019-04-17 14:54:06 -0700409 '-o',
Daniel Norman03747412022-02-25 10:38:37 -0800410 os.path.abspath(output_zip),
Daniel Normane5b134a2019-04-17 14:54:06 -0700411 '-C',
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900412 source_dir,
Daniel Normaneaf5c1d2021-02-09 11:01:42 -0800413 '-r',
Daniel Normane5b134a2019-04-17 14:54:06 -0700414 output_target_files_list,
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800415 ]
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900416
Daniel Norman03747412022-02-25 10:38:37 -0800417 logger.info('creating %s', output_zip)
Daniel Normaneaf5c1d2021-02-09 11:01:42 -0800418 common.RunAndCheckOutput(command, verbose=True)
Daniel Norman03747412022-02-25 10:38:37 -0800419 logger.info('finished creating %s', output_zip)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900420
421
Daniel Norman03747412022-02-25 10:38:37 -0800422def merge_target_files(temp_dir):
Tao Bao2ad4b822019-06-27 16:52:12 -0700423 """Merges two target files packages together.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900424
Daniel Norman03747412022-02-25 10:38:37 -0800425 This function uses framework and vendor target files packages as input,
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900426 performs various file extractions, special case processing, and finally
427 creates a merged zip archive as output.
428
429 Args:
430 temp_dir: The name of a directory we use when we extract items from the
431 input target files packages, and also a scratch directory that we use for
432 temporary files.
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900433 """
434
435 logger.info('starting: merge framework %s and vendor %s into output %s',
Daniel Norman03747412022-02-25 10:38:37 -0800436 OPTIONS.framework_target_files, OPTIONS.vendor_target_files,
437 OPTIONS.output_target_files)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900438
Daniel Norman03747412022-02-25 10:38:37 -0800439 output_target_files_temp_dir = create_merged_package(temp_dir)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900440
Daniel Norman21c34f72020-11-11 17:25:50 -0800441 partition_map = common.PartitionMapFromTargetFiles(
442 output_target_files_temp_dir)
443
Daniel Norman2465fc82022-03-02 12:01:20 -0800444 compatibility_errors = merge_compatibility_checks.CheckCompatibility(
445 target_files_dir=output_target_files_temp_dir,
446 partition_map=partition_map)
447 if compatibility_errors:
448 for error in compatibility_errors:
449 logger.error(error)
450 raise ExternalError(
451 'Found incompatibilities in the merged target files package.')
Daniel Normand3351562020-10-29 12:33:11 -0700452
Daniel Norman571e1822021-06-25 17:18:25 -0700453 # Include the compiled policy in an image if requested.
Daniel Norman03747412022-02-25 10:38:37 -0800454 if OPTIONS.rebuild_sepolicy:
455 rebuild_image_with_sepolicy(output_target_files_temp_dir)
Daniel Norman48603ff2021-02-22 15:15:24 -0800456
Daniel Norman03747412022-02-25 10:38:37 -0800457 generate_missing_images(output_target_files_temp_dir)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900458
Daniel Norman03747412022-02-25 10:38:37 -0800459 generate_super_empty_image(output_target_files_temp_dir,
460 OPTIONS.output_super_empty)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900461
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900462 # Finally, create the output target files zip archive and/or copy the
463 # output items to the output target files directory.
464
Daniel Norman03747412022-02-25 10:38:37 -0800465 if OPTIONS.output_dir:
Daniel Norman2465fc82022-03-02 12:01:20 -0800466 merge_utils.CopyItems(output_target_files_temp_dir, OPTIONS.output_dir,
467 OPTIONS.output_item_list)
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900468
Daniel Norman03747412022-02-25 10:38:37 -0800469 if not OPTIONS.output_target_files:
Kiyoung Kim7cbeda72019-06-28 13:26:04 +0900470 return
471
Daniel Norman03747412022-02-25 10:38:37 -0800472 create_target_files_archive(OPTIONS.output_target_files,
473 output_target_files_temp_dir, temp_dir)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800474
Daniel Norman74eb74b2019-09-18 14:01:48 -0700475 # Create the IMG package from the merged target files package.
Daniel Norman03747412022-02-25 10:38:37 -0800476 if OPTIONS.output_img:
477 img_from_target_files.main(
478 [OPTIONS.output_target_files, OPTIONS.output_img])
Daniel Norman74eb74b2019-09-18 14:01:48 -0700479
Daniel Norman3b64ce12019-04-16 16:11:35 -0700480 # Create the OTA package from the merged target files package.
481
Daniel Norman03747412022-02-25 10:38:37 -0800482 if OPTIONS.output_ota:
483 ota_from_target_files.main(
484 [OPTIONS.output_target_files, OPTIONS.output_ota])
Daniel Norman3b64ce12019-04-16 16:11:35 -0700485
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700486
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800487def main():
488 """The main function.
489
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800490 Process command line arguments, then call merge_target_files to
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800491 perform the heavy lifting.
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800492 """
493
494 common.InitLogging()
495
Bill Peckhamf753e152019-02-19 18:02:46 -0800496 def option_handler(o, a):
497 if o == '--system-target-files':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700498 logger.warning(
499 '--system-target-files has been renamed to --framework-target-files')
500 OPTIONS.framework_target_files = a
501 elif o == '--framework-target-files':
502 OPTIONS.framework_target_files = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800503 elif o == '--system-item-list':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700504 logger.warning(
505 '--system-item-list has been renamed to --framework-item-list')
506 OPTIONS.framework_item_list = a
507 elif o == '--framework-item-list':
508 OPTIONS.framework_item_list = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800509 elif o == '--system-misc-info-keys':
Daniel Norman4cc9df62019-07-18 10:11:07 -0700510 logger.warning('--system-misc-info-keys has been renamed to '
511 '--framework-misc-info-keys')
Daniel Normand5d70ea2019-06-05 15:13:43 -0700512 OPTIONS.framework_misc_info_keys = a
513 elif o == '--framework-misc-info-keys':
514 OPTIONS.framework_misc_info_keys = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800515 elif o == '--other-target-files':
Daniel Normand5d70ea2019-06-05 15:13:43 -0700516 logger.warning(
517 '--other-target-files has been renamed to --vendor-target-files')
518 OPTIONS.vendor_target_files = a
519 elif o == '--vendor-target-files':
520 OPTIONS.vendor_target_files = a
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800521 elif o == '--other-item-list':
Daniel Norman2d7989a2021-04-05 17:40:47 +0000522 logger.warning('--other-item-list has been renamed to --vendor-item-list')
Daniel Normand5d70ea2019-06-05 15:13:43 -0700523 OPTIONS.vendor_item_list = a
524 elif o == '--vendor-item-list':
525 OPTIONS.vendor_item_list = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800526 elif o == '--output-target-files':
527 OPTIONS.output_target_files = a
Daniel Normanfdb38812019-04-15 09:47:24 -0700528 elif o == '--output-dir':
529 OPTIONS.output_dir = a
530 elif o == '--output-item-list':
531 OPTIONS.output_item_list = a
Daniel Norman3b64ce12019-04-16 16:11:35 -0700532 elif o == '--output-ota':
533 OPTIONS.output_ota = a
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700534 elif o == '--output-img':
535 OPTIONS.output_img = a
Daniel Normanf0318252019-04-15 11:34:56 -0700536 elif o == '--output-super-empty':
537 OPTIONS.output_super_empty = a
Daniel Norman2465fc82022-03-02 12:01:20 -0800538 elif o == '--rebuild_recovery' or o == '--rebuild-recovery':
Daniel Normana4911da2019-03-15 14:36:21 -0700539 OPTIONS.rebuild_recovery = True
Daniel Normanb0c75912020-09-24 14:30:21 -0700540 elif o == '--allow-duplicate-apkapex-keys':
541 OPTIONS.allow_duplicate_apkapex_keys = True
Daniel Norman571e1822021-06-25 17:18:25 -0700542 elif o == '--vendor-otatools':
543 OPTIONS.vendor_otatools = a
544 elif o == '--rebuild-sepolicy':
545 OPTIONS.rebuild_sepolicy = True
Bill Peckham364c1cc2019-03-29 18:27:23 -0700546 elif o == '--keep-tmp':
Bill Peckhamf753e152019-02-19 18:02:46 -0800547 OPTIONS.keep_tmp = True
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700548 elif o == '--framework-dexpreopt-config':
549 OPTIONS.framework_dexpreopt_config = a
550 elif o == '--framework-dexpreopt-tools':
551 OPTIONS.framework_dexpreopt_tools = a
552 elif o == '--vendor-dexpreopt-config':
553 OPTIONS.vendor_dexpreopt_config = a
Bill Peckhamf753e152019-02-19 18:02:46 -0800554 else:
555 return False
556 return True
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800557
Bill Peckhamf753e152019-02-19 18:02:46 -0800558 args = common.ParseOptions(
Daniel Normane5b134a2019-04-17 14:54:06 -0700559 sys.argv[1:],
560 __doc__,
Bill Peckhamf753e152019-02-19 18:02:46 -0800561 extra_long_opts=[
562 'system-target-files=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700563 'framework-target-files=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800564 'system-item-list=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700565 'framework-item-list=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800566 'system-misc-info-keys=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700567 'framework-misc-info-keys=',
Bill Peckhamf753e152019-02-19 18:02:46 -0800568 'other-target-files=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700569 'vendor-target-files=',
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800570 'other-item-list=',
Daniel Normand5d70ea2019-06-05 15:13:43 -0700571 'vendor-item-list=',
Bill Peckhamf753e152019-02-19 18:02:46 -0800572 'output-target-files=',
Daniel Normanfdb38812019-04-15 09:47:24 -0700573 'output-dir=',
574 'output-item-list=',
Daniel Norman3b64ce12019-04-16 16:11:35 -0700575 'output-ota=',
Daniel Norman1bd2a1d2019-04-18 12:32:18 -0700576 'output-img=',
Daniel Normanf0318252019-04-15 11:34:56 -0700577 'output-super-empty=',
Jose Galmes9c8f6eb2021-07-21 09:34:08 -0700578 'framework-dexpreopt-config=',
579 'framework-dexpreopt-tools=',
580 'vendor-dexpreopt-config=',
Daniel Normana4911da2019-03-15 14:36:21 -0700581 'rebuild_recovery',
Daniel Norman2465fc82022-03-02 12:01:20 -0800582 'rebuild-recovery',
Daniel Normanb0c75912020-09-24 14:30:21 -0700583 'allow-duplicate-apkapex-keys',
Daniel Norman571e1822021-06-25 17:18:25 -0700584 'vendor-otatools=',
585 'rebuild-sepolicy',
Bill Peckham364c1cc2019-03-29 18:27:23 -0700586 'keep-tmp',
Bill Peckhamf753e152019-02-19 18:02:46 -0800587 ],
588 extra_option_handler=option_handler)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800589
Tao Bao2ad4b822019-06-27 16:52:12 -0700590 # pylint: disable=too-many-boolean-expressions
Daniel Normand5d70ea2019-06-05 15:13:43 -0700591 if (args or OPTIONS.framework_target_files is None or
592 OPTIONS.vendor_target_files is None or
Daniel Normane5b134a2019-04-17 14:54:06 -0700593 (OPTIONS.output_target_files is None and OPTIONS.output_dir is None) or
Daniel Norman5f476772022-03-02 15:46:34 -0800594 (OPTIONS.output_dir is not None and not OPTIONS.output_item_list) or
Po Hua6c59122022-02-16 08:41:29 +0000595 (OPTIONS.rebuild_recovery and not OPTIONS.rebuild_sepolicy)):
Bill Peckhamf753e152019-02-19 18:02:46 -0800596 common.Usage(__doc__)
Bill Peckham889b0c62019-02-21 18:53:37 -0800597 sys.exit(1)
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800598
Dennis Song5bfa43e2023-03-30 18:28:00 +0800599 framework_namelist = merge_utils.GetTargetFilesItems(
600 OPTIONS.framework_target_files)
601 vendor_namelist = merge_utils.GetTargetFilesItems(
602 OPTIONS.vendor_target_files)
Daniel Norman5f476772022-03-02 15:46:34 -0800603
Daniel Normand5d70ea2019-06-05 15:13:43 -0700604 if OPTIONS.framework_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800605 OPTIONS.framework_item_list = common.LoadListFromFile(
606 OPTIONS.framework_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800607 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800608 OPTIONS.framework_item_list = merge_utils.InferItemList(
609 input_namelist=framework_namelist, framework=True)
Daniel Norman2465fc82022-03-02 12:01:20 -0800610 OPTIONS.framework_partition_set = merge_utils.ItemListToPartitionSet(
Daniel Norman03747412022-02-25 10:38:37 -0800611 OPTIONS.framework_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800612
Daniel Normand5d70ea2019-06-05 15:13:43 -0700613 if OPTIONS.framework_misc_info_keys:
Daniel Norman03747412022-02-25 10:38:37 -0800614 OPTIONS.framework_misc_info_keys = common.LoadListFromFile(
Daniel Normand5d70ea2019-06-05 15:13:43 -0700615 OPTIONS.framework_misc_info_keys)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800616 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800617 OPTIONS.framework_misc_info_keys = merge_utils.InferFrameworkMiscInfoKeys(
618 input_namelist=framework_namelist)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800619
Daniel Normand5d70ea2019-06-05 15:13:43 -0700620 if OPTIONS.vendor_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800621 OPTIONS.vendor_item_list = common.LoadListFromFile(OPTIONS.vendor_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800622 else:
Daniel Norman5f476772022-03-02 15:46:34 -0800623 OPTIONS.vendor_item_list = merge_utils.InferItemList(
624 input_namelist=vendor_namelist, framework=False)
Daniel Norman2465fc82022-03-02 12:01:20 -0800625 OPTIONS.vendor_partition_set = merge_utils.ItemListToPartitionSet(
Daniel Norman03747412022-02-25 10:38:37 -0800626 OPTIONS.vendor_item_list)
Daniel Norman2c99c5b2019-03-07 13:01:48 -0800627
Daniel Normanfdb38812019-04-15 09:47:24 -0700628 if OPTIONS.output_item_list:
Daniel Norman03747412022-02-25 10:38:37 -0800629 OPTIONS.output_item_list = common.LoadListFromFile(OPTIONS.output_item_list)
Daniel Normanfdb38812019-04-15 09:47:24 -0700630
Daniel Norman2465fc82022-03-02 12:01:20 -0800631 if not merge_utils.ValidateConfigLists():
Daniel Normane5964522019-03-19 10:32:03 -0700632 sys.exit(1)
633
Daniel Norman2465fc82022-03-02 12:01:20 -0800634 temp_dir = common.MakeTempDir(prefix='merge_target_files_')
635 try:
636 merge_target_files(temp_dir)
637 finally:
638 if OPTIONS.keep_tmp:
639 logger.info('Keeping temp_dir %s', temp_dir)
640 else:
641 common.Cleanup()
Bill Peckhame9eb5f92019-02-01 15:52:10 -0800642
643
644if __name__ == '__main__':
Bill Peckham889b0c62019-02-21 18:53:37 -0800645 main()