update_engine: De-duplicate zlib blocks of squashfs images
Having duplicate zlib blocks in the squashfs image (e.g. from two
hard linked files) will cause problems for processing puffin. Recently
ARC++ image introduced two files that where hard linked of each other
and this caused the squashfs parsing algorithm fail and the ARC++ image
does not get splitted into smaller files. Hence we got massive OOM and
Timeout problems in the builders. This CL, simpliy removes duplicate
zlib blocks when constructing the list of such items.
BUG=chromium:955145
TEST=cros_generate_update_payload --image gs://chromeos-releases/stable-channel/nautilus/11647.154.0/chromeos_11647.154.0_nautilus_recovery_stable-channel_mp-v2.bin --src_image gs://chromeos-releases/stable-channel/nautilus/10575.55.0/chromeos_10575.55.0_nautilus_recovery_stable-channel_mp.bin --output payload.bin
Change-Id: If4eb4731355e1180953a5b327be3a16ff13e6d96
Reviewed-on: https://chromium-review.googlesource.com/1578219
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
diff --git a/payload_generator/squashfs_filesystem.cc b/payload_generator/squashfs_filesystem.cc
index c423b69..e43bc0a 100644
--- a/payload_generator/squashfs_filesystem.cc
+++ b/payload_generator/squashfs_filesystem.cc
@@ -261,6 +261,14 @@
return a.offset < b.offset;
});
+ // Sometimes a squashfs can have a two files that are hard linked. In this
+ // case both files will have the same starting offset in the image and hence
+ // the same zlib blocks. So we need to remove these duplicates to eliminate
+ // further potential probems. As a matter of fact the next statement will
+ // fail if there are duplicates (there will be overlap between two blocks).
+ auto last = std::unique(zlib_blks.begin(), zlib_blks.end());
+ zlib_blks.erase(last, zlib_blks.end());
+
// Sanity check. Make sure zlib blocks are not overlapping.
auto result = std::adjacent_find(
zlib_blks.begin(),