vendor: import roomservice from android-7.1
build: tools: add in roomservice
This is a new implementation of roomservice
use the new github search api to reduce the memory footprint, and make it less processor intensive
allow for unofficial devices to fetch dependencies. just specify your github organization
to fetch dependencies:
add a file called 'omni.dependencies' to the root of your device tree
an example configuration for the dependency file is:
[
{
"repository": "android_device_sony_common",
"target_path": "device/sony/common"
},
{
"repository": "legacybop/android_hardware_qcom_display-legacy",
"target_path": "hardware/qcom/display-legacy",
"revision": "omni-4.3"
}
please note that comments can not go into this file. It must be valid json
PatchSet 7:
add support to change port script to another team with config
finished pep8'ing
more checks on manifest parsing
Change-Id: I3d7a8d46d3effbb3d2480c136c9b6b6e610b9b2e
Support fetching dependencies even if device tree fails.
This is required when device tree depends on dependencies to lunch.
Change-Id: If62cd9777caf768f8070570ca0835052645c6b20
roomservice: Fix os.path.isdir calls
Apparently on some Python installations
it throws exception instead of returning False.
Change-Id: I9d901e3260bab411cf3466346c421594f0922562
roomservice: the search api is out of preview now
Change-Id: I95f088daf8537784caf774ad6863037f7019f22d
roomservice: Add -f and --no-clone-bundle flags to repo sync
Suppress scary-looking common warnings during resolving.
Change-Id: I4337ec95992ad68ff4471b0d0b7346e933269503
roomservice: Change default remote to omnirom
Using the default of github results in people unable to use gerrit
to submit patches to repos brought in by roomservice
Change-Id: Idccd6c35bf81a35135e893ed03f6c5745059ec82
roomservice: Fix handling of devices with _ in their name
Currently, roomservice takes every _ in the device tree name and
replaces it with /
Instead, strip the device name (such as pollux_windy) from the repo
name before replacing _ with / and then append it again when we're done
Change-Id: Ieabe85ecd8829c959a7296e5cd73f577879c4ffc
roomservice: urgent: add "+fork:true"
Enable a device config to be found on github if it was forked
thx @cybojenix
Change-Id: Ic63f82b9ed754dcba17725f5aa75d0011896b09c
Roomservice: Search gerrit instead of github
* Search for devices using gerrit api instead of github
* Minor cleanups & fixes
Change-Id: I54fc898f3c773f79936568818996ae5fab11e491
Fix grammar error in roomservice; Only add xml files in local_manifests
Change-Id: Ia132acddd2761ec4c1bea7fb510e3bea306ea9fc
roomservice: support same repo using different path and revision
Change-Id: Ifd47cbeab2ca7e99a117883e3a3ecb880581e110
roomservice: catch revision changes for existing entries
delete existing roomservice file and restart in that case
Change-Id: I2adc62da01fb75cd8ee5b13b5864043fb2594d0f
roomservice: dynamically remove duplicate projects from roomservice manifest
Change-Id: Idf4da69ca829fcd59d5264b05b9ed1c37d0195f8
roomservice: Handling URLError exception during fetch device process
BEFORE:
humberos@fedora:/android/omni-6.0$ bib leo
WARNING: Trying to fetch a device that's already there
Traceback (most recent call last):
File "build/tools/roomservice.py", line 319, in <module>
fetch_dependencies(device)
File "build/tools/roomservice.py", line 275, in fetch_dependencies
fetch_device(device)
File "build/tools/roomservice.py", line 287, in fetch_device
git_data = search_gerrit_for_device(device)
File "build/tools/roomservice.py", line 71, in search_gerrit_for_device
response = urllib.request.urlopen(git_req)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1242, in https_open
context=self._context)
File "/usr/lib64/python2.7/urllib2.py", line 1199, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -5] No address associated with hostname>
AFTER:
humberos@fedora:/android/omni-6.0$ bib leo
WARNING: Trying to fetch a device that's already there
WARNING: No network connection available.
Signed-off-by: Humberto Borba <humberos@gmail.com>
Change-Id: I18950704c5b7d13553374611d164c2464c1ceab2
roomservice: Create manifest entries from common repos
With this change we will be able to read omni.dependencies from common repositories.
Signed-off-by: Humberto Borba <humberos@gmail.com>
Change-Id: Ic66ceead3601263df0c0e181a45b9ef7e7a32b77
roomservice: used 8.0 for default rev
Change-Id: Idfc79aa8ed36bcafbfce134d2ec71a93e00ce66a
diff --git a/build/tools/roomservice.py b/build/tools/roomservice.py
new file mode 100755
index 0000000..9e43bd8
--- /dev/null
+++ b/build/tools/roomservice.py
@@ -0,0 +1,353 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2013 Cybojenix <anthonydking@gmail.com>
+# Copyright (C) 2013 The OmniROM Project
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import print_function
+import json
+import sys
+import os
+import os.path
+import re
+from xml.etree import ElementTree as ES
+# Use the urllib importer from the Cyanogenmod roomservice
+try:
+ # For python3
+ import urllib.request
+except ImportError:
+ # For python2
+ import imp
+ import urllib2
+ urllib = imp.new_module('urllib')
+ urllib.request = urllib2
+
+# Config
+# set this to the default remote to use in repo
+default_rem = "omnirom"
+# set this to the default revision to use (branch/tag name)
+default_rev = "android-8.0"
+# set this to the remote that you use for projects from your team repos
+# example fetch="https://github.com/omnirom"
+default_team_rem = "omnirom"
+# this shouldn't change unless google makes changes
+local_manifest_dir = ".repo/local_manifests"
+# change this to your name on github (or equivalent hosting)
+android_team = "omnirom"
+# url to gerrit repository
+gerrit_url = "gerrit.omnirom.org"
+
+
+def check_repo_exists(git_data, device):
+ re_match = "^android_device_.*_{device}$".format(device=device)
+ matches = filter(lambda x: re.match(re_match, x), git_data)
+ if len(matches) != 1:
+ raise Exception("{device} not found,"
+ "exiting roomservice".format(device=device))
+
+ return git_data[matches[0]]
+
+
+def search_gerrit_for_device(device):
+ # TODO: In next gerrit release regex search with r= should be supported!
+ git_search_url = "https://{gerrit_url}/projects/?m={device}".format(
+ gerrit_url=gerrit_url,
+ device=device
+ )
+ git_req = urllib.request.Request(git_search_url)
+ try:
+ response = urllib.request.urlopen(git_req)
+ except urllib.request.HTTPError as e:
+ print("There was an issue connecting to gerrit."
+ " Please try again in a minute")
+ except urllib.request.URLError as e:
+ print("WARNING: No network connection available.")
+ else:
+ # Skip silly gerrit "header"
+ response.readline()
+ git_data = json.load(response)
+ device_data = check_repo_exists(git_data, device)
+ print("found the {} device repo".format(device))
+ return device_data
+
+
+def parse_device_directory(device_url, device):
+ pattern = "^android_device_(?P<vendor>.+)_{}$".format(device)
+ match = re.match(pattern, device_url)
+
+ if match is None:
+ raise Exception("Invalid project name {}".format(device_url))
+ return "device/{vendor}/{device}".format(
+ vendor=match.group('vendor'),
+ device=device,
+ )
+
+
+# Thank you RaYmAn
+def iterate_manifests():
+ files = []
+ for file in os.listdir(local_manifest_dir):
+ if file.endswith(".xml"):
+ files.append(os.path.join(local_manifest_dir, file))
+ files.append('.repo/manifest.xml')
+ for file in files:
+ try:
+ man = ES.parse(file)
+ man = man.getroot()
+ except IOError, ES.ParseError:
+ print("WARNING: error while parsing %s" % file)
+ else:
+ for project in man.findall("project"):
+ yield project
+
+
+def check_project_exists(url, revision, path):
+ for project in iterate_manifests():
+ if project.get("name") == url and project.get("revision") == revision and project.get("path") == path:
+ return True
+ return False
+
+
+def check_target_exists(directory):
+ return os.path.isdir(directory)
+
+
+# Use the indent function from http://stackoverflow.com/a/4590052
+def indent(elem, level=0):
+ i = ''.join(["\n", level*" "])
+ if len(elem):
+ if not elem.text or not elem.text.strip():
+ elem.text = ''.join([i, " "])
+ if not elem.tail or not elem.tail.strip():
+ elem.tail = i
+ for elem in elem:
+ indent(elem, level+1)
+ if not elem.tail or not elem.tail.strip():
+ elem.tail = i
+ else:
+ if level and (not elem.tail or not elem.tail.strip()):
+ elem.tail = i
+
+
+def create_manifest_project(url, directory,
+ remote=default_rem,
+ revision=default_rev):
+ project_exists = check_project_exists(url, revision, directory)
+
+ if project_exists:
+ return None
+
+ project = ES.Element("project",
+ attrib={
+ "path": directory,
+ "name": url,
+ "remote": remote,
+ "revision": revision
+ })
+ return project
+
+
+def append_to_manifest(project):
+ try:
+ lm = ES.parse('/'.join([local_manifest_dir, "roomservice.xml"]))
+ lm = lm.getroot()
+ except IOError, ES.ParseError:
+ lm = ES.Element("manifest")
+ lm.append(project)
+ return lm
+
+
+def write_to_manifest(manifest):
+ indent(manifest)
+ raw_xml = ES.tostring(manifest).decode()
+ raw_xml = ''.join(['<?xml version="1.0" encoding="UTF-8"?>\n'
+ '<!--Please do not manually edit this file-->\n',
+ raw_xml])
+
+ with open('/'.join([local_manifest_dir, "roomservice.xml"]), 'w') as f:
+ f.write(raw_xml)
+ print("wrote the new roomservice manifest")
+
+
+def parse_device_from_manifest(device):
+ for project in iterate_manifests():
+ name = project.get('name')
+ if name.startswith("android_device_") and name.endswith(device):
+ return project.get('path')
+ return None
+
+
+def parse_device_from_folder(device):
+ search = []
+ for sub_folder in os.listdir("device"):
+ if os.path.isdir("device/%s/%s" % (sub_folder, device)):
+ search.append("device/%s/%s" % (sub_folder, device))
+ if len(search) > 1:
+ print("multiple devices under the name %s. "
+ "defaulting to checking the manifest" % device)
+ location = parse_device_from_manifest(device)
+ elif len(search) == 1:
+ location = search[0]
+ else:
+ print("Your device can't be found in device sources..")
+ location = parse_device_from_manifest(device)
+ return location
+
+
+def parse_dependency_file(location):
+ dep_file = "omni.dependencies"
+ dep_location = '/'.join([location, dep_file])
+ if not os.path.isfile(dep_location):
+ print("WARNING: %s file not found" % dep_location)
+ sys.exit()
+ try:
+ with open(dep_location, 'r') as f:
+ dependencies = json.loads(f.read())
+ except ValueError:
+ raise Exception("ERROR: malformed dependency file")
+ return dependencies
+
+# if there is any conflict with existing and new
+# delete the roomservice.xml file and create new
+def check_manifest_problems(dependencies):
+ for dependency in dependencies:
+ repository = dependency.get("repository")
+ target_path = dependency.get("target_path")
+ revision = dependency.get("revision", default_rev)
+ remote = dependency.get("remote", default_rem)
+
+ # check for existing projects
+ for project in iterate_manifests():
+ if project.get("path") == target_path and project.get("revision") != revision:
+ print("WARNING: detected conflict in revisions for repository ", repository)
+ current_dependency = str(project.get(repository))
+ file = ES.parse('/'.join([local_manifest_dir, "roomservice.xml"]))
+ file_root = file.getroot()
+ for current_project in file_root.findall('project'):
+ new_dependency = str(current_project.find('revision'))
+ if new_dependency == current_dependency:
+ file_root.remove(current_project)
+ file.write('/'.join([local_manifest_dir, "roomservice.xml"]))
+ return
+
+def create_dependency_manifest(dependencies):
+ projects = []
+ for dependency in dependencies:
+ repository = dependency.get("repository")
+ target_path = dependency.get("target_path")
+ revision = dependency.get("revision", default_rev)
+ remote = dependency.get("remote", default_rem)
+
+ # not adding an organization should default to android_team
+ # only apply this to github
+ if remote == "github":
+ if "/" not in repository:
+ repository = '/'.join([android_team, repository])
+ project = create_manifest_project(repository,
+ target_path,
+ remote=remote,
+ revision=revision)
+ if project is not None:
+ manifest = append_to_manifest(project)
+ write_to_manifest(manifest)
+ projects.append(target_path)
+ if len(projects) > 0:
+ os.system("repo sync -f --no-clone-bundle %s" % " ".join(projects))
+
+
+def create_common_dependencies_manifest(dependencies):
+ dep_file = "omni.dependencies"
+ common_list = []
+ if dependencies is not None:
+ for dependency in dependencies:
+ try:
+ index = common_list.index(dependency['target_path'])
+ except ValueError:
+ index = None
+ if index is None:
+ common_list.append(dependency['target_path'])
+ dep_location = '/'.join([dependency['target_path'], dep_file])
+ if not os.path.isfile(dep_location):
+ sys.exit()
+ else:
+ try:
+ with open(dep_location, 'r') as f:
+ common_deps = json.loads(f.read())
+ except ValueError:
+ raise Exception("ERROR: malformed dependency file")
+
+ if common_deps is not None:
+ print("Looking for dependencies on: ",
+ dependency['target_path'])
+ check_manifest_problems(common_deps)
+ create_dependency_manifest(common_deps)
+ create_common_dependencies_manifest(common_deps)
+
+
+def fetch_dependencies(device):
+ location = parse_device_from_folder(device)
+ if location is None or not os.path.isdir(location):
+ raise Exception("ERROR: could not find your device "
+ "folder location, bailing out")
+ dependencies = parse_dependency_file(location)
+ check_manifest_problems(dependencies)
+ create_dependency_manifest(dependencies)
+ create_common_dependencies_manifest(dependencies)
+ fetch_device(device)
+
+def check_device_exists(device):
+ location = parse_device_from_folder(device)
+ if location is None:
+ return False
+ return os.path.isdir(location)
+
+
+def fetch_device(device):
+ if check_device_exists(device):
+ print("WARNING: Trying to fetch a device that's already there")
+ git_data = search_gerrit_for_device(device)
+ if git_data is not None:
+ device_url = git_data['id']
+ device_dir = parse_device_directory(device_url, device)
+ project = create_manifest_project(device_url,
+ device_dir,
+ remote=default_team_rem)
+ if project is not None:
+ manifest = append_to_manifest(project)
+ write_to_manifest(manifest)
+ # In case a project was written to manifest, but never synced
+ if project is not None or not check_target_exists(device_dir):
+ print("syncing the device config")
+ os.system('repo sync -f --no-clone-bundle %s' % device_dir)
+
+
+if __name__ == '__main__':
+ if not os.path.isdir(local_manifest_dir):
+ os.mkdir(local_manifest_dir)
+
+ product = sys.argv[1]
+ try:
+ device = product[product.index("_") + 1:]
+ except ValueError:
+ device = product
+
+ if len(sys.argv) > 2:
+ deps_only = sys.argv[2]
+ else:
+ deps_only = False
+
+ if not deps_only:
+ fetch_device(device)
+ fetch_dependencies(device)