Index: /package_tags/invirt-dev/0.1.1/README =================================================================== --- /package_tags/invirt-dev/0.1.1/README (revision 3048) +++ /package_tags/invirt-dev/0.1.1/README (revision 3048) @@ -0,0 +1,10 @@ +Installation: + + - The secret of the gpg key identified in invirt.config.apt.keyid + must be in the keyring of the repository user. + + The public key should then be in the apt keyring of systems using + the packages. + + - Packages should be located at + /srv/git/invirt/packages/${packagename}.git Index: /package_tags/invirt-dev/0.1.1/README.invirtibuilder =================================================================== --- /package_tags/invirt-dev/0.1.1/README.invirtibuilder (revision 3048) +++ /package_tags/invirt-dev/0.1.1/README.invirtibuilder (revision 3048) @@ -0,0 +1,397 @@ +============================ +Design of The Invirtibuilder +============================ + +Introduction +============ + +The Invirtibuilder is an automated Debian package builder, APT +repository manager, and Git repository hosting tool. It is intended +for projects that consist of a series of Debian packages each tracked +as a separate Git repository, and designed to keep the Git and APT +repositories in sync with each other. The Invirtibuilder supports +having multiple threads, or "pockets" of development, and can enforce +different access control and repository consistency rules for each +pocket. + +Background and Goals +==================== + +The Invirtibuilder was originally developed for Invirt_, a project of +the MIT SIPB_. When we went to develop a tool for managing our APT and +Git repositories, we had several goals, each of which informed the +design of the Invirtibuilder: + +* One Git repository per Debian package. + + Because of how Git tracks history, it's better suited for tracking a + series of small repositories, as opposed to one large one + [#]_. Furthermore, most preexisting tools and techniques for dealing + with Debian packages in Git repositories (such as git-buildpackage_ + or `VCS location information`_) are designed exclusively for this + case. + +* Synchronization between Git and APT repositories. + + In our previous development models, we would frequently merge + development into trunk without necessarily being ready to deploy it + to our APT repository (and by extension, our servers) yet. However, + once the changes had been merged in, it was no longer possible to + see the current state of the APT repository purely from inspection + of the source control repository. + +* Support for multiple *pockets* of development. + + For the Invirt_ project, we maintain separate production and + development environments. Initially, they each shared the same APT + repository. To test changes, we had to install them into the APT + repository and install the update on our development cluster, and + simply wait to take the update on our production cluster until + testing was completed. When designing the Invirtibuilder, we wanted + the set of packages available to our development cluster to be + separate from the packages in the production cluster. + +* Different ACLs for different pockets. + + Access to our development cluster is relatively unrestricted—we + freely grant access to interested developers to encourage + contributions to the project. Our production cluster, on the other + hand, has a much higher standard of security, and access is limited + to the core maintainers of the service. The Invirtibuilder needed to + support that separation of privilege. + +* Tool-enforced version number restrictions. + + Keeping our packages in APT repositories adds a few restrictions to + the version numbers of packages. First, version numbers in the APT + repository must be unique. That is, you can not have two different + packages of the same name and version number. Second, version + numbers are expected to be monotonically increasing. If a newer + version of a package had a lower version number than the older + version, dpkg would consider this a downgrade. Downgrades are not + supported by dpkg, and will not even be attempted by APT. + + In order to avoid proliferation of version numbers used only for + testing purposes, we opted to bend the latter rule for our + development pocket. + +* Tool-enforced consistent history. + + In order for the Git history to be meaningful, we chose to require + that each version of a package that is uploaded into the APT + repository be a fast-forward of the previous version. + + Again, to simplify and encourage testing, we bend this rule for the + development pocket as well. + +Design +====== + +Configuration +------------- + +For the Invirt_ project's use of the Invirtibuilder, we adapted our +existing configuration mechanism. Our configuration file consists of a +single YAML_ file. Here is the snippet of configuration we use for our +build configuration:: + + build: + pockets: + prod: + acl: system:xvm-root + apt: stable + dev: + acl: system:xvm-dev + apt: unstable + allow_backtracking: yes + tagger: + name: Invirt Build Server + email: invirt@mit.edu + +The Invirtibuilder allows naming Invirtibuilder pockets separately +form their corresponding Git branches or APT components. However, if +either the ``git`` or ``apt`` properties of the pocket are +unspecified, they are assumed to be the same as the name of the +pocket. + +The ``acl`` attributes for each pocket are interpreted within our +authorization modules to determine who is allowed to request builds on +a given pocket. ``system:xvm-root`` and ``system:xvm-dev`` are the +names of AFS groups, which we use for authorization. + +The ``tagger`` attribute indicates the name and e-mail address to be +used whenever the Invirtibuilder generates new Git repository objects, +such as commits or tags. + +Finally, it was mentioned in `Background and Goals`_ that we wanted +the ability to not force version number consistency or Git +fast-forwards for our development pocket. The ``allow_backtracking`` +attribute was introduced to indicate that preference. When it is set +to ``yes`` (i.e. YAML's "true" value), then neither fast-forwards nor +increasing-version-numbers are enforced when validating builds. The +attribute is assumed to be false if undefined. + +Git Repositories +---------------- + +In order to make it easy to check out all packages at once, and for +version controlling the state of the APT repository, we create a +"superproject" using Git submodules [#]_. + +There is one Git branch in the superproject corresponding to each +pocket of development. Each branch contains a submodule for each +package in the corresponding component of the APT repository, and the +submodule commit referred to by the head of the Git branch matches the +revision of the package currently in the corresponding component of +the APT repository. Thus, the heads of the Git superproject match the +state of the components in the APT repository. + +Each of the submodules also has a branch for each pocket. The head of +that branch points to the revision of the package that is currently in +the corresponding component of the APT repository. This provides a +convenient branching point for new development. Additionally, there is +a Git tag for every version of the package that has ever been uploaded +to the APT repository. + +Because the Invirtibuilder and its associated infrastructure are +responsible for keeping the superproject in sync with the state of the +APT repository, an update hook disallows all pushes to the +superproject. + +Pushes to the submodules, on the other hand, are almost entirely +unrestricted. Like with the superproject, the Git branches for each +pocket and Git tags are maintained by the build infrastructure, so +pushes to them are disallowed. Outside of that, we make no +restrictions on the creation or deletion of branches, nor are pushes +required to be fast-forwards. + +The Build Queue +--------------- + +We considered several ways to trigger builds of new package versions +using Git directly. However, we realized that what we actually wanted +was a separate build queue where each build request was handled and +processed independently of any requests before or after it. It's not +possible to have these semantics using Git as a signaling mechanism +without breaking standard assumptions about how remote Git +repositories work. + +In order to trigger builds, then, we needed a side-channel. Since it +was already widely used in the Invirt_ project, we chose to use +remctl_, a GSSAPI-authenticated RPC protocol with per-command ACLs. + +To trigger a new build, a developer calls remctl against the build +server with a pocket, a package, and a commit ID from that package's +Git repository. The remctl daemon then calls a script which validates +the build and adds it to the build queue. Because of the structure of +remctl's ACLs, we are able to have different ACLs depending on which +pocket the build is destined for. This allows us to fulfill our design +goal of having different ACLs for different pockets. + +For simplicity, the queue itself is maintained as a directory of +files, where each file is a queue entry. To maintain order in the +queue, the file names for queue entries are of the form +``YYYYMMDDHHMMSS_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX``, where ``X`` +indicates a random hexadecimal digit. Each file contains the +parameters passed in over remctl (pocket, package, and commit ID to +build), as well as the Kerberos principal of the user that requested +the build, for logging. + +The Build Daemon +---------------- + +To actually execute builds, we run a separate daemon to monitor for +new build requests in the build queue. The daemon uses inotify so that +it's triggered whenever a new item is added to the build +queue. Whenever an item in the build queue triggers the build daemon, +the daemon first validates the build, then executes the build, and +finally updates both the APT repository and Git superproject with the +results of the build. The results of all attempted builds are recorded +in a database table for future reference. + +Build Validation +```````````````` + +The first stage of processing a new build request is validating the +build. First, the build daemon checks the version number of the +requested package in each pocket of the repository. If the package is +present in any other pocket with the same version number, but the Git +commit for the package is different, the build errors out, because it +is not possible for an APT repository to contain two different +packages with the same name and version number. + +Next, the build daemon checks to make sure that the version number of +the new package is a higher version number than the version currently +in the APT repository, as version numbers must be monotonically +increasing. + +Finally, we require new packages to be fast-forwards in Git of the +previous version of the package. This is verified as well. + +As mentioned above, the ``allow_backtracking`` attribute can be set +for a pocket to bypass the latter two checks in development +environments. + +When the same package with the same version is inserted into multiple +places in the same APT repository, the MD5 hash of the package is used +to validate that it hasn't changed. Because rebuilding the same +package causes the MD5 hash to change, when a version of a package +identical to a version already in the APT repository is added to +another pocket, we need to copy it directly. Since the validation +stage already has all of the necessary information to detect this +case, if the same version of a package is already present in another +pocket, the validation stage returns this information. + +Build Execution +``````````````` + +Once the build has been validated, it can be executed. The requested +version of the package is exported from Git, and then a Debian source +package is generated. Next, the package itself is built using sbuild. + +sbuild creates an ephemeral build chroot for each build that has only +essential build packages and the build dependencies for the package +being built installed. We use sbuild for building packages for several +reasons. First, it helps us verify that all necessary build +dependencies have been included in our packages. Second, it helps us +ensure that configuration files haven't been modified from their +upstream defaults (which could cause problems for packages using +config-package-dev_). + +The build daemon keeps the build logs from all attempted builds on the +filesystem for later inspection. + +Repository Updates +`````````````````` + +Once the build has been successfully completed, the APT and Git +repositories are updated to match the new state. First, a new tag is +added to the package's Git repository for the current version +[#]_. Next, the pocket tracking branch in the submodule is also +updated with the new version of the package. Then the a new commit is +created on the superproject which updates the package's submodule to +point to the new version of the package. Finally, the new version of +the package is included in the appropriate component of the APT +repository. + +Because the Git superproject, the Git submodules, and the APT +repository are all updated simultaneously to reflect the new package +version, the Git repositories and the APT repository always stay in +sync. + +Build Failures +`````````````` + +If any of the above stages of executing a build fail, that failure is +trapped and recorded for later inspection, and recorded along with the +build record in the database. Regardless of success or failure, the +build daemon runs any scripts in a hook directory. The hook directory +could contain scripts to publish the results of the build in whatever +way is deemed useful by the developers. + +Security +======== + +As noted above, our intent was for a single instance of the +Invirtibuilder to be used for both our trusted production environment +and our untrusted development environment. In order to be trusted for +the production environment, the Invirtibuilder needs to run in the +production environment as well. However, it would be disastrous if +access to the development environment allowed a developer to insert +malicious packages into the production apt repository. + +In terms of policy, we enforce this distinction using the remctl ACL +mechanism described in `The Build Queue`_. But is that mechanism on +its own actually secure? + +Only mostly, it turns out. + +While actual package builds run unprivileged (with the help of the +fakeroot_ tool), packages can declare arbitrary build dependencies +that must be installed for the package build to run. Packages' +maintainer scripts (post-install, pre-install, pre-removal, and +post-removal scripts) run as root. This means that by uploading a +malicious package that another package build-depends on, then +triggering a build of the second package, it is possible to gain root +privileges. Since breaking out of the build chroot as root is trivial +[#], it is theoretically possible for developers with any level of +access to the APT repositories to root the build server. + +One minor protection from this problem is the Invirtibuilder's +reporting mechanism. A single independent malicious build can't +compromise the build server on its own. Even if a second build +compromises the build server, the first build will have already been +reported through the hook mechanism described in `Build Failures`_. We +encourage users of the Invirtibuilder to include hooks that send +notifications of builds over e-mail or some other mechanism such that +there are off-site records. The server will still be compromised, but +there will be an audit trail. + +Such a vulnerability will always be a concern so long as builds are +isolated using chroots. It is possible to protect against this sort of +attack by strengthening the chroot mechanism (e.g. with grsecurity_) +or by using a more isolated build mechanism +(e.g. qemubuilder_). However, we decided that the security risk didn't +justify the additional implementation effort or runtime overhead. + +Future Directions +================= + +While the Invirtibuilder was written as a tool for the Invirt_ +project, taking advantage of infrastructure specific to Invirt, it was +designed with the hope that it could one day be expanded to be useful +outside of our infrastructure. Here we outline what we believe the +next steps for development of the Invirtibuilder are. + +One deficiency that affects Invirt_ development already is the +assumption that all packages are Debian-native [#]. Even for packages +which have a non-native version number, the Invirtibuilder will create +a Debian-native source package when the package is exported from Git +as part of the `Build Execution`_. Correcting this requires a means to +find and extract the upstream tarball from the Git repository. This +could probably be done by involving the pristine-tar_ tool. + +The Invirtibuilder is currently tied to the configuration framework +developed for the Invirt_ project. To be useful outside of Invirt, the +Invirtibuilder needs its own, separate mechanism for providing and +parsing configuration. It should not be difficult to use a separate +configuration file but a similar YAML configuration mechanism for the +Invirtibuilder. And of course, as part of that process, filesystem +paths and the like that are currently hard-coded should be replaced +with configuration options. + +The Invirtibuilder additionally relies on the authentication and +authorization mechanisms used for Invirt_. Our RPC protocol of choice, +remctl_, requires a functional Kerberos environment for +authentication, limiting its usefulness for one-off projects not +associated with an already existing Kerberos realm. We would like to +provide support for some alternative RPC mechanism—possibly +ssh. Additionally, there needs to be some way to expand the build ACLs +for each pocket that isn't tied to Invirt's authorization +framework. One option would be providing an executable in the +configuration that, when passed a pocket as a command-line argument, +prints out all of the principals that should have access to that +pocket. + +.. _config-package-dev: http://debathena.mit.edu/config-packages +.. _fakeroot: http://fakeroot.alioth.debian.org/ +.. _git-buildpackage: https://honk.sigxcpu.org/piki/projects/git-buildpackage/ +.. _grsecurity: http://www.grsecurity.net/ +.. _Invirt: http://invirt.mit.edu +.. _pristine-tar: http://joey.kitenet.net/code/pristine-tar/ +.. _qemubuilder: http://wiki.debian.org/qemubuilder +.. _remctl: http://www.eyrie.org/~eagle/software/remctl/ +.. _SIPB: http://sipb.mit.edu +.. _VCS location information: http://www.debian.org/doc/developers-reference/best-pkging-practices.html#bpp-vcs +.. _YAML: http://yaml.org/ + +.. [#] http://lwn.net/Articles/246381/ +.. [#] A Git submodule is a second Git repository embedded at a + particular path within the superproject and fixed at a + particular commit. +.. [#] Because we don't force any sort of version consistency for + pockets with ``allow_backtracking`` set to ``True``, we don't + create new tags for builds on pockets with + ``allow_backtracking`` set to ``True`` either. +.. [#] http://kerneltrap.org/Linux/Abusing_chroot +.. [#] http://people.debian.org/~mpalmer/debian-mentors_FAQ.html#native_vs_non_native Index: /package_tags/invirt-dev/0.1.1/build-hooks/failed-build =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/failed-build (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/failed-build (revision 3048) @@ -0,0 +1,1 @@ +link post-build Index: /package_tags/invirt-dev/0.1.1/build-hooks/failed-submit =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/failed-submit (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/failed-submit (revision 3048) @@ -0,0 +1,35 @@ +#!/bin/sh + +set -e +set -u + +escape() { + echo "$1" | sed -e 's/@/@@/g' +} + +pocket=$(escape "$1") +package=$(escape "$2") +commit=$(escape "$3") +principal=$(escape "$4") + +base=build.hooks.failed_submit.zephyr +class=$(invirt-getconf "$base.class" 2>/dev/null || :) +instance=$(invirt-getconf "$base.instance" 2>/dev/null || :) +zsig=$(invirt-getconf "$base.zsig" 2>/dev/null || :) + +if [ -z "$class" ]; then + echo "I don't know where to send a commit zephyr!" >&2 + echo "Please provide a value for $base.class in" >&2 + echo "your invirt config file." >&2 + exit 1 +fi + +(echo "A new job has @{@color(red)failed} to be submitted to the Invirtibuilder:" + echo + echo "pocket: $pocket" + echo "package: $package" + echo "commit: $commit"k + echo "principal: $principal" + echo + echo -n "Failure: "; + ( cat | sed -e 's/@/@@/g' ) ) | zwrite -c "$class" -i "${instance:-$commit}" -s "${zsig:-failed-submit}: $pocket" -d Index: /package_tags/invirt-dev/0.1.1/build-hooks/post-add-repo =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/post-add-repo (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/post-add-repo (revision 3048) @@ -0,0 +1,26 @@ +#!/bin/sh + +set -e +set -u + +escape() { + echo "$1" | sed -e 's/@/@@/g' +} + +category=$(escape "$1") +name=$(escape "$2") +principal=$(escape "$3") + +base=build.hooks.post_add_repo.zephyr +class=$(invirt-getconf "$base.class" 2>/dev/null || :) +instance=$(invirt-getconf "$base.instance" 2>/dev/null || :) +zsig=$(invirt-getconf "$base.zsig" 2>/dev/null || :) + +if [ -z "$class" ]; then + echo "I don't know where to send a commit zephyr!" >&2 + echo "Please provide a value for $base.class in" >&2 + echo "your invirt config file." >&2 + exit 1 +fi + +(echo "$principal just created a new repository, $category/$name.git") | zwrite -c "$class" -i "${instance:-add-repo}" -s "${zsig:-Make a new repo}" -d Index: /package_tags/invirt-dev/0.1.1/build-hooks/post-build =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/post-build (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/post-build (revision 3048) @@ -0,0 +1,157 @@ +#!/usr/bin/env python +""" +A script for reporting the results of an invirtibuild. Supports any +combination of the following reporting mechanisms. + +Note that all strings are interpolated with a dictionary containing +the following keys: + +build_id, commit, failed_stage, inserted_at, package, +pocket, principal, result, short_commit, traceback, version + +== zephyr == + +To configure zephyr, add something like the following to your invirt config: + +build: + hooks: + post_build: + zephyr: &post_build_zepyhr + class: myclass [required] + instance: myinstance [optional] + zsig: myzsig [optional] + failed_build: + zephyr: *post_build_zephyr + +== mail == + +To configure email notifications, add something like the following to your invirt config: + +build: + hooks: + post_build: + mail: &post_build_mail + to: myemail@example.com [required] + from: myemail@example.com [required] + subject: My Subject [optional] + failed_build: + mail: *post_build_mail + +post_build values will be used when this script is invoked as +post-build, while failed_build values will be used when it is invoked +as failed-build. +""" + +import optparse +import os +import re +import sys +import textwrap + +from email.mime import text + +from invirt import common, database, builder +from invirt.config import structs as config + +def make_msg(build, values, verbose=True, success=lambda x: x, failure=lambda x: x): + values = dict(values) + if not verbose and values['traceback'] is not None: + # TODO: better heuristic + values['traceback'] = textwrap.fill('\n'.join(values['traceback'].split('\n')[-2:])) + + if build.succeeded: + values['result'] = success(values['result']) + msg = """Build of %(package)s v%(version)s in %(pocket)s %(result)s. + +Branch %(pocket)s has been advanced to %(short_commit)s. + +(Build %(build_id)s was submitted by %(principal)s at %(inserted_at)s.)""" % values + else: + values['result'] = failure(values['result']) + msg = """Build of %(package)s v%(version)s in %(pocket)s %(result)s while %(failed_stage)s. + +%(traceback)s + +(Build %(build_id)s was submitted by %(principal)s at %(inserted_at)s.)""" % values + return msg + +def zephyr_escape(m): + m = re.sub('@', '@@', m) + m = re.sub('}', '@(})', m) + return m + +def zephyr_success(m): + return '}@{@color(green)%s}@{' % zephyr_escape(m) + +def zephyr_failure(m): + return '}@{@color(red)%s}@{' % zephyr_escape(m) + +def main(): + parser = optparse.OptionParser('Usage: %prog build_id') + opts, args = parser.parse_args() + if len(args) != 1: + parser.print_help() + return 1 + prog = os.path.basename(sys.argv[0]) + + database.connect() + build = database.Build.query().get(args[0]) + short_commit = builder.canonicalize_commit(build.package, build.commit, shorten=True) + values = { 'build_id' : build.build_id, + 'commit' : build.commit, + 'failed_stage' : build.failed_stage, + 'inserted_at' : build.inserted_at, + 'package' : build.package, + 'pocket' : build.pocket, + 'principal' : build.principal, + 'short_commit' : short_commit, + 'traceback' : build.traceback, + 'version' : build.version } + if build.succeeded: + values['result'] = 'succeeded' + else: + values['result'] = 'failed' + + try: + if prog == 'post-build': + hook_config = config.build.hooks.post_build + elif prog == 'failed-build': + hook_config = config.build.hooks.failed_build + else: + print >>sys.stderr, '{post,failed}-build invoke with unrecognized name %s' % prog + return 2 + except common.InvirtConfigError: + print >>sys.stderr, 'No hook configuration found for %s.' % prog + return 1 + + try: + zephyr_config = hook_config.zephyr + klass = zephyr_config['class'] % values + except common.InvirtConfigError: + print >>sys.stderr, 'No zephyr configuration specified for %s.' % prog + else: + msg = '@{%s}' % make_msg(build, values, verbose=False, + success=zephyr_success, failure=zephyr_failure) + instance = zephyr_config.get('instance', 'build_%(build_id)s') % values + zsig = zephyr_config.get('zsig', 'XVM Buildbot') % values + common.captureOutput(['zwrite', '-c', klass, '-i', instance, '-s', + zsig, '-d', '-m', msg], + stdout=None, stderr=None) + + try: + mail_config = hook_config.mail + to = mail_config.to % values + sender = mail_config['from'] % values + except common.InvirtConfigError: + print >>sys.stderr, 'No email configuration specified for %s.' % prog + else: + msg = make_msg(build, values) + email = text.MIMEText(msg) + email['To'] = to % values + email['From'] = sender % values + email['Subject'] = mail_config.get('subject', 'XVM build %(build_id)s has %(result)s') % values + common.captureOutput(['sendmail', '-t'], email.as_string(), + stdout=None, stderr=None) + +if __name__ == '__main__': + sys.exit(main()) Index: /package_tags/invirt-dev/0.1.1/build-hooks/post-submit =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/post-submit (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/post-submit (revision 3048) @@ -0,0 +1,32 @@ +#!/bin/sh + +set -e +set -u + +escape() { + echo "$1" | sed -e 's/@/@@/g' +} + +pocket=$(escape "$1") +package=$(escape "$2") +commit=$(escape "$3") +principal=$(escape "$4") + +base=build.hooks.post_submit.zephyr +class=$(invirt-getconf "$base.class" 2>/dev/null || :) +instance=$(invirt-getconf "$base.instance" 2>/dev/null || :) +zsig=$(invirt-getconf "$base.zsig" 2>/dev/null || :) + +if [ -z "$class" ]; then + echo "I don't know where to send a commit zephyr!" >&2 + echo "Please provide a value for $base.class in" >&2 + echo "your invirt config file." >&2 + exit 1 +fi + +(echo "A new job has been submitted to the Invirtibuilder:"; + echo; + echo "pocket: $pocket"; + echo "package: $package"; + echo "commit: $commit"; + echo "principal: $principal") | zwrite -c "$class" -i "${instance:-$commit}" -s "${zsig:-Git}: $pocket" -d Index: /package_tags/invirt-dev/0.1.1/build-hooks/pre-build =================================================================== --- /package_tags/invirt-dev/0.1.1/build-hooks/pre-build (revision 3048) +++ /package_tags/invirt-dev/0.1.1/build-hooks/pre-build (revision 3048) @@ -0,0 +1,33 @@ +#!/bin/sh + +set -e +set -u + +escape() { + echo "$1" | sed -e 's/@/@@/g' +} + +build_id=$(escape "$1") +pocket=$(escape "$2") +package=$(escape "$3") +commit=$(escape "$4") +principal=$(escape "$5") +version=$(escape "$6") +inserted_at=$(escape "$7") + +base=build.hooks.pre_build.zephyr +class=$(invirt-getconf "$base.class" 2>/dev/null || :) +instance=$(invirt-getconf "$base.instance" 2>/dev/null || :) +zsig=$(invirt-getconf "$base.zsig" 2>/dev/null || :) + +if [ -z "$class" ]; then + echo "I don't know where to send a commit zephyr!" >&2 + echo "Please provide a value for $base.class in" >&2 + echo "your invirt config file." >&2 + exit 1 +fi + +(echo "About to begin an Invirtibuild of $package v$version in $pocket." + echo "from commit $commit."; + echo + echo "(Build $build_id was submitted by $principal at $inserted_at.)") | zwrite -c "$class" -i "${instance:-build_$build_id}" -s "${zsig:-pre-build}: $pocket" -d Index: /package_tags/invirt-dev/0.1.1/debian/changelog =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/changelog (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/changelog (revision 3048) @@ -0,0 +1,301 @@ +invirt-dev (0.1.1) unstable; urgency=low + + * Added missing imports + * Other minor functional changes + * Added zephyr and email reporting + * Added script for creating new repositories + + -- Greg Brockman Mon, 05 Jul 2010 01:43:45 -0400 + +invirt-dev (0.1.0) unstable; urgency=low + + * Switch to using git instead of svn. + * Start a git-daemon to serve the git repositories. + * Generate the apt repository configuration using gen-files instead of + rolling our own. + * Since the prerm code to remove the repo group from sudoers runs on + upgrades, make sure the postinst code to add it back runs on all + upgrades, too. + * Use the .. Perl operator to simplify the prerm. + + -- Evan Broder Thu, 27 May 2010 11:55:06 -0400 + +invirt-dev (0.0.14) unstable; urgency=low + + * When building an architecture: any package, pass --arch-all to sbuild + for one of the builds. + + -- Evan Broder Sat, 02 May 2009 12:43:30 -0400 + +invirt-dev (0.0.13) unstable; urgency=low + + * Add a UDebComponents definition so that we can upload the invirtified + ssh + + -- Evan Broder Sun, 26 Apr 2009 11:25:22 -0400 + +invirt-dev (0.0.12) unstable; urgency=low + + * Add a invirt-system component for invirtified packages. + + -- Evan Broder Sat, 25 Apr 2009 20:02:50 -0400 + +invirt-dev (0.0.11) unstable; urgency=low + + * add i386 to apt repo + * extend invirt-build-release to handle amd64 + i386 + * invirt-build-release: clean up some shell and some perl + * invirt-build-release: separate distribution, architecture + + -- Greg Price Mon, 22 Dec 2008 00:04:34 -0500 + +invirt-dev (0.0.10) unstable; urgency=low + + * Clean up /etc/sudoers in the prerm instead of the postinst + + -- Evan Broder Tue, 25 Nov 2008 23:09:03 -0500 + +invirt-dev (0.0.9) unstable; urgency=low + + * Don't add the same line a bunch of times to /etc/sudoers + + -- Evan Broder Tue, 25 Nov 2008 08:16:17 -0500 + +invirt-dev (0.0.8) unstable; urgency=low + + * Support building multiple packages in sequence + + -- Evan Broder Sat, 15 Nov 2008 23:19:38 -0500 + +invirt-dev (0.0.7) unstable; urgency=low + + * Undo my attempts at correctly splitting the package list; do it less + correctly + + -- Evan Broder Mon, 10 Nov 2008 04:01:09 -0500 + +invirt-dev (0.0.6) unstable; urgency=low + + * Punt duplicate dirs file and unused docs file in package + * Add invirt-ood-build for building out of date packages + + -- Evan Broder Mon, 10 Nov 2008 03:43:41 -0500 + +invirt-dev (0.0.5) unstable; urgency=low + + * Add new invirt-ood-packages script + + -- Evan Broder Sat, 08 Nov 2008 23:52:09 -0500 + +invirt-dev (0.0.4) unstable; urgency=low + + * Get the svn repo from the config file, instead of /srv/checkout + + -- Evan Broder Sat, 01 Nov 2008 02:11:59 -0400 + +invirt-dev (0.0.3) unstable; urgency=low + + * sipb-xen-base -> invirt-base + + -- Evan Broder Tue, 28 Oct 2008 04:23:25 -0400 + +invirt-dev (0.0.2) unstable; urgency=low + + * document need to add users to sbuild group + + -- Greg Price Sun, 26 Oct 2008 00:03:03 -0400 + +invirt-dev (0.0.1) unstable; urgency=low + + * sipb-xen-dev -> invirt-dev + * sx-build-release -> invirt-build-release + + -- Greg Price Sat, 25 Oct 2008 23:23:26 -0400 + +sipb-xen-dev (26.2) unstable; urgency=low + + * remove vacuous README files + * remove sipb-xen-repository Apache config file, + which has been in sipb-xen-www, now invirt-web, since May + + -- Greg Price Sat, 25 Oct 2008 22:48:33 -0400 + +sipb-xen-dev (26.1) unstable; urgency=low + + * Fix a typo where the source package didn't get uploaded + + -- Evan Broder Thu, 23 Oct 2008 22:03:40 -0400 + +sipb-xen-dev (26) unstable; urgency=low + + * Update sx-build-release and reprepro-env to work with sbuild instead + of dpkg-buildpackage + + -- Evan Broder Thu, 23 Oct 2008 21:56:17 -0400 + +sipb-xen-dev (25) unstable; urgency=low + + * Don't sign packages when they're built + + -- Evan Broder Sun, 05 Oct 2008 00:25:52 -0400 + +sipb-xen-dev (24) unstable; urgency=low + + * depend on quilt, patchutils, config-package-dev + + -- Greg Price Wed, 01 Oct 2008 01:00:40 -0400 + +sipb-xen-dev (23) unstable; urgency=low + + * in postinst, create user repository and group repo + and add /etc/sudoers line + + -- Greg Price Tue, 30 Sep 2008 23:06:04 -0400 + +sipb-xen-dev (22) unstable; urgency=low + + [ Sam Hartman ] + * Do not include a key id for dpkg-buildpackage per discussion on class xvm + + [ Greg Price ] + * depend on equivs + + -- Greg Price Sun, 24 Aug 2008 15:58:14 -0400 + +sipb-xen-dev (21) unstable; urgency=low + + * Depend on sipb-xen-base + * don't require packages be signed on upload + + -- Sam Hartman Tue, 5 Aug 2008 21:23:44 +0000 + +sipb-xen-dev (20) unstable; urgency=low + + * dumb bugfix in init script + + -- Greg Price Mon, 4 Aug 2008 02:11:04 -0400 + +sipb-xen-dev (19) unstable; urgency=low + + * use CDBS, which doesn't forget dh_installinit like I did + + -- Greg Price Mon, 4 Aug 2008 02:02:41 -0400 + +sipb-xen-dev (18) unstable; urgency=low + + * sign the packages, getting keyid from invirt.config + + -- Greg Price Mon, 4 Aug 2008 00:43:58 -0400 + +sipb-xen-dev (17) unstable; urgency=low + + * use reprepro more correctly (and include in both stable and unstable) + + -- Greg Price Sat, 26 Apr 2008 23:47:55 -0400 + +sipb-xen-dev (16) unstable; urgency=low + + * usage as "sx-build-release ", no "trunk/packages" + * clobber the build-release/ directory automatically + + -- Greg Price Sat, 26 Apr 2008 23:12:03 -0400 + +sipb-xen-dev (15) unstable; urgency=low + + * Stop requiring the current directory to be in the checkout. + * Don't do unnecessary write operations + (hence prompt less often for a password.) + + -- Greg Price Sat, 26 Apr 2008 22:45:42 -0400 + +sipb-xen-dev (14) unstable; urgency=low + + * Use the shallower directory structure. + + -- Eric Price Sat, 29 Mar 2008 17:25:20 -0400 + +sipb-xen-dev (13) unstable; urgency=low + + * Update the svn uri automatically. + + -- Eric Price Fri, 28 Mar 2008 21:38:37 -0400 + +sipb-xen-dev (12) unstable; urgency=low + + * Update the svn uri to point to the new repository + + -- Nelson Elhage Wed, 15 Aug 2007 22:58:49 -0400 + +sipb-xen-dev (11) unstable; urgency=low + + * Use svn export not co to avoid installing .svn directories + + -- Sam Hartman Sat, 4 Aug 2007 19:15:58 -0400 + +sipb-xen-dev (10) unstable; urgency=low + + * Use svn export not svn co so we don't get .svn directories installed + + -- Sam Hartman Sat, 4 Aug 2007 19:14:51 -0400 + +sipb-xen-dev (9) unstable; urgency=low + + * Add /debian to apache config + + -- Sam Hartman Sat, 4 Aug 2007 18:54:43 -0400 + +sipb-xen-dev (8) unstable; urgency=low + + * Can't use .. + + -- Sam Hartman Sun, 15 Jul 2007 17:29:54 -0400 + +sipb-xen-dev (7) unstable; urgency=low + + * Can't use .. + + -- Sam Hartman Sun, 15 Jul 2007 17:27:59 -0400 + +sipb-xen-dev (6) unstable; urgency=low + + * Can't use .. + + -- Sam Hartman Sun, 15 Jul 2007 17:26:17 -0400 + +sipb-xen-dev (5) unstable; urgency=low + + * Bug fix + + -- Sam Hartman Sun, 15 Jul 2007 17:25:00 -0400 + +sipb-xen-dev (4) unstable; urgency=low + + * Blah + + -- Sam Hartman Sun, 15 Jul 2007 17:21:28 -0400 + +sipb-xen-dev (3) unstable; urgency=low + + * Working sx-build-release + + + -- Sam Hartman Sun, 15 Jul 2007 16:58:46 -0400 + +sipb-xen-dev (2) unstable; urgency=low + + * Add sipb-xen-repository apache config + + -- Sam Hartman Sun, 15 Jul 2007 14:04:03 -0400 + +sipb-xen-dev (1) unstable; urgency=low + + * Add repository config + * update dependencies + + -- Sam Hartman Sun, 15 Jul 2007 13:43:08 -0400 + +sipb-xen-dev (0) unstable; urgency=low + + * Initial Release. + + -- unknown Fri, 6 Jul 2007 00:52:28 -0400 Index: /package_tags/invirt-dev/0.1.1/debian/compat =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/compat (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/compat (revision 3048) @@ -0,0 +1,1 @@ +5 Index: /package_tags/invirt-dev/0.1.1/debian/control =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/control (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/control (revision 3048) @@ -0,0 +1,12 @@ +Source: invirt-dev +Section: servers +Priority: important +Maintainer: Invirt project +Build-Depends: cdbs (>= 0.4.23-1.1), debhelper (>= 5), python-all-dev, python-support, python-setuptools, python-debian, python-apt +Standards-Version: 3.7.2 + +Package: invirt-dev +Architecture: all +Depends: ${shlibs:Depends}, ${misc:Depends}, dpkg-dev-el, emacs21, reprepro, apache2, postfix, screen, dh-make, fakeroot, quilt, patchutils, config-package-dev, sbuild, equivs, invirt-base, invirt-database, remctl-server, update-inetd, openbsd-inetd | inet-superserver, python-pyinotify, python-debian, git-core (>= 1.6.4) +Description: Invirt build and apt server + This packages the build scripts and apt-repository configuration for Invirt. Index: /package_tags/invirt-dev/0.1.1/debian/copyright =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/copyright (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/copyright (revision 3048) @@ -0,0 +1,16 @@ +This software was written as part of the Invirt project . + +Copyright : + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + +On Debian systems, the complete text of the GNU General Public License +can be found in the file /usr/share/common-licenses/GPL. Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.cron.daily =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.cron.daily (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.cron.daily (revision 3048) @@ -0,0 +1,3 @@ +#!/bin/sh + +[ -x /usr/bin/invirt-build-conf ] && /usr/bin/invirt-build-conf Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.dirs =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.dirs (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.dirs (revision 3048) @@ -0,0 +1,4 @@ +var/lib/invirt-dev/queue +var/log/invirt/builds +usr/share/invirt-dev/build-hooks +usr/share/invirt-dev/git-hooks Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.install =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.install (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.install (revision 3048) @@ -0,0 +1,5 @@ +reprepro-env usr/bin +invirt-configure-git-hooks usr/bin +repository-config/* srv/repository/conf +build-hooks usr/share/invirt-dev +git-hooks usr/share/invirt-dev Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.invirtibuilder.init =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.invirtibuilder.init (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.invirtibuilder.init (revision 3048) @@ -0,0 +1,82 @@ +#!/bin/bash +### BEGIN INIT INFO +# Provides: invirt-dev +# Required-Start: $local_fs $remote_fs +# Required-Stop: $local_fs $remote_fs +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Invirt development configuration +# Description: +### END INIT INFO + +# Author: Invirt project + +# Do NOT "set -e" + +NAME=invirtibuilder +DESC="the Invirt build daemon" +DAEMON="/usr/bin/$NAME" +PIDFILE="/var/run/$NAME.pid" +GEN_FILES=/srv/repository/conf/distributions + +[ -x "$DAEMON" ] || exit 0 + +. /lib/init/std-init.sh +. /lib/init/gen-files.sh + +do_start() +{ + # Return + # 0 if daemon has been started + # 1 if daemon was already running + # 2 if daemon could not be started + ret=0 + + gen_files || ret=2 + for d in db dists lists pool; do + dir="/srv/repository/$d" + mkdir -p "$dir" + chown -R repository:nogroup "$dir" + done + reprepro-env export || ret=2 + + invirt-build-conf || ret=2 + + if [ "$ret" = 0 ]; then + # Return + # 0 if daemon has been started + # 1 if daemon was already running + # 2 if daemon could not be started + log_daemon_msg "Starting $DESC" "$NAME" + if daemon --running -n $NAME; then + log_daemon_msg "$NAME is already running!" + return 1 + fi + daemon -r -O daemon.info -E daemon.err -n $NAME -U $DAEMON || return 2 + fi + return $ret +} + +do_stop() +{ + # Return + # 0 if daemon has been stopped + # 1 if daemon was already stopped + # 2 if daemon could not be stopped + # other if a failure occurred + log_daemon_msg "Stopping $DESC" "$NAME" + daemon --stop -n $NAME + RETVAL="$?" + [ "$RETVAL" = 2 ] && return 2 + # Many daemons don't delete their pidfiles when they exit. + # rm -f $PIDFILE + return "$RETVAL" +} + +do_reload() +{ + do_stop + do_start +} + +std_init "$@" Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.postinst =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.postinst (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.postinst (revision 3048) @@ -0,0 +1,64 @@ +#!/bin/sh +set -e + +#DEBHELPER# + +case "$1" in + configure) + if ! getent passwd repository >/dev/null 2>&1; then + adduser --system repository + fi + if ! getent group repo >/dev/null 2>&1; then + addgroup --system repo + fi + + if ! getent passwd git >/dev/null 2>&1; then + adduser --system --home /srv/git --shell /usr/bin/git-shell git + fi + + if ! [ -d /srv/git ]; then + mkdir -p /srv/git + fi + + cat >>/etc/sudoers <&2 + exit 1 + ;; +esac + +if [ -x /etc/init.d/invirtibuilder ]; then + if hash invoke-rc.d 2>/dev/null; then + invoke-rc.d invirtibuilder start + else + /etc/init.d/invirtibuilder start + fi +fi Index: /package_tags/invirt-dev/0.1.1/debian/invirt-dev.prerm =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/invirt-dev.prerm (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/invirt-dev.prerm (revision 3048) @@ -0,0 +1,51 @@ +#!/bin/sh +# prerm script for #PACKAGE# +# +# see: dh_installdeb(1) + +set -e + +# summary of how this script can be called: +# * `remove' +# * `upgrade' +# * `failed-upgrade' +# * `remove' `in-favour' +# * `deconfigure' `in-favour' +# `removing' +# +# for details, see http://www.debian.org/doc/debian-policy/ or +# the debian-policy package + + +case "$1" in + remove|upgrade|deconfigure) + [ "$1" = "remove" ] && update-inetd --disable git + + perl -i.bak -ne 'print unless /^### BEGIN invirt-dev/../^### END invirt-dev/' /etc/sudoers + ;; + + failed-upgrade) + ;; + + *) + echo "prerm called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +if [ -x /etc/init.d/invirt-dev ]; then + if hash invoke-rc.d 2>/dev/null; then + invoke-rc.d invirt-dev stop + else + /etc/init.d/invirt-dev stop + fi +fi + +# dh_installdeb will replace this with shell code automatically +# generated by other debhelper scripts. + +#DEBHELPER# + +exit 0 + + Index: /package_tags/invirt-dev/0.1.1/debian/pyversions =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/pyversions (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/pyversions (revision 3048) @@ -0,0 +1,1 @@ +2.5- Index: /package_tags/invirt-dev/0.1.1/debian/rules =================================================================== --- /package_tags/invirt-dev/0.1.1/debian/rules (revision 3048) +++ /package_tags/invirt-dev/0.1.1/debian/rules (revision 3048) @@ -0,0 +1,17 @@ +#!/usr/bin/make -f + +DEB_PYTHON_SYSTEM=pysupport + +# We use --no-start instead of letting dh_installinit do its thing +# because, in the postinst, dh_installinit's automatically added code +# runs before dh_pysupport's automatically added code. +# +# This means that when the initscript is started, Python modules +# installed by the package haven't been fully installed yet. +DEB_DH_INSTALLINIT_ARGS += --no-start --name invirtibuilder + +include /usr/share/cdbs/1/rules/debhelper.mk +include /usr/share/cdbs/1/class/python-distutils.mk + +clean:: + rm -rf python/invirt.builder.egg-info Index: /package_tags/invirt-dev/0.1.1/git-hooks/other/post-receive =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/other/post-receive (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/other/post-receive (revision 3048) @@ -0,0 +1,1 @@ +link ../sub/post-receive Index: /package_tags/invirt-dev/0.1.1/git-hooks/other/zephyr-post-receive =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/other/zephyr-post-receive (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/other/zephyr-post-receive (revision 3048) @@ -0,0 +1,1 @@ +link ../sub/zephyr-post-receive Index: /package_tags/invirt-dev/0.1.1/git-hooks/sub/post-receive =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/sub/post-receive (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/sub/post-receive (revision 3048) @@ -0,0 +1,8 @@ +#!/bin/sh + +file=`mktemp -t git-post-receive.XXXXXX` || exit $? +cat >$file + +"$PWD"/hooks/zephyr-post-receive <$file + +rm -f $file Index: /package_tags/invirt-dev/0.1.1/git-hooks/sub/update =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/sub/update (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/sub/update (revision 3048) @@ -0,0 +1,49 @@ +#!/bin/sh + +ref="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check + +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$ref" ] || [ -z "$oldrev" ] || [ -z "$newrev" ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +# --- Disallow pushing to the branches for pockets + +pocket_to_git() { + local pocket git + pocket="$1" + git="$(invirt-getconf "git.pockets.$pocket.git" 2>/dev/null)" + if [ $? != 0 ]; then + git="$pocket" + fi + echo "$git" +} + +for pocket in $(invirt-getconf -l build.pockets); do + if [ "$ref" = "refs/heads/$(pocket_to_git "$pocket")" ]; then + echo "*** Pushing to a pocket branch in this repository is not allowed" >&2 + exit 1 + fi +done + +# --- Disallow pushing tags + +case "$ref" in + refs/heads/*) + ;; + *) + echo "*** Pushing non-branches to this repository is not allowed" >&2 + exit 1 + ;; +esac Index: /package_tags/invirt-dev/0.1.1/git-hooks/sub/zephyr-post-receive =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/sub/zephyr-post-receive (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/sub/zephyr-post-receive (revision 3048) @@ -0,0 +1,49 @@ +#!/bin/sh +# +# This script is run after receive-pack has accepted a pack and the +# repository has been updated. It is passed arguments in through stdin +# in the form +# +# For example: +# aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master + +base=build.hooks.post_commit.zephyr + +class=$(invirt-getconf "$base.class" 2>/dev/null) +instance=$(invirt-getconf "$base.instance" 2>/dev/null) +zsig=$(invirt-getconf "$base.zsig" 2>/dev/null) +color=$(invirt-getconf "$base.color" 2>/dev/null) + +if [ "${color:-true}" = "true" ]; then + usecolor="--color" +else + usecolor="" +fi + +if [ -z "$class" ]; then + echo "I don't know where to send a commit zephyr!" >&2 + echo "Please provide a value for $base.class in" >&2 + echo "your invirt config file." >&2 + exit 1 +fi +while read oldrev newrev refname; do + if [ "$oldrev" = "0000000000000000000000000000000000000000" ]; then + # dammit git + zwrite -c "$class" -i "$(basename "$refname")" -s "${zsig:-Git}: $refname" -d \ + -m "New branch created." + continue + fi + git rev-list --first-parent --reverse "$oldrev..$newrev" | while read rev; do + shortrev=`git log -1 --pretty=format:%h "$rev"` + (git show --stat -M $usecolor "$rev" | + sed -e 's/@/@@/g' \ + -e 's/}/@(})/g' \ + -e 's/\[m/}@{/g' \ + -e 's/\[33m/@color(yellow)/g' \ + -e 's/\[31m/@color(red)/g' \ + -e 's/\[32m/@color(green)/g' \ + -e '1s/^/@{/' \ + -e '$s/$/}/') | + zwrite -c "$class" -i "${instance:-$shortrev}" -s "${zsig:-Git}: $refname" -d + done +done Index: /package_tags/invirt-dev/0.1.1/git-hooks/super/update =================================================================== --- /package_tags/invirt-dev/0.1.1/git-hooks/super/update (revision 3048) +++ /package_tags/invirt-dev/0.1.1/git-hooks/super/update (revision 3048) @@ -0,0 +1,26 @@ +#!/bin/sh + +ref="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check + +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$ref" ] || [ -z "$oldrev" ] || [ -z "$newrev" ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +# --- Disallow all pushes + +echo "*** Pushing to the superproject is not allowed" >&2 +echo "***" >&2 +echo "*** If you would like to update the superproject, use remctl" >&2 +exit 1 Index: /package_tags/invirt-dev/0.1.1/invirt-add-repo =================================================================== --- /package_tags/invirt-dev/0.1.1/invirt-add-repo (revision 3048) +++ /package_tags/invirt-dev/0.1.1/invirt-add-repo (revision 3048) @@ -0,0 +1,46 @@ +#!/usr/bin/env python + +import optparse +import os +import shutil +import sys + +from invirt import builder, common +from invirt.config import structs as config + +REPO_BASE = '/srv/git/invirt' +HOOK_BASE = '/usr/share/invirt-dev/git-hooks' + +def main(): + parser = optparse.OptionParser('%prog repo ') + opts, args = parser.parse_args() + + if len(args) != 3: + parser.print_help() + return 1 + + category = args[1] + name = args[2] + principal = os.environ['REMOTE_USER'] + repo_path = os.path.join(REPO_BASE, category, '%s.git' % name) + + if os.path.exists(repo_path): + print >>sys.stderr, '%s already exists!' % repo_path + return 1 + + print 'Creating new repo at %s' % repo_path + os.makedirs(repo_path) + common.captureOutput(['git', 'init', '--bare'], cwd=repo_path) + print 'Replacing hooks directory with a symlink' + hooks_dir = os.path.join(repo_path, 'hooks') + shutil.rmtree(hooks_dir) + common.captureOutput(['chown', '-R', 'git:', repo_path]) + if category == 'packages': + os.symlink(os.path.join(HOOK_BASE, 'sub'), hooks_dir) + else: + os.symlink(os.path.join(HOOK_BASE, 'other'), hooks_dir) + + builder.runHook('post-add-repo', [category, name, principal]) + +if __name__ == '__main__': + sys.exit(main()) Index: /package_tags/invirt-dev/0.1.1/invirt-build-conf =================================================================== --- /package_tags/invirt-dev/0.1.1/invirt-build-conf (revision 3048) +++ /package_tags/invirt-dev/0.1.1/invirt-build-conf (revision 3048) @@ -0,0 +1,68 @@ +#!/usr/bin/python + +"""Re-generate the remctl configuration for build submissions. + +This script generates the remctl ACL and configuration for each build +pocket defined in the configuration. It also updates the .k5login for +the git user that developers can push through. +""" +from __future__ import with_statement + +import contextlib +import os +import tempfile + +from invirt import authz +from invirt import builder +from invirt.config import structs as config + + +def userToPrinc(user): + """Convert an AFS principal to a Kerberos v5 principal.""" + if '@' in user: + (princ, realm) = user.split('@') + else: + princ = user + realm = config.kerberos.realm + + return princ.replace('.', '/') + '@' + realm + +def acl_path(pocket): + return '/etc/remctl/acl/build-%s' % pocket + +@contextlib.contextmanager +def atomic_write(file): + tmp_fd, tmp_name = tempfile.mkstemp() + tmp = os.fdopen(tmp_fd, 'r+') + yield tmp + tmp.close() + os.rename(tmp_name, file) + +def main(): + all_devs = set() + build_handler = '/usr/bin/invirt-submit-build' + + for pocket in config.build.pockets: + acl = authz.expandAdmin(getattr(config.build.pockets, pocket).acl, None) + with atomic_write(acl_path(pocket)) as f: + princs = [userToPrinc(a) for a in acl] + print >>f, '\n'.join(princs) + all_devs.update(set(princs)) + + with atomic_write('/etc/remctl/conf.d/build') as f: + for pocket in config.build.pockets: + print >>f, 'build %s %s %s' % (pocket, build_handler, acl_path(pocket)) + + with atomic_write('/etc/remctl/acl/repo_admin') as f: + acl = authz.expandAdmin(config.build.repo_admin, None) + print >>f, '\n'.join(userToPrinc(a) for a in acl) + + with atomic_write('/etc/remctl/conf.d/repo_admin') as f: + print >>f, 'create repo /usr/bin/invirt-add-repo /etc/remctl/acl/repo_admin' + + with atomic_write(os.path.join(builder._REPO_DIR, '.k5login')) as f: + print >>f, '\n'.join(all_devs) + + +if __name__ == '__main__': + main() Index: /package_tags/invirt-dev/0.1.1/invirt-configure-git-hooks =================================================================== --- /package_tags/invirt-dev/0.1.1/invirt-configure-git-hooks (revision 3048) +++ /package_tags/invirt-dev/0.1.1/invirt-configure-git-hooks (revision 3048) @@ -0,0 +1,21 @@ +#!/bin/sh + +set -e +set -u + +link_to() { + rm -rf "$1" + ln -s "$2" "$1" +} + +package_base=/srv/git/invirt +hook_base=/usr/share/invirt-dev/git-hooks + +link_to "$package_base/packages.git/hooks" "$hook_base/super" + +for pkg in "$package_base"/*/*.git; do + case "$pkg" in + "$package_base/packages/*.git") link_to "$pkg/hooks" "$hook_base/sub" ;; + *) link_to "$pkg/hooks" "$hook_base/other" ;; + esac +done Index: /package_tags/invirt-dev/0.1.1/invirt-submit-build =================================================================== --- /package_tags/invirt-dev/0.1.1/invirt-submit-build (revision 3048) +++ /package_tags/invirt-dev/0.1.1/invirt-submit-build (revision 3048) @@ -0,0 +1,70 @@ +#!/usr/bin/python + +"""Validate and add a new item to the Invirt build queue. + +This script, intended to be invoked by remctl, first validates the +build submitted parameters, and then adds a new item to the +Invirtibuilder build queue, triggering the Invirtibuilder to start the +build. + +The expected arguments are + + pocket package commit + +This script will also automatically extract the Kerberos principal +used to submit the job, and include that in the queue file for records +keeping. +""" + + +import datetime +import optparse +import os +import sys +import tempfile +import uuid + +import invirt.builder as b + + +def main(): + parser = optparse.OptionParser('Usage: %prog pocket package commit') + opts, args = parser.parse_args() + if len(args) != 3: + parser.print_help() + return 1 + pocket, package, commit = args + principal = os.environ['REMOTE_USER'] + request_time = datetime.datetime.utcnow() + q_path = os.path.join(b._QUEUE_DIR, + '%s_%s' % (request_time.strftime('%Y%m%d%H%M%S'), + uuid.uuid4())) + + try: + # TODO: clean up this interface. + b.ensureValidPackage(package) + commit = b.canonicalize_commit(package, commit) + b.validateBuild(pocket, package, commit) + except b.InvalidBuild, e: + msg = "E: %s" % e + print >>sys.stderr, msg + # Prevent an attack by submitting excessively long arguments + args = [arg[:min(len(arg), 80)] for arg in (pocket, package, commit)] + b.runHook('failed-submit', args + [principal], stdin_str=msg) + sys.exit(1) + + # To keep from triggering the Invirtibuilder before we've actually + # written the file out, first write the queue entry to a temporary + # file, and then move it into the queue directory. + q_fd, q_name = tempfile.mkstemp() + q = os.fdopen(q_fd, 'r+') + print >>q, "%s %s %s %s" % (pocket, package, commit, principal) + q.close() + os.rename(q_name, q_path) + short_commit = b.canonicalize_commit(package, commit, shorten=True) + b.runHook('post-submit', [pocket, package, short_commit, principal]) + print '%s, your job to build %s for %s:%s has been submitted!' % (principal, short_commit, package, pocket) + + +if __name__ == '__main__': + main() Index: /package_tags/invirt-dev/0.1.1/invirtibuilder =================================================================== --- /package_tags/invirt-dev/0.1.1/invirtibuilder (revision 3048) +++ /package_tags/invirt-dev/0.1.1/invirtibuilder (revision 3048) @@ -0,0 +1,410 @@ +#!/usr/bin/python + +"""Process the Invirt build queue. + +The Invirtibuilder handles package builds and uploads. On demand, it +attempts to build a particular package. + +If the build succeeds, the new version of the package is uploaded to +the apt repository, tagged in its git repository, and the Invirt +superproject is updated to point at the new version. + +If the build fails, the Invirtibuilder sends mail with the build log. + +The build queue is tracked via files in /var/lib/invirt-dev/queue. In +order to maintain ordering, all filenames in that directory are the +timestamp of their creation time. + +Each queue file contains a file of the form + + pocket package hash principal + +where pocket is one of the pockets globally configured in +build.pockets. For instance, the pockets in XVM are "prod" and "dev". + +principal is the Kerberos principal that requested the build. +""" + + +from __future__ import with_statement + +import contextlib +import glob +import os +import re +import shutil +import subprocess +import tempfile +import traceback + +import pyinotify + +from debian_bundle import deb822 + +import invirt.builder as b +import invirt.common as c +from invirt import database +from invirt.config import structs as config + + +DISTRIBUTION = 'hardy' + + +def getControl(package, ref): + """Get the parsed debian/control file for a given package. + + This returns a list of debian_bundle.deb822.Deb822 objects, one + for each section of the debian/control file. Each Deb822 object + acts roughly like a dict. + """ + return deb822.Deb822.iter_paragraphs( + b.getGitFile(package, ref, 'debian/control').split('\n')) + + +def getBinaries(package, ref): + """Get a list of binary packages in a package at a given ref.""" + return [p['Package'] for p in getControl(package, ref) + if 'Package' in p] + + +def getArches(package, ref): + """Get the set of all architectures in any binary package.""" + arches = set() + for section in getControl(package, ref): + if 'Architecture' in section: + arches.update(section['Architecture'].split()) + return arches + + +def getDscName(package, ref): + """Return the .dsc file that will be generated for this package.""" + v = b.getVersion(package, ref) + if v.debian_version: + v_str = '%s-%s' % (v.upstream_version, + v.debian_version) + else: + v_str = v.upstream_version + return '%s_%s.dsc' % ( + package, + v_str) + + +def sanitizeVersion(version): + """Sanitize a Debian package version for use as a git tag. + + This function strips the epoch from the version number and + replaces any tildes with periods.""" + if v.debian_version: + v = '%s-%s' % (version.upstream_version, + version.debian_version) + else: + v = version.upstream_version + return v.replace('~', '.') + + +def aptCopy(package, commit, dst_pocket, src_pocket): + """Copy a package from one pocket to another.""" + binaries = getBinaries(package, commit) + c.captureOutput(['reprepro-env', 'copy', + b.pocketToApt(dst_pocket), + b.pocketToApt(src_pocket), + package] + binaries) + + +def sbuild(package, ref, arch, workdir, arch_all=False): + """Build a package for a particular architecture.""" + args = ['sbuild', '-v', '-d', DISTRIBUTION, '--arch', arch] + if arch_all: + args.append('-A') + args.append(getDscName(package, ref)) + c.captureOutput(args, cwd=workdir) + + +def sbuildAll(package, ref, workdir): + """Build a package for all architectures it supports.""" + arches = getArches(package, ref) + if 'all' in arches or 'any' in arches or 'amd64' in arches: + sbuild(package, ref, 'amd64', workdir, arch_all=True) + if 'any' in arches or 'i386' in arches: + sbuild(package, ref, 'i386', workdir) + + +def tagSubmodule(pocket, package, commit, principal, version, env): + """Tag a new version of a submodule. + + If this pocket does not allow_backtracking, then this will create + a new tag of the version at ref. + + This function doesn't need to care about lock + contention. git-receive-pack updates one ref at a time, and only + takes out a lock for that ref after it's passed the update + hook. Because we reject pushes to tags in the update hook, no push + can ever take out a lock on any tags. + + I'm sure that long description gives you great confidence in the + legitimacy of my reasoning. + """ + if not config.build.pockets[pocket].get('allow_backtracking', False): + branch = b.pocketToGit(pocket) + tag_msg = ('Tag %s of %s\n\n' + 'Requested by %s' % (version.full_version, + package, + principal)) + + c.captureOutput( + ['git', 'tag', '-m', tag_msg, commit], + env=env, + cwd=b.getRepo(package)) + + +def updateSubmoduleBranch(pocket, package, commit): + """Update the appropriately named branch in the submodule.""" + branch = b.pocketToGit(pocket) + c.captureOutput( + ['git', 'update-ref', 'refs/heads/%s' % branch, commit], cwd=b.getRepo(package)) + + +def uploadBuild(pocket, workdir): + """Upload all build products in the work directory.""" + force = config.build.pockets[pocket].get('allow_backtracking', False) + apt = b.pocketToApt(pocket) + for changes in glob.glob(os.path.join(workdir, '*.changes')): + upload = ['reprepro-env', '--ignore=wrongdistribution', + 'include', apt, changes] + try: + c.captureOutput(upload) + except subprocess.CalledProcessError, e: + if not force: + raise + package = deb822.Changes(open(changes).read())['Binary'] + c.captureOutput(['reprepro-env', 'remove', apt, package]) + c.captureOutput(upload) + + +def updateSuperproject(pocket, package, commit, principal, version, env): + """Update the superproject. + + This will create a new commit on the branch for the given pocket + that sets the commit for the package submodule to commit. + + Note that there's no locking issue here, because we disallow all + pushes to the superproject. + """ + superproject = os.path.join(b._REPO_DIR, 'invirt/packages.git') + branch = b.pocketToGit(pocket) + tree = c.captureOutput(['git', 'ls-tree', branch], + cwd=superproject).strip() + + new_tree = re.compile( + r'^(160000 commit )[0-9a-f]*(\t%s)$' % package, re.M).sub( + r'\g<1>%s\g<2>' % commit, + tree) + + new_tree_id = c.captureOutput(['git', 'mktree', '--missing'], + cwd=superproject, + stdin_str=new_tree).strip() + + commit_msg = ('Update %s to version %s\n\n' + 'Requested by %s' % (package, + version.full_version, + principal)) + new_commit = c.captureOutput( + ['git', 'commit-tree', new_tree_id, '-p', branch], + cwd=superproject, + env=env, + stdin_str=commit_msg).strip() + + c.captureOutput( + ['git', 'update-ref', 'refs/heads/%s' % branch, new_commit], + cwd=superproject) + + +def makeReadable(workdir): + os.chmod(workdir, 0755) + +@contextlib.contextmanager +def packageWorkdir(package, commit): + """Checkout the package in a temporary working directory. + + This context manager returns that working directory. The requested + package is checked out into a subdirectory of the working + directory with the same name as the package. + + When the context wrapped with this context manager is exited, the + working directory is automatically deleted. + """ + workdir = tempfile.mkdtemp() + try: + p_archive = subprocess.Popen( + ['git', 'archive', + '--remote=file://%s' % b.getRepo(package), + '--prefix=%s/' % package, + commit, + ], + stdout=subprocess.PIPE, + ) + p_tar = subprocess.Popen( + ['tar', '-x'], + stdin=p_archive.stdout, + cwd=workdir, + ) + p_archive.wait() + p_tar.wait() + + yield workdir + finally: + shutil.rmtree(workdir) + +def build(): + """Deal with items in the build queue. + + When triggered, iterate over build queue items one at a time, + until there are no more pending build jobs. + """ + while True: + stage = 'processing incoming job' + queue = os.listdir(b._QUEUE_DIR) + if not queue: + break + + build = min(queue) + job = open(os.path.join(b._QUEUE_DIR, build)).read().strip() + pocket, package, commit, principal = job.split() + + database.session.begin() + db = database.Build() + db.package = package + db.pocket = pocket + db.commit = commit + db.principal = principal + database.session.save_or_update(db) + database.session.commit() + + database.session.begin() + + try: + db.failed_stage = 'validating job' + # Don't expand the commit in the DB until we're sure the user + # isn't trying to be tricky. + b.ensureValidPackage(package) + db.commit = commit = b.canonicalize_commit(package, commit) + src = b.validateBuild(pocket, package, commit) + version = b.getVersion(package, commit) + db.version = str(version) + b.runHook('pre-build', [str(db.build_id), db.pocket, db.package, + db.commit, db.principal, db.version, str(db.inserted_at)]) + + env = dict(os.environ) + env['GIT_COMMITTER_NAME'] = config.build.tagger.name + env['GIT_COMMITTER_EMAIL'] = config.build.tagger.email + + # If validateBuild returns something other than True, then + # it means we should copy from that pocket to our pocket. + # + # (If the validation failed, validateBuild would have + # raised an exception) + if src != True: + # TODO: cut out this code duplication + db.failed_stage = 'tagging submodule before copying package' + tagSubmodule(pocket, package, commit, principal, version, env) + db.failed_stage = 'updating submodule branches before copying package' + updateSubmoduleBranch(pocket, package, commit) + db.failed_stage = 'updating superproject before copying package' + updateSuperproject(pocket, package, commit, principal, version, env) + db.failed_stage = 'copying package from another pocket' + aptCopy(package, commit, pocket, src) + + # If we can't copy the package from somewhere, but + # validateBuild didn't raise an exception, then we need to + # do the build ourselves + else: + db.failed_stage = 'checking out package source' + with packageWorkdir(package, commit) as workdir: + db.failed_stage = 'preparing source package' + packagedir = os.path.join(workdir, package) + + # We should be more clever about dealing with + # things like non-Debian-native packages than we + # are. + # + # If we were, we could use debuild and get nice + # environment scrubbing. Since we're not, debuild + # complains about not having an orig.tar.gz + c.captureOutput(['dpkg-buildpackage', '-us', '-uc', '-S'], + cwd=packagedir, + stdout=None) + + try: + db.failed_stage = 'building binary packages' + sbuildAll(package, commit, workdir) + finally: + logdir = os.path.join(b._LOG_DIR, str(db.build_id)) + if not os.path.exists(logdir): + os.makedirs(logdir) + + for log in glob.glob(os.path.join(workdir, 'build-*.log')): + os.copy(log, logdir) + + db.failed_stage = 'tagging submodule' + tagSubmodule(pocket, package, commit, principal, version, env) + db.failed_stage = 'updating submodule branches' + updateSubmoduleBranch(pocket, package, commit) + db.failed_stage = 'updating superproject' + updateSuperproject(pocket, package, commit, principal, version, env) + db.failed_stage = 'relaxing permissions on workdir' + makeReadable(workdir) + db.failed_stage = 'uploading packages to apt repo' + uploadBuild(pocket, workdir) + + db.failed_stage = 'cleaning up' + except: + db.traceback = traceback.format_exc() + else: + db.succeeded = True + db.failed_stage = None + finally: + database.session.save_or_update(db) + database.session.commit() + + # Finally, now that everything is done, remove the + # build queue item + os.unlink(os.path.join(b._QUEUE_DIR, build)) + + if db.succeeded: + b.runHook('post-build', [str(db.build_id)]) + else: + b.runHook('failed-build', [str(db.build_id)]) + +class Invirtibuilder(pyinotify.ProcessEvent): + """Process inotify triggers to build new packages.""" + def process_default(self, event): + """Handle an inotify event. + + When an inotify event comes in, trigger the builder. + """ + build() + + +def main(): + """Initialize the inotifications and start the main loop.""" + database.connect() + + watch_manager = pyinotify.WatchManager() + invirtibuilder = Invirtibuilder() + notifier = pyinotify.Notifier(watch_manager, invirtibuilder) + watch_manager.add_watch(b._QUEUE_DIR, + pyinotify.EventsCodes.ALL_FLAGS['IN_CREATE'] | + pyinotify.EventsCodes.ALL_FLAGS['IN_MOVED_TO']) + + # Before inotifying, run any pending builds; otherwise we won't + # get notified for them. + build() + + while True: + notifier.process_events() + if notifier.check_events(): + notifier.read_events() + + +if __name__ == '__main__': + main() Index: /package_tags/invirt-dev/0.1.1/python/invirt/builder.py =================================================================== --- /package_tags/invirt-dev/0.1.1/python/invirt/builder.py (revision 3048) +++ /package_tags/invirt-dev/0.1.1/python/invirt/builder.py (revision 3048) @@ -0,0 +1,160 @@ +"""Invirt build utilities. + +This module contains utility functions used by both the invirtibuilder +and the remctl submission scripts that insert items into its queue. +""" + + +import os +import subprocess + +from debian_bundle import changelog + +import invirt.common as c +from invirt.config import structs as config + + +_QUEUE_DIR = '/var/lib/invirt-dev/queue' +_REPO_DIR = '/srv/git' +_LOG_DIR = '/var/log/invirt/builds' +_HOOKS_DIR = '/usr/share/invirt-dev/build-hooks' + + +class InvalidBuild(ValueError): + pass + + +def getRepo(package): + """Return the path to the git repo for a given package.""" + return os.path.join(_REPO_DIR, 'invirt/packages', '%s.git' % package) + +def ensureValidPackage(package): + """Perform some basic sanity checks that the requested repo is in a + subdirectory of _REPO_DIR/invirt/packages. This prevents weirdness + such as submitting a package like '../prod/...git'. Also ensures that + the repo exists.""" + # TODO: this might be easier just to regex + repo = os.path.abspath(getRepo(package)) + parent_dir = os.path.dirname(repo) + prefix = os.path.join(_REPO_DIR, 'invirt/packages') + if not parent_dir.startswith(prefix): + raise InvalidBuild('Invalid package name %s' % package) + elif not os.path.exists(repo): + raise InvalidBuild('Nonexisting package %s' % package) + +def canonicalize_commit(package, commit, shorten=False): + if shorten: + flags = ['--short'] + else: + flags = [] + return c.captureOutput(['git', 'rev-parse'] + flags + [commit], + cwd=getRepo(package)).strip() + +def pocketToGit(pocket): + """Map a pocket in the configuration to a git branch.""" + return getattr(getattr(config.build.pockets, pocket), 'git', pocket) + + +def pocketToApt(pocket): + """Map a pocket in the configuration to an apt repo pocket.""" + return getattr(getattr(config.build.pockets, pocket), 'apt', pocket) + + +def getGitFile(package, ref, path): + """Return the contents of a path from a git ref in a package.""" + return c.captureOutput(['git', 'cat-file', 'blob', '%s:%s' % (ref, path)], + cwd=getRepo(package)) + + +def getChangelog(package, ref): + """Get a changelog object for a given ref in a given package. + + This returns a debian_bundle.changelog.Changelog object for a + given ref of a given package. + """ + return changelog.Changelog(getGitFile(package, ref, 'debian/changelog')) + +def runHook(hook, args=[], stdin_str=None): + """Run a named hook.""" + hook = os.path.join(_HOOKS_DIR, hook) + try: + c.captureOutput([hook] + args, stdin_str=stdin_str) + except OSError: + pass + +def getVersion(package, ref): + """Get the version of a given package at a particular ref.""" + return getChangelog(package, ref).get_version() + + +def validateBuild(pocket, package, commit): + """Given the parameters of a new build, validate that build. + + The checks this function performs vary based on whether or not the + pocket is configured with allow_backtracking. + + A build of a pocket without allow_backtracking set must be a + fast-forward of the previous revision, and the most recent version + in the changelog most be strictly greater than the version + currently in the repository. + + In all cases, this revision of the package can only have the same + version number as any other revision currently in the apt + repository if they have the same commit ID. + + If it's unspecified, it is assumed that pocket do not + allow_backtracking. + + If this build request fails validation, this function will raise a + InvalidBuild exception, with information about why the validation + failed. + + If this build request can be satisfied by copying the package from + another pocket, then this function returns that pocket. Otherwise, + it returns True. + """ + ensureValidPackage(package) + package_repo = getRepo(package) + new_version = getVersion(package, commit) + + ret = True + + for p in config.build.pockets: + if p == pocket: + continue + + b = pocketToGit(p) + try: + current_commit = c.captureOutput(['git', 'rev-parse', b], + cwd=package_repo).strip() + except subprocess.CalledProcessError: + # Guess we haven't created this pocket yet + continue + + current_version = getVersion(package, b) + + if current_version == new_version: + if current_commit == commit: + ret = p + else: + raise InvalidBuild('Version %s of %s already available is in ' + 'pocket %s from commit %s' % + (new_version, package, p, current_commit)) + + if not config.build.pockets[pocket].get('allow_backtracking', False): + branch = pocketToGit(pocket) + current_version = getVersion(package, branch) + if new_version <= current_version: + raise InvalidBuild('New version %s of %s is not newer than ' + 'version %s currently in pocket %s' % + (new_version, package, current_version, pocket)) + + # Almost by definition, A is a fast-forward of B if B..A is + # empty + if not c.captureOutput(['git', 'rev-list', '%s..%s' % (commit, branch)], + cwd=package_repo): + raise InvalidBuild('New commit %s of %s is not a fast-forward of' + 'commit currently in pocket %s' % + (commit, package, pocket)) + + return ret Index: /package_tags/invirt-dev/0.1.1/repository-config/distributions.mako =================================================================== --- /package_tags/invirt-dev/0.1.1/repository-config/distributions.mako (revision 3048) +++ /package_tags/invirt-dev/0.1.1/repository-config/distributions.mako (revision 3048) @@ -0,0 +1,17 @@ +<% +from invirt.config import structs as cfg +from invirt import builder as b +%>\ +% for pocket in cfg.build.pockets: +CodeName: ${b.pocketToApt(pocket)} +Components: main invirt-system +UDebComponents: main invirt-system +SignWith: ${cfg.apt.keyid} +Architectures: amd64 i386 source +Origin: Invirt +Description: Invirt ${pocket} pocket +Contents: . .gz +DebIndices: Packages Release . .gz .bz2 +DscIndices: Sources Release . .gz .bz2 + +% endfor Index: /package_tags/invirt-dev/0.1.1/reprepro-env =================================================================== --- /package_tags/invirt-dev/0.1.1/reprepro-env (revision 3048) +++ /package_tags/invirt-dev/0.1.1/reprepro-env (revision 3048) @@ -0,0 +1,5 @@ +#!/bin/sh +set -e +export HOME=/home/repository +exec sudo -H -u repository /usr/bin/reprepro -b /srv/repository --ignore=wrongdistribution "$@" + Index: /package_tags/invirt-dev/0.1.1/setup.py =================================================================== --- /package_tags/invirt-dev/0.1.1/setup.py (revision 3048) +++ /package_tags/invirt-dev/0.1.1/setup.py (revision 3048) @@ -0,0 +1,25 @@ +#!/usr/bin/python + +from os import path +from debian_bundle.changelog import Changelog +from debian_bundle.deb822 import Deb822 +from email.utils import parseaddr +from glob import glob +from setuptools import setup + +version = Changelog(open(path.join(path.dirname(__file__), 'debian/changelog')).read()).\ + get_version().full_version + +maintainer_full = Deb822(open(path.join(path.dirname(__file__), 'debian/control')))['Maintainer'] +maintainer, maintainer_email = parseaddr(maintainer_full) + +setup( + name='invirt.builder', + version=version, + maintainer=maintainer, + maintainer_email=maintainer_email, + + py_modules = ['invirt.builder'], + package_dir = {'': 'python'}, + scripts = ['invirtibuilder', 'invirt-build-conf', 'invirt-submit-build', 'invirt-add-repo'] +)