repeatable one off builds with 'emerge'

May 4th, 2009

I have to say I do actually use quite a lot of software. Not all the time, but I like to try things. I'm not one of those "email and web" people that you hear about. And because I like to be able to run some application that's not widespread without it causing me any extra hassle I'm really only interested in the distros that have the largest repos. Such as Gentoo and Ubuntu.

Even so, there are those package requirements that fall through the cracks. It's hard to get a hold of bleeding edge builds sometimes. Even if there are some dedicated packagers for a particular project working for a particular distro, these ad-hoc efforts don't cover everything. Not to mention the fact that unofficial builds on Ubuntu are not necessarily easily reversible. I spent some time recently working on a project in Mono, and there's just no real option for me to get close-to-svn builds. Even Gentoo is quite far behind and I never have any luck with the absolute latests ebuilds.

Hence the svn builds.

In any case, there will always be times when custom builds are necessary. After all that's what developers do: take on the scary world of unstable so that in the end we can make sure we ship something users can run both on the current version of their distro, and (hopefully) for some time to come.

To that end I wrote a messy perl script to do mono svn builds. That was over a month ago. I was looking to add a feature just the other day and it hit me just how awful perl is. Every time I come back to it after a period of absence I feel like a stranger in the valley of death, with the auto expanding arrays and other land mines. It's not the first time I've written one of these build scripts, I need them from time to time while I'm on some project. And I often end up with a couple bash scripts so I can reproduce these installs later on.

But do I really want to be writing these ad-hoc installers every time? That's the thing about one off builds, they're not really one off. Often you want to be able to reproduce it.

Nothing whatsoever to do with Gentoo portage!

I know I should have picked a different name, but the parallels are so clear that I just couldn't resist. emerge is a simple package installer. Emphasis on installer, it does not manage packages, it does not even keep track of what has been installed. No notion of versions, no safety checks, just the user supplied shell commands wrapped in a nicer package.

  • Phases: fetch, configure, build, install
  • Other actions: list, search, pretend
  • Switches: nodeps, revision
  • Fetch support: git, svn, archive (tar,gzip,bzip2,zip)

Here's the idea:

  1. Write a build file containing a number of packages.
  2. Record dependencies between them.
  3. Use emerge to run any combination of actions on them.

It's really that simple. To use the running mono example, here's a taste:

emerge_search

The build file is an optional argument if there's only one in the current directory. Most of the command line options are identical to portage:

emerge_opts

And if you want to just see the depgraph that works too. Just don't put cycles in it or weird things will happen.

emerge_pretend

And if you need to set certain environmental variables during building/running (so that the package doesn't meddle with your system) it supports that too.

Build files

Build files are basically just a listing of the shell commands necessary to build the package. I workshopped the idea of writing them in YAML, since it's a nice and compact format. But I decided against it, because it's not really buying me anything, and since YAML's grammar is rather quirky the parse errors are harder to understand than Python's. So the build files are just regular Python modules. Here's an excerpt from mono.py:

src_path = "/ex/mono-sources"
ins_path = "/ex/mono"
svnbase = "svn://anonsvn.mono-project.com/source/trunk"

conf = "./autogen.sh --prefix=%s" % ins_path
build = "make"
install = "make install"

env = os.environ.get

for v in ("libgdiplus", "mcs", "olive", "mono", "debugger", "mono-addins",
          "mono-tools", "gtk-sharp", "gnome-sharp", "monodoc-widgets",
          "monodevelop", "paint-net"):
    exec("%s='%s'" % (v.replace("-", "_"), v))

project = {
    "src_path": src_path,
    "ins_path": ins_path,
    
    "environment": {
        "DYLD_LIBRARY_PATH": "%s/lib:%s" % (ins_path, env("DYLD_LIBRARY_PATH","")),
        "LD_LIBRARY_PATH": "%s/lib:%s" % (ins_path, env("LD_LIBRARY_PATH","")),
        "C_INCLUDE_PATH": "%s/include:%s" % (ins_path, env("C_INCLUDE_PATH","")),
        "ACLOCAL_PATH": "%s/share/aclocal" % ins_path,
        "PKG_CONFIG_PATH": "%s/lib/pkgconfig" % ins_path,
        "XDG_DATA_HOME": "%s/share:%s" % (ins_path, env("(XDG_DATA_HOME","")),
        "XDG_DATA_DIRS": "%s/share:%s" % (ins_path, env("XDG_DATA_DIRS","")),
        "PATH": "%s/bin:%s:%s" % (ins_path, ins_path, env("PATH","")),
        "PS1": "[mono] \\w \$? @ ",
    },

    "packages": {
        libgdiplus: {
            "svnurl": "%s/%s" % (svnbase, libgdiplus),
            "configure": conf,
            "build": build,
            "install": install,
        },
        mcs: {
            "svnurl": "%s/%s" % (svnbase, mcs),
            "deps": [libgdiplus],
        },
        olive: {
            "svnurl": "%s/%s" % (svnbase, olive),
            "deps": [libgdiplus],
        },
        mono: {
            "svnurl": "%s/%s" % (svnbase, mono),
            "configure": conf,
            "build": "make get-monolite-latest && %s" % build,
            "install": install,
            "deps": [libgdiplus, mcs, olive],
        },

The only thing that gets read is the dict in the global scope called project. And then it has an optional member called environment, and the packages declared under packages. It should be pretty easy to figure out. If src_path isn't set fetching defaults to /tmp.

Anyway, at about 500 lines it's decent bang for buck I think. It's a classic 80/20 effort, getting 80% payoff for 20% effort.

Love it or hate it, here's the goodies:

:: random entries in this category ::

4 Responses to "repeatable one off builds with 'emerge'"

  1. piyo says:

    > I know I should have picked a different name

    Yes!

    > but the parallels are so clear that I just couldn’t resist.

    No. Gentoo emerge handles packging and uninstall.
    So what do you call your emerge when you're using gentoo? ;-)

    This kinda reminds me of Technomancy's roastbeef.

  2. numerodix says:

    Well, I really love the name emerge :) And besides, it's modeled on it very closely. If you know how to use Gentoo's emerge then noone needs to tell you how to use this emerge, because you already know. And I think it's really nice if you can give users credit for having learned something by being able to reuse that familiarity.

    Well, I call the regular one "emerge" and I call this one "~/code/emerge/emerge" :)

  3. loki_val says:

    Better idea, emerge --sync, look into /usr/portage/profiles/package.mask, get the atoms you want to unmask, for example for mono-2.4-1 branch (dev-lang/mono-2.4.9999 and friends), put them into /etc/portage/package.unmask and let portage do its magic
    :-)

  4. numerodix says:

    Except if I'm not on Gentoo then that doesn't work.