Build environment of The Meta-Environment

Gerco Ballintijn

Taeke Kooiker

Tijs van der Storm

Jurgen Vinju

2008-04-25 13:19:50 +0200 (Fri, 25 Apr 2008)

Table of Contents

Introduction
The 'meta-build-env' package
Usage
Files
Building Meta-Environment packages from Subversion using Sisyphus
How to use boot-roll and roll to checkout and build software
Installing Sisyphus

Introduction

All packages use the following tools as a build environment:

  • GNU make, automake, autoconf, libtool, autoreconf

  • For C projects we use GCC

  • For Java projects we use Sun SDK

  • For Java projects we use ant (called by make)

  • Sisyphus for combining Subversion checkouts, and calling build commands in the correct order according to the package import structure

  • Autobundle for composing distributable source packages (composed from individual packages automatically by autobundle)

This document is incomplete, please help. You may find The Goat Book documentation useful.

The 'meta-build-env' package

Since we use autoconf, automake, gmake, and libtool, and we have a rather modular package system, building a package can become complex. The package meta-build-env helps to make this easier. The goals of this package are:

  • to provide each package with the same build interface

  • to have no duplication/cloning of m4 macros and make rules

  • to hide autoconf/automake ideosyncrasies as much as possible (not 100% possible)

  • to have a single 'bill-of-materials' file for each package that can be used by other tools (like sisyphys, pkg-config, autobundle, etc.)

Usage

To use the build environment, you first have to check it out from Subversion, then configure, make and make install it. This installs the tool meta-build, which can be seen as an extension of aclocal. Now this is what you need to do to use the build environment for a certain package:

  1. Go to the root of the package.

  2. Call meta-build (or ./reconf as explained below)

  3. Use the normal configure, make, make install, make distcheck routine that is implemented by automake and autoconf.

The convention is to let the ./reconf script do the call to meta-build. It copies all relevant build environment files to the local directory, or makes a softlink to these files. From now on, the package is independent of the meta-build-env package, and can be distributed without it.

Files

The following files are installed by meta-build. The use of these files is obligatory, not optional. This rule makes sure that all packages implement the same interface.

  • meta-build.m4: macros that extend autoconf to use pkg-config files to retrieve dependencies, and to set up a Java build environment, etc.

  • Makefile.top.meta: Makefile to include in all toplevel Makefile.am's of C-based packages

  • Makefile.java.meta: Makefile to include in all toplevel Makefile.am's of Java-based packages

  • Makefile.meta: Makefile to include in all other Makefile.am's

  • meta-build.ant: ant targets for Java compilation

The following files should be present in the toplevel directory of each package:

  • configure.ac: calls AC_INIT, then calls META_SETUP, and then META_JAVA_SETUP if it is a Java package, and calls appropriate AC_ macros to resolve C configuration issues.

  • package.pc.in: contains most information about the package, including the name and the version and the dependencies. This file is used by meta-build.m4 to generate configure flags and to set up the CLASSPATH, and all other things needed for modular source tree composition.

  • ./reconf: contains a call to meta-build

  • ChangeLog: contains a description of all changes (usually equal to the commit message, but sometimes more extensive, or on a slightly higher level of abstraction)

  • Makefile.am: includes either Makefile.top.meta or Makefile.java.meta, and declares the SUBDIRS variable, contains

    ACLOCAL_AMFLAGS = -I
  • COPYING: contains a copy of the GNU Lesser General Public License

This is an example package.pc.in file for a C based package:

prefix=@prefix@

Name: toolbuslib
Description: ToolBus adapter library
Version: 0.8
Libs: -L${prefix}/lib -lATB @SOCKETLIBS@
Cflags: -I${prefix}/include
#uninstalled Libs: -L@abs_top_builddir@/src -lATB @SOCKETLIBS@
#uninstalled Cflags: -I@abs_top_builddir@/src
Requires: aterm

The prefix line is obligatory, as well as Name, Description and Version. The #uninstalled lines declare the location of exported libraries in the local source tree. This is used to compose source trees. Libs and Cflags declare the location of headers and libraries after installation. This file is constructed by configure, so you may use configure variables that you introduced in configure.ac (e.g. @SOCKETLIBS@).

This is an example package.pc.in file for a Java based package:

prefix=@prefix@

Name: aterm-java
Description: Java version of the ATerm library
Version: 1.6.3
Requires: shared-objects,JJTraveler
JarFile=aterm-java-1.6.3.jar
Packages=aterm,aterm.pure,aterm.test
LocalJars=
ExternalJars=@EXTERNAL_JARS@
TestClass=

Some extra variables are declared (notice the '=', instead of the ':'). Java packages all have the same directory structure. The sources must be in './src'. The extra variables are used to configure ant to do it's job. They are all obligatory. The ExternalJars=@EXTERNAL_JARS@ line is only needed if the packages has dependencies in the 'Requires' field. EXTERNAL_JARS will be substituted by all Jar files needed to compile and run this package, but you may add your own external dependencies here. LocalJars can be used to add jars that are needed to build the package, but are not build from Java source directly. These are typically third party packages like junit.jar.

Building Meta-Environment packages from Subversion using Sisyphus

To set up a build environment for the Meta-Environment can be hard because:

  • It is so modular, that it is hard to know which packages to checkout, configure, build etc, and in which order.

  • The configure and build interface of the packages (using the GNU autotools) is sometimes hard to remember.

Sisyphus is a tool that manages this complexity for us. For one thing, we use it in our continuous integration, but we can reuse it to construct a build environment for a developer.

Sisyphus takes a top-level package as argument, and a Sisyphus configuration, and uses the central repositories to find out in which order packages need to be build and with which commands. This information is cached locally in a .rollrc file. The .rollrc file is constructed by running boot-roll once.

How to use boot-roll and roll to checkout and build software

Step 1: get access to the Sisyphus tools boot-roll and roll

You can get access to boot-roll and roll as follows:

  • When you are working in the CWI domain add /ufs/sen1/software/installed/sisyphus/linux/i686/bin to your PATH or add 'sisyphus' to your .pkgrc file.

  • Otherwise, you should install Sisyphus yourself as explained below.

Step 2: Initialize boot-roll

Make a file .bootrollrc in your home directory and edit the build_dir, install_dir and vcs_user variables according to your settings. The following is an example .bootrollrc file:

boot_roll:
  build_dir: /ufs/<yourlogin>/glt/build
  install_dir: /ufs/<yourlogin>/glt/installed
  vcs_user: <yourlogin>
  roots: [asfsdf-meta]
sisyphus:
  url: sisyphus-configuration
  host: sjofar.sen.cwi.nl
  port: 3960
  protocol: svn
  user: daybuild
  password: daybuild
templates:
 reconf: ./reconf
 configure: CFLAGS="-g -O2 -m32 -Wall -Werror" ./configure --prefix=/ufs/<yourlogin>/glt/installed <%deps.each do |d|%>--with-<%= d.name%>=/ufs/<yourlogin>/glt/installed <%end%> --enable-maintainer-mode
 make: echo 'use make install'
 install: CFLAGS="-g -O2 -m32 -Wall -Werror" make install
 check: CFLAGS="-g -O2 -m32 -Wall -Werror" make check
 distcheck: CFLAGS="-g -O2 -m32 -Wall -Werror" DISTCHECK_CONFIGURE_FLAGS="<%deps.each do |d|%>--with-<%= d.name%>=/ufs/<yourlogin>/glt/installed <%end%>" make distcheck
 clean: make uninstall clean 
 update: make uninstall ; svn update
 maintainer-clean: make uninstall maintainer-clean

If your workstation is outside the CWI domain, it is necessary to use a tunnel specification. Replace the sisyphus tag with the following to tunnel from ssh.cwi.nl to sjofar.sen.cwi.nl:

sisyphus:
  url: sisyphus-configuration
  host: sjofar.sen.cwi.nl
  port: 3960
  protocol: svn
  user: daybuild
  password: daybuild
  tunnel:
    host: ssh.cwi.nl
    user: <yourlogin>

Warning

The connection with the Subversion repository uses the Secure Shell. Be aware that this set up will not work if you use password-protected identities like id_rsa as used by the Secure Shell. Currently, the only solution is to remove those passwords.

Step 3: Run boot-roll

Run the command:

boot-roll

This generates a file '.rollrc' which contains, amongst others, the dependencies between the packages. The .rollrc file will be used by subsequent calls to the roll command.

Step 4: Add your installation bin directory to your PATH environment variable

export PATH=/ufs/<yourlogin>/glt/installed/bin:${PATH}

Adapt to the syntax of your favorite shell.

Step 4: Run roll

As explained earlier, each package can be built by way of the standard operation sequence: reconf, configure, install. For first time installations, it is mandatory to apply these operations to each package first before moving on to the next package. This is achieved by executing:

roll -s reconf configure install

The option '-s' means apply to each package the sequence of operations that follow it. roll -h and roll -l give helpful hints on the usage of roll.

The command roll is no substitute for understanding the build interface of the Meta-Environment. It just automates a number of repetitive tasks, and manages the order in which they are executed. You are expected to fully understand the contents of your .bootrollrc file and your .rollrc file.

Installing Sisyphus

The following steps are needed for a full installation of Sisyphus:

  • install ruby-1.8.x

  • make sure your ruby binary is called 'ruby' and not 'ruby1.8' or something similar.

  • install rubygems, using the following script:

    wget http://rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
    tar zxvf rubygems-0.8.11.tgz
    cd rubygems-0.8.11
    sudo ruby1.8 setup.rb
    sudo gem install rubygems-update
    sudo gem install rails
  • install Subversion

  • download sisyphus from the sisyphus website

  • ./configure --prefix=/your/prefix

  • make install