Ant Build Templates: The Script

Posted on Tuesday, September 23rd, 2008 at 8:59pm by Rob Wilkerson

This is the sixth and final post in a series of posts meant to outline my Ant build script template. If you’re interested, it may be helpful to read the first five:

1. Prelude
2. Environment
3. Process
4. Why Two Files?
5. External Tasks

It took entirely too long to get here, but here it is, the scripts themselves. I was trying to decide what I should write to accompany these scripts, but the truth is, I’ve probably written too much already. There may be some properties or tasks that could stand for some additional clarification, but I think both files are pretty well documented so I’m just going to set them loose on the world. Most questions are probably answered either in the script documentation itself or in one of the previous posts.

To use this script:

1. Ensure that Ant is correctly installed. Yeah, I know it’s obvious, but if I didn’t say it you just know I’d get questions.
2. Ensure that the required external tasks are installed.
3. Download build.properties.sample and build.xml to any directory, but keep both files in the same directory.
4. Copy build.properties.sample to build.properties.
5. Modify the newly created build.properties file, as required.
6. Modify build.xml, as required.
7. Execute, execute, execute.

Feel free to ask questions or make suggestions either in the comments for this post or by using the contact form.

ant, build 0 Comments

Ant Build Templates: External Tasks

Posted on Friday, September 19th, 2008 at 4:43pm by Rob Wilkerson

This is the fifth in a series of posts meant to outline my Ant build script template. If you’re interested, it may be helpful to read the first four:

1. Prelude
2. Environment
3. Process
4. Why Two Files?

So at this point in this thing that I’ll loosely call a series, I’ve talked a lot, but in the put-up-or-shut-up debate, I’m still leaning heavily towards the shut up side. Today I hope to move that needle, at least a little bit, in the other direction.

Ant is the cat’s meow. The bee’s knees, if you’d rather. But then, I probably don’t need to sell anyone on the fact that I’m a fan. What Ant is not, though, is perfection. At least not without help. My build script requires three external task libraries to function properly. Technically, it’s two external libraries and one external dependency for a native task library, but at least we’re starting to talk code. First, a little information about the required tasks and then some information about making them work in the build script.

SSH and SCP

Fundamental to any build that includes deployment is the ability to move resources from one server to another. Unless absolutely necessary, I prefer to use SCP or, at the very least, SFTP. Both of these are better than FTP, but my infrastructure supports SCP, so that’s what I use. My build template also requires SSH, the protocol on which SCP is built, so there’s a nice two-for-the-price-of-one effect. Ant ships with SSH and SCP tasks but, perhaps counter intuitively, not with the library required to actually execute those tasks, jsch, a pure Java implementation of the SSH2 protocol.

There’s more information in a post I wrote over a year ago, but be sure to download the 0.1.29 version even though it’s not current. I’ve tried every version between 0.1.30 and 0.1.33 and they don’t work. Versions above 0.1.33 may work, but I can vouch for the fact that 0.1.29 works so it may be worth sticking with that.

SvnAnt

For years now, I’ve been using Subversion as my source control solution. I’m taking some early stabs at learning git, but I’m still primarily a Subversion user. It’s not hard to figure out, I’m sure, that SvnAnt is a collection of Ant tasks for interacting with Subversion. This library supports the full range of commands available with Subversion (certainly every one that I need) in addition to the ability to connect using an installed svn executable or via JavaHL.

AntelopeTasks

Antelope is a very useful – so useful that I actually think of it as critical – library of utility tasks designed to give Ant more power. The library includes conditionals, try/catch, grep, string manipulation and several other handy tasks that are used throughout my build script. Its capabilities are similar to those of AntContrib, but I think that Antelope offers more functionality and more options.

Including External Tasks

There’s an easy way to run external tasks in a build script and a hard way. I actually prefer the hard way. The easy way is to simply download the task library and extract its jar files anywhere in the system classpath. A typical approach would be to drop the jar files in the ANT_HOME/lib/ directory. The “hard” way is to extract the libraries anywhere at all and tell the script where they’re located. Sort of a lazy loading technique, I suppose. I prefer the latter method because it allows me to keep my external tasks, well, external.

I extract my external libraries to ~/Library/ant/tasks/ (again, let me state that this can be anywhere). I then set a few variables in my build.properties file:

tasklib.dir=/Users/me/Library/ant/tasks
<!--
   Libraries associated with the SvnAnt task
 -->
tasklib.svnant.tasks=${tasklib.dir}/svnant-1.0.0/lib/svnant.jar
tasklib.svnant.adapter=${tasklib.dir}/svnant-1.0.0/lib/svnClientAdapter.jar
tasklib.svnant.javahl=${tasklib.dir}/svnant-1.0.0/lib/svnjavahl.jar
<!-- 
   Antelope only has one library
 -->
tasklib.antelope.tasks=${tasklib.dir}/AntelopeTasks_3.4.2/
AntelopeTasks_3.4.2.jar

Note that any wrapping that occurred in this list of properties is a function of the limited display space on this site. Each property should appear on a single line in the properties file itself. In this code, any line that does not begin with “tasklib” (and is not a comment) is wrapped from the line above. I couldn’t think of a way to indicate that in the code without it being more, rather than less, confusing.

Once I have properties to indicate where my tasks reside, I can explicitly include them in my classpath and load the tasks by adding this code to my build.xml file:

<!-- load properties set in the build.properties file -->
<property file="build.properties" />
<!-- 
   add external library file paths to  the project classpath
 -->
<path id="project.classpath">
   <pathelement location="${tasklib.svnant.tasks}" />
   <pathelement location="${tasklib.svnant.adapter}" />
   <pathelement location="${tasklib.svnant.javahl}" />
   <pathelement location="${tasklib.antelope.tasks}" />
</path>
<!-- make the svn tasks available -->
<taskdef resource="svntask.properties"
         classpathref="project.classpath"
/>
<!-- make the antelope tasks available -->
<taskdef resource="ise/antelope/tasks/antlib.xml"
         classpathref="project.classpath"
/>

That covers SvnAnt and Antelope, but the jsch library still needs to go somewhere. I treat that one a little differently, but the difference makes sense in my head. Since jsch is an external library that fulfills a dependency for a native task, I just drop the library in my ANT_HOME/lib/ directory. There it gets picked up automatically.

Alternatively, if I plan to run builds exclusively from within Eclipse (which is often the case), I might store it in ~/Library/ant/lib/ and tell Eclipse to find it there in its preferences. To set that preference:

1. Go to Preferences > Ant > Runtime
2. Select the Classpath tab
3. Highlight Global Entries
4. Select Add External JARs
5. Navigate to jsch-0.1.29.jar, wherever its located
6. Click Open in the file browser dialog
7. Click OK in the Eclipse preferences dialog)

With external libraries in place and properly added to the classpath, previously unavailable or unusable tasks are now available for use. Up next: the unveiling.

ant, build 0 Comments

Ant Build Templates: Why Two Files?

Posted on Monday, September 15th, 2008 at 4:38pm by Rob Wilkerson

This is the fourth in a series of posts meant to outline my Ant build script template. If you’re interested, it may be helpful to read the first three:

1. Prelude
2. Environment
3. Process

If you’ve been reading this series (or any of my other Ant-related content), you may have noticed that I occasionally reference my build scripts. Plural. It’s kind of involuntary, but you may have wondered exactly how many there are. Although there’s really only one build script (per project), I have, and recommend to anyone else, an accompanying properties file. Two files that I occasionally, and inconsistently, mind you, lump together into a plural use of the word “script”.

So there it is. Two files. One is the actual Ant build script, traditionally build.xml and the other, build.properties, an external set of property definitions. I do this for exactly the same reason that I – and most other developers I know – maintain a configuration file of some sort for my applications; it’s nice to know where to go when something needs to be changed. Moreover, it’s nice to know that it’s the only place that those changes need to be made.

Build operations, especially when deployment is part of that build operation, typically require an awful lot of information – file paths, for example – about the build or deployment environments, information – think usernames – about the user running the build, sensitive information like authentication credentials or simply information about the build itself. The latter might include the project name and version or the source control revision number.

All of this information is usually, or at least could be, variable. I deploy to multiple environments and I’m not the only person in my group that can run builds. Additionally, I want developers themselves to test their own build scripts. Finally, I want build scripts to be versioned and I sure as hell don’t want to be dropping my authentication credentials into a build file and then committing those credentials to a revision all their own until the end of days. That would be bad.

Given all of this variable and/or sensitive information, I could still just throw everything directly in the build script. I have to manage and update it no matter where it goes, so the where doesn’t really matter. Um, in theory. Maybe it’ll be easier to understand when you see the file, but the build script (build.xml is much more verbose. There’s a lot of stuff going on in there. Remembering what needs to change, finding it and then changing it isn’t a trivial effort and it would have to be done for every build. No thanks.

Instead, I move the variable data out to build.properties. In addition, I store a generic version named build.properties.sample in source control. That generic version contains dummy values for most of the properties. I copy that file to build.properties in my working copy, set my source control configuration to ignore the copied file and then I can merrily go about including sensitive data without fear of an accidental commit. There are also other benefits to having a separate properties file:

1. It contains nothing but variable values. If a property is in this file then it either is or could be variable. There’s no other stuff to wade through.
2. Likewise, the build script itself is fixed. Nothing in there should need to be changed (unless, of course, the fundamental nature of the build changes).
3. It’s much, much smaller than the executable script so specific properties can be found much quicker that they would be in the larger build script.
4. Specific properties will only exist once in the properties file. They may be used any number of times in the script itself.
5. Risk mitigation. At build time, I don’t have to touch the executable that’s actually doing all of the work. All I have to do is update the properties file and run the build.

The moral of the story: use a properties file. You’ll be glad you did. Your sanity is important so preserve it. In this case, less is not more. Two is better than one.

build, ant 4 Comments

Ant Build Templates: Process

Posted on Saturday, September 13th, 2008 at 9:40am by Rob Wilkerson

This is the third in a series of posts meant to outline my Ant build script template. If you’re interested, it may be helpful to read the first two:

1. Prelude
2. Environment

As it was with the environment, so it is with the process. Ultimately, my script was designed and constructed to facilitate my process. With that in mind, it may be useful to know a bit about that process and the standards within it.

We work in a loose, Scrum-like environment. I don’t know exactly how to explain that except to say that we keep our iteration cycles short, but I don’t think anyone would consider our process a formal one. We’re still working on hammering that out. We define a project version and, in two-week cycles, conspire to deliver that version. At the end of each cycle, we build (no, we’re not rocking continuous integration yet) and deploy to staging. At the end of any such cycle, after seeing the project in staging, the business owner has the option of pushing that release into production. As soon as a project goes to production, the version is locked. Any future pushes to production increment either the major, minor or maintenance version.

Project version numbers are formatted as major.minor.maintenance.revision. This is pretty common, in my experience. When a project build is pushed to staging, it is tagged in source control (Subversion, in my case) as /tags/build/tag-name. Projects can only be pushed to production if the exact same revision is used as that which is approved in staging. When pushed to production, a tag is created as /tags/release/tag-name. For historical purposes, I find it helpful to know, at a glance, what revisions have been pushed to production.

That’s it. Somehow it seems more complicated from the inside, but I can’t think of anything else that’s relevant to the build script itself, so I’ll keep this short and consider it a win. Next up: Two Files or One? Now we start to dive into the fun stuff – the scripts themselves and the thought processes behind them.

ant, development process 2 Comments

Ant Build Templates: The Environment

Posted on Wednesday, September 10th, 2008 at 7:34pm by Rob Wilkerson

Second day of my Ant template lead up, second post. Hello, inertia. Nice to meet you. Check out the prelude, if you haven’t already.

Although my script is templated, it’s templated for my environment – both corporate and personal. Because I’m fortunate enough to be able to guide our corporate infrastructure, it’s been molded in the image of my personal environment. The script has certain expectations that, within the context of that infrastructure, are entirely fair and valid. Outside of that infrastructure, those expectations may not be valid. As such, I think it might be helpful to know a little bit about my environment; it may assist in reading the scripts and in understanding where changes could or should be made.

As is the case in most shops I’ve ever seen or worked in, I have three clusters: development, staging and production. There are also infrastructure and configuration aspects that are shared. We’ll start with the “universal” components.

Universal Elements

To dispense with the generics, the environment is a LAMP stack. Sure, there are some outlier projects that run other packages, but those can be ignored in this context.

The File System

In addition to a directory reserved for shared services that has been added to our include_path in %(technical)php.ini, we have any number of project roots (one per project, as you might expect). Project roots are organized like so:

<project root>/ _meta/ classes/ html/ maint/

The _meta/ directory contains any versioned files and code that are not required at runtime. For example, documentation and the build script itself. In fact, one of the things the build does is delete this directory once the deployment is complete. This directory does contain files that may be used during deployment like SQL files to create a database or shell scripts to aggregate any number of actions.

The classes/ directory contains the project’s supporting class files. These aren’t part of the web root, but are added to the include_path for the project. This keeps business logic outside of the web root. Handy for all of that top secret, for-your-eyes-only work I do. Heh.

The html/ directory is the project web root and contains all of the runtime files that need to be available to the web server.

The maint/ directory contains resources to display a maintenance screen. Slightly oversimplified, if the file /maint/maintenance.htm exists, it will be displayed. Period. One of the first tasks the build script performs is to enable the maintenance screen so that users aren’t greeted with a big server area.

NAS Mount

Most projects I’ve been involved with involve some degree of what I call user-contributed content. What I mean is that users can upload files to the system. Sometimes that means a lot of files. I consolidate that content in a single directory, named bin/, that sits in the web root. To help manage and serve that content, the bin/ directory and its subdirectories are NAS mounts. Every project has one and it’s created when the project is created on the server. For developers, it just exists.

The build must adjust for the fact that it can’t be deleted, nor can its permissions be altered. Hence its mention here.

Development

I do the bulk of my work locally as, I believe, should all developers. It just eliminates so many potential problems. It’s also a good learning experience. I think all web developers should be able to perform at least basic debugging and maintenance tasks on their web and database servers, etc. That said, I do have a development cluster that serves to meet needs that can’t be met on my laptop.

My development cluster includes a deployment server where developers can test, what else? The build script they’ve tailored to their own application and its deployment needs. It’s also available to test integration with third party systems that may not be available to the desktop environment, to provide shared access where business owners and stakeholders can review progress, etc.

In the development cluster, all machines have the following naming convention:

machinename.dev.domain.tld

For example, buildsrv.dev.robwilkerson.org.

Staging & Production

The goal, of course, is to have staging mirror production as closely as possible. I think this is the goal in every shop and it’s no different here. We do a pretty decent job, I think, so I feel comfortable lumping these two environments for the purpose of discussion.

Machines in these clusters are predictably named according to the following convention:

stagingmachine.stg.domain.tld productionmachine.prd.domain.tld

I’m admittedly not as anal as I should be about the naming convention (which is surprising for me). “dev” is sometimes “devel”, “prd” is sometimes “prod”, etc. The script covers all of my typical variations, as you’ll see.

In order to lock down these machines, we have a specialized build user with developer-level access to touch the areas of the file system required by builds. That user has specialized sudo privileges that allow it to do whatever it needs to do within the context of the build. Similarly, we have a debug user with those same privileges. Once the build is executed on one of these clusters, developers may need to physically access the machine in order to debug any problems. This user is locked by default and rarely unlocked, but is available if the need arises.

So that’s the world my template lives in. Hopefully, by understanding the differences between my world and yours, it will be easier to make any necessary modifications. Next up: the deployment process that my script must facilitate. Stay tuned.

configuration, ant 0 Comments