Fun with ClassAds

One of the new technologies the OSG Technology area is working on is the HTCondor-CE.  While that is a topic for a different post, it led me on a surprising journey over my Christmas break.

Working with the HTCondor-CE, I found that creating a job hook to be surprisingly difficult.  A job hook for HTCondor is an external script, invoked by HTCondor in lieu of running internal logic. This allows a sysadmin to add custom logic to HTCondor internals without resorting to writing C++ code.  The hook in question is the job transformation step for the JobRouter.

The problem with hooks is they are surprisingly difficult to write.  For the transform hook, a job's ClassAd is written to the script's stdin and the JobRouter expects to read the transformed ClassAd from stdout.  [Actually, it's a touch more complicated than that, but this simplification will do for our discussion.]  ClassAds are an expressive and powerful language - but a language difficult to parse via Unix scripting!  There are complex quoting and attribute evaluation rules.

Sysadmins are left with a decision - either spend quite some time implementing a ClassAd parser or only do the bare minimum and hope no one submits a complex ClassAd.  I found the situation unsatisfactory and decided to write python bindings for the ClassAd library.

I found the endeavor fairly straightforward using the Boost.Python library, and ended up with a new GitHub project.  Now, a job transform hook is as simple as this:

#!/usr/bin/python

import sys
import classad

route_ad = classad.ClassAd(sys.stdin.readline())
separator_line = sys.stdin.readline()
assert separator_line == "------\n"
ad = classad.parseOld(sys.stdin)

ad["Universe"] = 5
ad["GridResource"] = "condor localhost localhost"
if "x509UserProxyFirstFQAN" in ad and "/cms" in ad.eval("x509UserProxyFirstFQAN"):
    ad["AccountingGroup"] = "cms.%s" % ad.eval("Owner")
else:
    ad["AccountingGroup"] = "other.%s" % ad.eval("Owner")

print ad.printOld(),

The above script will read the ad from stdin and change the AccountingGroup
attribute based on the contents of the x509UserProxyFirstFQAN attribute.

Note ClassAds can be constructed from a string or a file object.  Each ad can be treated like a python dictionary.  Literals are converted to the equivalent python objects; expressions are exposed as objects.  For example:

>>> import classad
>>> ad = classad.ClassAd()
>>> expr = classad.ExprTree("2+2")
>>> ad["foo"] = expr
>>> print ad["foo"]
2 + 2
>>> print ad["foo"].eval()
4

Most of the functionality is exposed; see the GitHub project for examples and unit tests.  To make the C++ library safe to export to python, some minor semantics have been changed.  Sub-ClassAds and Lists are not yet available via python, but shouldn't be too hard to add.

ClassAd Python bindings - maybe not the most life-changing software project in the world.  However, they have potential to become one of life's little pleasures for those of us who deal with HTCondor every day!