Agile Application Development: Romilly Cocking's Review

Comfort for Application Developers from Tom de Marco

2011-11-30T09:35:00.001Z

Thanks to a tweet from @flowchainsensei, I've just been enjoying an excellent article by Tom de Marco. It's well worth a read; I took away two key thoughts.

Developing software is much harder than developing hardware
Software developers spend much of their life feeling bad about late delivery when it's usually not their fault.

To put his core conclusion another way, most software projects are a waste of effort. They have such low marginal utility that they are doomed never to show a return for the organisation that funds them. If we just didn't do them the world would be better off.

Of course, we'd need to find work for the ex-developers. Maybe they could learn to teach. We could certainly use a few more skilled IT teachers.

I2C on the Beagleboard xM

2011-11-04T17:12:00.000Z

It's really easy driving I2C™ devices from the Beagleboard. I'm using the Trainer-xM from tincantools; that takes care of shifting voltage levels from the 1.8v used by the Beagleboard to the 5v levels that my I2C boards expect.

There are a few I2C coding examples on the web; since my BB image includes python and geany, I've written the I2C code in Python and can edit and test it on the board. Much faster than an edit-compile-upload-test cycle.

Here's the code:

import i2c, time

extender = i2c.I2C_device(0x20)
extender.begin_transmission()
extender.send(chr(0))
time.sleep(5)
for count in range(0, 16):
    extender.send(chr(count))
    time.sleep(0.5)
extender.end_transmission()

See it running.

From little Acorns do great Raspberries grow

2011-11-03T17:15:00.001Z

I recently re-discovered an old friend: the great-great-grandfather of the ARM-based Raspberry Pi . It's the original Acorn micro-computer (ca. 1979). Acorn went to on make a succession of machines, including the BBC Micro and the Archimedes.

The Archimedes was based on a proprietary RISC chip which is the grandfather of today's ARM chips. The Raspberry Pi is part-funded by Dave Braben, who wrote Elite (arguably the most famous BBC micro game). Broadcom, who are make the Pi's ARM chip, have strong historic links with Acorn.

I'll try the micro out in a day or two; I still have the (fairly minimal) original documentation, and there is more information on Wikipedia. The micro was arguably one of the best investments made by Cocking and Drury; my colleague Jonathan Barman used it to learn assembler, and went on to master 370 Assembler, through which he landed us a major contract with a national brewery.

Blogging from the Beagleboard

2011-10-15T11:53:00.000+01:00

I'm posting this using the Midori browser running under a custom Angstrom image on my brand-new Beagleboard Xm. The Beagleboard continues to amaze me.

Setting up a fresh sd card with a custom image has proved a little tricky; it's taken about 4 hours, of which one was spent waiting for a very slow Kingston microSD card to get configured on first boot. I'll write up the process and post it later this weekend.

Busy with the BeagleBoard

2011-10-13T16:43:00.001+01:00

Yesterday I decided that the time had come to get to grips with embedded Linux. I ordered a BeagleBoard Xm from Farnell, a suitable wall-wart from Maplin, and started counting the hours until they arrived.

Both turned up this morning and my brand new BeagleBoard is up and running. My biggest worry was the possibility of blowing the board by connecting the power supply the wrong way round; maybe the manual tells you about the required polarity of the power plug, but I couldn't find the information anywhere. As usual, Google was my friend; it turns out that the board requires a centre-positive, earthed shield plug. The Maplin PSU is configurable, but the required polarity is the opposite of what the Maplin markings suggest is the norm. I strongly advise you to check the polarity of the plug with a multi-meter before you plug it into your precious BeagleBoard.

I'm still waiting for a couple of microSD cards; Argos has them at at three times the price I paid on Amazon, and for now I'm happy playing with the test distribution which comes with the kit.

The test distribution is a usable version of Angstrom. I followed the great getting started guide on IBM developerworks. It's helpful, but a little out-of-date; it suggests that the test distro is ram-disk based and loses changes to the filesystem on re-boot, but that's not the case. I took the screenshot shown below during my first session and it was still sitting on my desktop when I rebooted.

As you can see from the screenshot, the distro comes with a lightweight but functional web browser and many of the tools you'd expect to find in a desktop environment. It doesn't come with the nano editor; perhaps I'll finally get to grips with vi. (Doh. It's got gedit. I can stay a mouseman). It does include Python, which makes me very happy.

I started writing this post while the distro was updating. It's now finished.

This inexpensive board has much, much more computing power than the Atlas - a supercomputer I programmed in the late 60s. When the first Atlas was commissioned, people said that the UK's computer power doubled overnight.

I'm looking forward to a couple of months' happy ~~play~~ research with the BeagleBoard - and by then, with any luck, I'll be able to play with a Raspberry Pi.

VMWare: how to lose a customer in one easy lesson

2011-09-17T21:36:00.000+01:00

A while ago, when I was developing a lot of courseware, I got hooked on virtualisation. Virtual Machines are a great way to create a repeatable course environment, and if your course is Linux-based you can distribute the course images freely. I started off using VMWare; their Windows workstation was inexpensive, and their player was free.

I stuck with VMWare for years, and used it for more and more applications. In the good old days, when Dell were after market share rather than margin, I picked up a couple of servers for about £250 each - dual-core machines with 4Gb of ram and decent hard drives. I ran Ubuntu on them, and used VMWare Server to host several virtual machines.

All worked fine until VMWare announced that Server had reached end-of-life, and their remote desktop stopped working with current browsers. I found myself with a set of unusable VMs which represented many weeks of development and configuration.

At the time I was working on a contract a few desks away from Stuart Ervine, and he suggested I try VirtualBox. It's proved to be a more-than-adequate replacement for VMWare, who won't be getting business from me or the clients I advise in future. Of course VMWare were within their rights to end-of-life a free product, but they caused me enough grief to destroy six years of brand loyalty at a stroke.

Leiningen and Eclipse

2011-09-17T11:47:00.001+01:00

I'm now fairly comfortable with leiningen, though I still feel a frisson of anxiety whenever I run it. I know that Maven is lurking there somewhere, and my experiences with Maven have not been happy.

I'm also a bit leery of letting lein download and install arbitrary software from external sources. I'll need to set up a local repository, which will mean a closer brush with Maven that I've had before.

It's annoying, but seems unavoidable. Clojure libraries and tutorials generally assume that you're using leiningen.

At least the lein/Eclipse combo makes it simple to create and populate an Eclipse Clojure project.

Here's what you do:

In the directory in which you want to create your project, run lein new
cd to the newly-created directory
open project.clj in your text editor of choice
add :dev-dependencies [[lein-eclipse "1.0.0"]]
save the file
run lein deps
run lein eclipse
import your newly-created project into eclipse

At the end of stage 4, your project.clj file should look like this

(defproject hadoop-spike "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.2.1"]]
  :dev-dependencies [[lein-eclipse "1.0.0"]])

WabbitMQ runs too

2011-09-17T11:31:00.000+01:00

Off to a good start this morning;I set up a Clojure spike using WabbitMQ and got a simple test running in a few minutes. Now I can link up my Python and Clojure components. Eeeeeeeexcelent.

Run, RabbitMQ, run Rabbit, run, run, run

2011-09-15T15:38:00.000+01:00

I'm about to do some more work on Sknoman (a personal knowledge management application) and need to link together several disparate subsystems. Some are written in Python, but I'm developing one in clojure.

Nat Pryce speaks highly of RabbitMQ, so I decided to give it a try.

For reasons too boring to explain, I'm developing on a Ubuntu Lucid virtual machine. Lucid is almost as old as I am, so the packaged version of RabbitMQ is seriously out-of-date.

Once I'd installed an up-to-date version from the RabbitMQ Debian repo, I had the RabbitMQ Python send/receive example running in about five minutes. While that's hardly an exhaustive test, it's very reassuring.

I'll be trying a Clojure client next, but I'm hoping this is going to be painless.

PrecisionIR - great service, terrible software

2011-02-11T08:37:00.000Z

PrecisionIR - how not to run a service

I've been using the free annual reports service provided by PrecisionIR. It's a great service; you can get hundreds - no, thousands - of free annual reports and daily updates about the companies that you are interested in.

Twice now PrecisionIR has emailed me a survey abut their service. They say it's a short survey, and it is. Each time it's crashed on the second response.

The second time it happened I decided follow the advice in the error page and email support at PrecisionIR. The email bounced.

I guess they don't do a lot of testing.

Catching the train in J

2010-09-03T08:20:00.000+01:00

My mini array engine in Clojure is coming on apace, and another day should give me the functionality I want to right now.

I've been checking my specification against the behaviour of J, and this has had an interesting side effect.

I've tried to learn J several times. Each time I quickly mastered the basics but then hit a brick wall in understanding.

This time around the wall has collapsed.

The concept I was missing was that of a train of verbs. A verb in J is another name for a function. A train is an isolated sequence of verbs. For some reason I couldn't grasp why these were needed or how they worked, but now the need and its solution seem obvious. As a result I can read (and write) programs in J which previously eluded me.

Old habits die hard, if ever

2010-09-01T10:51:00.000+01:00

I've taken a bit of a break from electronics, and have been having great fun with Clojure.

In the distant past I spent about 19 years programming in a (fairly) functional programming language, so Closure feels very comfortable. The thing I miss most is a Romilly-friendly multi-dimensional array library.

There are high-performance numeric libraries in Java, and some of them have been wrapped in Clojure, but their authors don't think the way I do.What I want is a library that works like the array engines in APL or J wrapped in a Clojure API.

For now I value expressiveness more than performance, and it looks as if I can implement a useful subset of APL primitives in a couple of days.

When performance becomes an issue there are two tempting possibilities. Closure allows you to use native Java arrays performantly; and a Clojure wrapper for OpenGL looks as if it may allow easy access to a workstation's GPU. This might permit array operations that run faster than they would in Java or C++.

This blog has moved

2010-04-27T10:53:00.001+01:00

This blog is now located at http://romillycocking.blogspot.com/.
You will be automatically redirected in 30 seconds or you may click here.

For feed subscribers, please update your feed subscriptions to
http://romillycocking.blogspot.com/feeds/posts/default.

Host your own Windows7 launch party?

2009-09-28T11:19:00.002+01:00

At a loose end? Why not host your own Windows7 launch party? Here's how.

Thanks to Neil Hume of FT Alphaville for this.

What shall we do with Gordon Brown?

2009-07-05T11:17:00.002+01:00

Moonwalker Buzz Aldrin says a global leader should commit to sending a man to Mars.

Go for it, President Obama. And may I suggest Gordon Brown as the lucky spacefarer?

IBM says world will drown in data next year

2009-06-15T18:05:00.002+01:00

According to IBM, "Experts predict that by 2010, the amount of digital information will double every 11 hours."

That's really, really scary. If the amount of data were to double every 11 hours, it would quadruple every day. Over a month the quantity of data would increase by a factor of 36,893,488,147,419,103,232 (roughly speaking). Over a year it would increase by a factor of ...

No, I'm not going to work it out. But if anyone from IBM reads this please can you buy a calculator for the marketing department.

Element 112 needs a name

2009-06-13T08:03:00.003+01:00

A German team is looking for a name for an element that's just about to join the periodic table.

Element 112 is super-heavy and very unstable. It falls apart in milliSeconds.

Gordonium?

Oh, the Coder and the Tester can be Friends, can be Friends...

2009-03-06T07:34:00.004Z

(with apologies to Oklahoma!)

Elisabeth Hendrickson just tweeted about a thought-provoking analysis by Michael Bolton of the much-vaunted Continuous Deployment at IMVU.

I'm a big fan of TDD, Automated Tests and Continuous Integration, but I'd hate to develop without help from good Testers. A couple of years ago I worked on a dream project; a very small, highly skilled team which included a couple of developers, a tester and a business analyst. We were pair-programming, and my pairs came from a pool that included some of the best agile developers in the UK. The Tester and Business Analyst were also outstanding.

Our defect rate was very low: five defects in UAT, none in production over a twelve-month period. There's no doubt that this was a team effort; the Tester and the BA both caught many defects that were caused by problems we developers hadn't though of, and therefore never tested for. Equally, the developers tested so thoroughly that the Tester and BA could focus on the hard stuff, where their skills contributed maximum value.

For reasons that need not concern us, the application initially went into production on a machine that was controlled by the developers. We could have done continuous deployment, but we didn't. We always asked our Tester to do a final check-through on our staging box; mostly he found nothing, but once or twice he found defects that would have been real embarrasments in production. Of course, that wasn't his only contribution; our acceptance tests were the product of very close collaboration by the whole team.

So it was no surprise to read to read about Michael Bolton's experience when he did some manual testing of the IMVU site. IMVU practise Continuous Deployment, and claim to rely entirely on Automated Tests. Read Bolton's 50 Deployments A Day and The Perpetual Beta and see the consequences.

EtherPad - real-time collaborative text editing

2009-03-01T08:30:00.003Z

A tweet from Eric Mack just pointed me at EtherPad.

It's a free tool for real-time collaborative text editing. As Eric says, it's simple, and it just works.

If two or more of you need to collaborate over the web in real-time on a text document, this is all you need.

Refactoring is much easier...

2009-02-28T11:38:00.013Z

Refactoring is much easier when you have clearly expressed intent.

This came up a few days ago when I was talking with Steve and Nat.

We were holding a lunchtime retrospective about the latest delivery of our TDD course.

We talked about what had worked well, what we could do differently, and how we could customise the course for audiences with differing requirements.

Nat and Steve were keen to develop an exercise that would focus on refactoring. I liked the idea but I also wanted to keep an existing exercise which brings to life the value of intentional programming.

I felt that the ability to express intent was even more fundamental than the skill of refactoring.

It's risky to refactor unless you can check whether the code still does what is wanted. I don't think you can do that easily unless the tests and code express intention.

Nat reminded me that what you express, and how you express it, is different in code and tests. Good tests remind you of intent when they fail. This makes it easier to diagnose the cause of failure.

Refactoring is easiest with clean code, and clean code expresses clear intent. (Of course, as Steve pointed out, everything is easier if you have clean code).

You might refactor because the intent is not clear, but there are other reasons to do so. For example, you might refactor in order to make the code easier to change.

"The ability to pay back debt...depends on you writing code that is clean enough to be able to refactor as you come to understand your problem"

Ward Cunningham - Debt

Conclusion: we will develop a module on refactoring, but we'll keep our emphasis on writing clean code and tests that clearly express intent.

GetInLine version 2

2009-02-27T14:07:00.004Z

I'm currently working on GetInLine 2 - a complete rewrite of GetInLine, a record-processing DSL I wrote a couple of years ago.

The DSL is not yet as fluent as I would like, but I'm much happier with the underlying design. I'll also be switching to an Apache license; James Strachan encouraged me to do this for version 1, but I never got round to it.

I'm following the development approach that Steve Freeman and Nat Pryce advocate in their book (and which the three of us expound in our TDD course).

I started with a failing end-to-end test. Then I used mocks and unit tests to help me to pull the required interfaces into existence, filling out the application until the first end-to-end test passed.

The resulting code is much simpler to test and write than v1 was, and I've ended up with a highly pluggable set of components.

I'll post a URL as soon as GIL2 is ready for beta testing; if anyone wants to tale a look before then, let me know.

Opportunism knocks

2009-02-11T10:41:00.003Z

I've just angrily binned an invitation from the Professional Conractors' Guild.

I'm a PCG member, and I've been very happy with them. If you, like me, work as an IT contractor in the UK you'll find they offer an excellent and valuable range of services. I routinely recommend them to colleagues. So -what did they do to annoy me?

They invited me to their tenth birthday celebration.

For which I would have had to don a DJ and pay them £60.

Why on earth would they think I'd want to do that?

I'm glad they have been going for a while, but not glad enough to fork out £60. Even worse, I'd have to spend the evening talking to other IT contractors - and we'd all be wearing dinner jackets.

If the economy had been booming I'd just have tossed the invitation in the bin, but with things as they are the invitation was about as welcome as an invitation to play a round of golf with Fred the Shred. (I assume he plays golf, on the grounds that he must be good at something.)

So, PCG: I won't be joining in your narcissistic celebration, and please don't waste my membership fees on promoting events like this in future.

Read "Rocks into Gold" and spread the word!

2009-02-07T14:15:00.005Z

I've just read Clarke Ching's Rock's into Gold on SlideShare. It's a delightful short story which explains how incremental delivery can improve cash flow, saving jobs in the process.

I first met Clarke at XP Day a couple of years ago. Several friends told me how much they liked his Introduction to Agile Development, so I went along. I'm glad I did.

Clarke's presentation was excellent - clear, simple, business focussed, and very compelling. So's Rocks Into Gold. I've started to use it as a resource for prospective agile adopters; I may even save a few jobs!

Read it and spread the word.

How do new ideas spread?

2008-07-16T09:38:00.002+01:00

I've recently been studying how we change the way that we develop software. This links in with the broader question of how innovations spread. The best-known book on this is Geoffrey Moore's Crossing the Chasm, which explores the obstacles faced by high-tech innovation.

Crossing the Chasm is an influential and readable book, generous in its acknowledgements and full of good stories, but it has no bibliography. For sixteen years I've been looking for the book that introduced the technology life cycle. Moore refers to it but never names it.

I think I've found it in Diffusion of Innovations, by Everett M. Rogers. It's now in its fifth edition. Its examples are extensive and compelling, and it has an extensive bibliography. Best of all, it includes a checklist of the key attributes of innovations. Rogers calls them

Relative advantage
Compatibility
Complexity
Trialability and
Observability

These perceptions largely determine the rate at which we adopt an innovation, so anyone in the business of change needs to study them carefully.

Diffusion of Innovations is a scholarly work, and needs more time to read than Crossing the Chasm. For me that time has been well-spent.

SPA2009 open for session proposals

2008-07-16T07:47:00.002+01:00

SPA2009 is now accepting session proposals, and some have already arrived.

If you would like to present at this unique conference you'll find the call for proposals on the SPA2009 website.

If you want to attend the conference, consider proposing a session. As a session leader, you will

have fun
polish your communication skills
improve your profile in the industry
and get a substantial discount!

You'll be helped by a supportive shepherding process. SPA shepherds are experienced presenters and many of them have coming to the conference for years. The shepherds can help you develop a successful session that takes full advantage of SPA's interactive format.

The deadline for submissions is 6pm on Monday 15th September 2008.

miniSPA2008 and cfp for Spa2009

2008-07-12T08:11:00.004+01:00

I spent yesterday at miniSPA2008. MiniSPA is the free one-day taster for the annual Software Practice Advancement conference. Lots of new faces yesterday, as well as old friends. It looks as if we'll get lots of new session leaders for next year's conference, which is great.

Session leaders get a significant discount; for the next conference, one of them will pay nothing! Details on the SPA2009 "lead a session" page.

If you're new to leading conference sessions, a supportive shepherding process will help you to hone your skills. Quite a few big names on the conference circuit cut their teeth at SPA.

Whether you're an expert in the making or a hardened veteran, make sure you get your proposal in soon. Submissions open on Monday, and the deadline for submissions is 6pm on 15th September 2008.

TiddlyWiki to MindMap converter

2008-04-28T09:50:00.012+01:00

Knowledge is my stock in trade, and I use TiddlyWiki and Mind Mapping a lot.

Like all Wikis, TiddlyWiki is a great tool for capturing ideas and facts in text and linking them to other relevant topics. It's not so good at showing you the big picture.

Mind Maps (tm) are a great way of presenting the big picture, but programs like FreeMind and MindManager (tm) only show the notes for a single branch at a time.

What if you think of each tool as giving a different view on the same data? Then you could get the best of both worlds.

There's already a hypergraph extension for TiddlyWiki which allows you to see a visual representation of wiki contents but I prefer a more conventional Mind Map that I can edit.

I'm currently experimenting with a simple Python script which converts a TiddlyWiki into a FreeMind Mind Map. The converter works well for small wikis and I'm currently researching how best to structure larger maps.

The converter isn't ready for prime time yet, and documentation is minimal, but it's evolving fast. If you're interested please drop me a note ( at romilly dot cocking at gmail dot com) and I'll send you a copy.

History meme

2008-04-25T16:26:00.003+01:00

Mom, Tim made me do it...

romilly@samba:~$ history | awk
'{a[$2]++}END{for(i in a){print a[i] " " i}}'
| sort -rn | head
106 ls
96 sudo
52 cd
22 nano
16 man
16 find
16 exit
13 chmod
12 cat
10 scp

and, on the Virtual server that hosts samba,

romilly@pc031:~$  history | awk
'{a[$2]++}END{for(i in a){print a[i] " " i}}'
 | sort -rn | head
111 ls
78 cd
76 sudo
32 exit
23 top
21 ping
18 df
13 du
11 scp
11 man

Layers considered harmful: design to interfaces

2008-04-15T11:52:00.003+01:00

Mark Dalgarno recently published an excellent summary of a session presented by Steve Love at ACCU.

The talk offers a critique of the traditional doctrine that architectures should be layered. Instead, Love proposes a style based on designing to interfaces. That's music to my ears: it's exactly the approach we take in our new TDD with jMock2 course.

Love's presentation is available as a pdf. The first half consists of visuals alone; the second half combines the same visuals with notes.

Testing Hibernate-based Persistent objects without Spring

2008-04-06T11:30:00.005+01:00

Nat Pryce just pointed me at an excellent article on Testing Persistent Objects Without Spring by James Richardson.

It shows both the integration test and the code under test.

Lots of good ideas/good practice.

Testing RIAs with WebDriver

2008-03-28T14:19:00.006Z

If you need to test Rich Internet Applications from Java, take a look at Webdriver.

Things I like about it:

It uses real browsers to run the tests, quirks and all
It's got a literate testing interface based on Hamcrest and inspired by LiFT.

Here's a very simple example.It's doing simple tests on a static html site:

public class HomePageTest extends HamcrestWebDriverTestCase {
   
   @Override
   protected WebDriver createDriver() {
      return new HtmlUnitDriver();
   }
   
   public void testHomePageHasTitleAndHeading()
         throws Exception {
      goTo("http://test.intranet");
      assertPresenceOf(title()
        .with(text(startsWith("Agile"))));
      assertPresenceOf(title()
        .with(text(containsString("Cocking and Co."))));
      assertPresenceOf(heading(1)
        .with(text(equalTo("Agile Application Development"))));
   }
}

This won't quite work out of the box, as there is no heading finder, but adding one is easy:

public class HeadingFinder extends HtmlTagFinder {
  private final int level;

  public HeadingFinder(int level) {
     this.level = level;
  }

  @Override
  protected String tagDescription() {
     return "heading level "+level;
  }

  @Override
  protected String tagName() {
     return "h"+level;
  }

  public static HeadingFinder heading(int level) {
     return new HeadingFinder(level);
  }
}

It's fairly new, but there are now pre-built binaries available.

Category Theory/Haskell learning group in London

2008-03-25T10:00:00.004Z

Peter Marks organised a BOF (Birds of a feather) session on Category Theory at Spa2008. To my surprise about twelve of us turned up. We discovered a shared intuition (bizarre though it seems) that Category Theory will somehow shape the next big thing in application development. Less surprisingly, we also discovered a common enthusiasm for Haskell.

We want to build on this. Most of us are based in London, or can reach it easily. We plan to meet regularly (about once a month) to explore these ideas and to help each other to learn.

I'll post more about this as soon as we have a date and location for our first meeting.

Spa2008 retrospectve

2008-03-25T09:56:00.005Z

I'm back from Spa2008, and my head is buzzing.

I felt this was the best Spa conference in recent years. As chairman I'm biased, but I'm not responsible for the quality. That's down to the programme chairs (Eoin Woods and Ivan Moore) and to Andy Moorley who organised things perfectly.

I went to (and really enjoyed):

Mirror, Mirror on the wall - why me? Chris Cooper-Bland and Portia Tung
Thou Shalt Integrate by the Sweat of Thy Brow... Robert James and Eoin Woods
The Need for Speed Paul Dyson and John Nolan
Awesome Acceptance Testing Dan North and Joe Walnes
Spanish Guitar Recital David Harvey
Is Software Practice Advancing? John Daniels and friends
A few things I learned in 50 years of programming L Peter Deutsch
Big Ball of Money, Big Ball of Mud: Economics and Legacy Code Michael Feathers
Closing the Knowing Doing Gap Allan Kelly and Lise Hvatum

Now it's time to start thinking about Spa2009!

Fewer, better people

2008-02-26T11:36:00.004Z

I've just read Martin Fowler's post about the Cheaper Talent Hypothesis. It's an issue that most of us have encountered.

If you're in IT, feel underpaid, and think that you're above average (as > 50% of us do), you'll have wondered why you aren't rewarded in keeping with your worth.

I suspect the main reason that good developers are undervalued is that most people think that software is easy to write. Since many IT projects fail, the implication is that we can't very good at what we do.

The conclusion is false because the premise is false. Writing good software is one of the hardest things that people do. I'd love to find some way to allow non-developers to find that out for themselves.

IBM used to run introductory programming courses which included a session called keep the alien alive. The instructor played a dumb alien who obeyed spoken instructions literally; the class had to tell him/her how to drink a glass of water. It was surprisingly hard to do, and the instructor usually got soaked. But it really brought home the fact that programming is hard.

How could we update this for the 21st century?

Jython lives!

2007-10-16T12:23:00.000+01:00

It's good to see that Jython is under active development again.

I use cPython a lot. I'd pretty much stopped using Jython because it does not currently support language features that I find essential - decimals, generators and list comprehensions, for example.

The project seemed to be dormant, but Jython 2.2 has just been released, and it looks like Jython 2.5 is on the way, with all the critical features that I need. Excellent news!

Download all pdfs in a web page

2007-09-24T18:32:00.002+01:00

A while ago I found an out-of-print book with a web-page that listed a series of downloadable pdfs.

Since there were 17 chapters and several other sections, I didn't want to right-click my way through the list so I wrote a quick Python script to do it for me.

This must be a Python idiom. I've had to re-invent it so often, I thought I'd blog it so that I -and you - will never have to create it from scratch again.

Here it is.

import urllib, re, sys
# download each pdf linked to in a web-page.
# assumes that the urls are all relative,
# which is usually the case
pdflink = re.compile(r'<a href="(.*\.pdf)">')
baseURL = sys.argv[1]
page = urllib.urlopen(baseURL+"contents.html")

for line in page.readlines():
  match = pdflink.search(line)
  if match:
      filename = match.group(1)
      print "downloading %s" % filename
      urllib.urlretrieve(baseURL+filename, filename)

Google soft-launches social bookmarking tool

2007-09-24T15:00:00.000+01:00

Is Google shared stuff the latest hot thing? It's a new social bookmarking tool.

It's available now, although you won't yet find it in the list of Google services and tools, nor on the Google labs page.

As well as sharing with your friends, the new bookmarklet allows you to share your links via several third-party sites, including Facebook, digg, reddit and del.icio.us.

In itself, it wouldn't be worldshaking; but

It's from Google
There are rumors of a soon-to-be-released Google api for social networking.

Count Blog subscriptions with this Python script

2007-09-22T07:36:00.000+01:00

If you blog, you're probably keen to know the size of your readership. Many of your subscribers will use an RSS aggregation service, and these days there are lots of feed aggregators. How do you count the subscribers?

This short Python script does just that.

If you can access the httpd logs of the server that hosts your blog, you'll find that many aggregators send you a subscriber count when they check for new blog entries.

A log entry for a typical aggregator request looks like this:

65.214.44.29 - - [21/Sep/2007:04:12:34 +0100]
   "GET /atom.xml HTTP/1.1" 200 0 "-"
   "Bloglines/3.1 (http://www.bloglines.com; 10 subscribers)"

The Python script below filters out lines that contain the string "subscribers", extracts the Ip address and subscriber count, and calculates the total number of subscriptions.

The script is very simple, but there are a couple of wrinkles.

A singe log file can include multiple requests from a given aggregator. The script puts counts into a dictionary keyed by the aggregator's ip address, so the total count includes the latest value for each aggregator.

In theory, you could get a request that included the string 'subscriber' from some other source. (For example, someone might visit a URL for newsletter subscribers.)
The script has a fudge to treat such requests as if they came from a dummy aggregator with no subscribers.

Here's the script:

# count blog subscribers from httpd log file 
import re

ip =r"^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
anything = ".*"
count = r"\s(\d+)\s"
subscriber = "subscriber"
subscription_entry = re.compile(ip+anything+count+subscriber)

def ipc(line):
    match = subscription_entry.search(line)
    if match:
        return match.group(1,2)
    else:
        # just in case the line contains 'subscriber'
        # but does not match the regular expression
        null_entry = ("0.0.0.0", 0)
        return null_entry

def total_counts(ips_and_counts):
    # there may be multiple log entries for a given aggregator
    # so we use a dictionary and just keep the last value 
    #  which is the latest one for each aggregator
    dict = {}
    for (ip, count) in ips_and_counts:
        dict[ip] = int(count)
    return sum(dict.values())

def countSubscribersInLogFile(filename):
    file = open(filename)
    counts = [ipc(line) for line in file if subscriber in line]
    total = total_counts(counts)
    file.close()
    return total

if __name__ == "__main__":
    import os, sys
    filename = sys.argv[1]
    details = (countSubscribersInLogFile(filename), filename)
    print "%d subscriptions in %s" % details

Builders

2007-09-10T11:21:00.000+01:00

Nat Price has posted an excellent overview of Test Data Builders. (Don't be put off by the image!)

Sadly, he doesn't (yet) say much about the problem of unit testing builders, though he does point out that Builder classes may well find their way into production code. Nat, that's a hint...

XPDay7 is shaping up; now on to SPA2008!

2007-09-10T09:39:00.001+01:00

The programme committee for XPDay7 met last week to review session proposals. We'd received an outstanding collection of submissions, so there was no problem in filling the program with quality sessions. The hard part, as usual, was to decide which ones we would not include.

The meeting went very smoothly. Angela Martin and Ivan Moore are joint program chairs this year. They put a lot of hard work into preparation, and it really paid off.

If you've submitted a proposal for XPDay7, expect to hear from Angela or Ivan soon.

I'm now bending my mind to SPA2008. Ivan Moore and Eoin Woods are joint program chairs. As conference chair, I'm as keen as they are to see another outstanding programme.

If you submitted a proposal for XPDay7, consider submitting a session for SPA. We like a mix of new and proven sessions, so acceptance for XPay7 shouldn't stop you from submitting a similar proposal for SPA. Not should rejection - there were excellent submissions for which we just had no room, and they might well fit in the (longer) SPA programme.

The SPA deadline is Monday 17th September, but you don't need to submit finished material - just a proposal.

Backing up vital data with Amazon S3

2007-09-09T15:16:00.000+01:00

I finally got around to trying Amazon S3 for data backup.

Just in case you've missed out on S3, it's a commercial network-based data storage service offered by Amazon. They claim to use the same technology for S3 that lies behind the Amazon stores.

S3 comes with no service level guarantees and there have been some reports of occasional unavailability and/or slow transfer. I'm not too worried; the things that are critical to me are data security and cost.

I'm using the JetS3t java(tm) tool kit to manage the interaction with the S3 service. The cockpit application gives a GUI view of what's in your S3 buckets; the synchronize application gives a simple command line interface that allows you to keep data on S3 in step with data on your machine(s), and retrieve that data when necessary.

You need to be a little careful, as synchronization can delete data as well as add or update it. To mitigate this, there is a --noaction option which allows you to see what would happen to your data without actually changing anything.

Minor snags

I've hit a couple of minor snags so far; neither are caused by the S3 service itself, but you need to be aware of them.

The first arose when I tried to back up a large amount of data. I'm keen to keep a copy of my cvs repository off-site; it's well backed-up, but all the recent backups are in my home office. If we had a fire or a burglary I'd be in real trouble.

I realised soon after the upload started that it would take a while. The repository is about 2.7Gb and although I have fast broadband, my upstream speed is only 256 kb/s. The upload took 25 hours!

Smart synchronization

Synchronize is smart enough not to upload files if they are unchanged. I figured that the next upload would take a few minutes at most. The second snag arose when I tried to check that.

Part way through the synchronize application bombed with an out of memory error. I've now modified the script to give it 512M of heap space, and it runs just fine. And now a burglary would be inconvenient but not disastrous.

Measuring cultural differences

2007-09-07T13:20:00.000+01:00

James Manktelow has just published an interesting article about measuring and understanding cultural differences. It's based on Geert Hofstede's work on cultural dimensions.

Hofstede has identified five key dimensions which can be used to classify and understand local cultures. His analysis is based on a data base collected by IBM in the 60's and 70's surveying the values of employees in more than 70 countries.

Hofstede's Cultural Dimensions

Hofstede analyses culture in these dimensions:

Power Distance
Individualism
Masculinity
Uncertainty Avoidance and
Long-Term Orientation

It looks as if these measures would form a useful framework for analysing and understanding the culture of the organisations we work for.

Introducing Agile - Gerry Weinberg's approach

2007-09-07T10:57:00.000+01:00

David Petersen just sent me a link to a great interview: "5Qs on Agile with Gerald M. Weinberg".

The interview includes a neat way of getting people to want to adopt Agile techniques, rather than trying to force them.

Weinbeg is one of my favourite authors, but I wasn't subscribed to his blogs. I am now.

His writing blog has another interesting post about self-publishing.

Virtual Appliances: only run them if you trust them!

2007-08-31T14:21:00.000+01:00

I forgot to add an important warning to yesterday's post about Virtual Applicances!

When you start up a virtual appliance it will probably try to get an IP address on your private network via DHCP. If the Virtual Appliance contains malicious code, it could then do all kinds of nasty things.

The bottom line: as with any software that you install behind your firewall, don't run a virtual appliance unless you trust the source you got it from.

The Virtual Appliance: a simple way to install complex software

2007-08-30T16:17:00.000+01:00

This morning I saw an email asking for suggestions for a cheap or low-cost web content management system. There was one other important requirement - the software would be installed, managed and used by "semi-technical people".

A couple of people suggested drupal. I've heard good things about drupal, so I took a quick look at the website. Here's a sample of what hits you on the installation page:


Drupal requires a web server, PHP4 (4.3.3 or greater) or PHP5 (http://www.php.net/) and either MySQL (http://www.mysql.com/) or PostgreSQL (http://www.postgresql.org/).

That's hardly ideal for semi-technical people.

Once drupal is installed, however, it seems easy to configure and use. What my friend's users need is a simple way to install the application in the first place.

The best approach I can think of is to install a Virtual Appliance.

A Virtual Appliance is a virtual machine image which you can run using VMware, Xen or Parallels. It contains a complete application, pre-configured and ready to run.

Using a Virtual Appliance with the free VMware player takes just 5 steps:

Download VMware Player.
Install it. (on Windows, that's a one-click operation).
Download and unzip a suitable Virtual Appliance.
Start the Player and
Start the Appliance.

Now your chosen application is up and ready for use.

I tried out the drupal appliance from JumpBox. It took me about 5 minutes to install and start using it. (I had a bit of a head-start because I already had VMware player installed).

If you know someone who wants user-friendly software but who lacks the technical expertise to install it, look and see if you can find a virtual appliance to suit.

More and more software (open source and commercial) is being offered this way, and it's also a great way to distribute software that you've written or configured.

Automated end-to-end testing made easier with VMWare Server

2007-08-27T17:52:00.000+01:00

Automated testing is one of the most important practices in Agile Development. I like to have a hierarchy of automated tests that cover the whole system as deployed and all of its component parts, right down to unit tests that tell me whether each class is fulfilling its contract.

Starting with end-to-end tests

There's a lot published about unit testing and acceptance tests, but much less about end-to-end testing. That's disappointing, given that end-to-end tests are arguably the first thing you should work on when developing a new application.

The need for a clean deployment environment

End-to-end testing gives you a chance to check out your deployment process and verify that the deployed application works in the target environment. Sometimes, though, a badly planned testing process can lull you into a false sense of security; most of us have encountered applications that work well in their test environment but fail when deployed to a clean machine.

It usually takes too much time and effort to create a fresh testing environment from scratch every time you want to do an end-to-end test. There is a simple fast alternative using VMWare Server.

VMware server to the rescue

VMware server is a free (as in beer) version of VMware's virtualisation software. I run it on an Ubuntu server, and use it to create a clean environment for applications that I'm developing. It has a snapshot capability that allows me to capture a clean "before deployment" image, and restore that in a few seconds prior to running a test. Best of all, there's an external scripting interface which allows you to automate the process.

You can download VMware Server for free (after registration). I also use their commercial VMware workstation product; the two play well together, as you'd expect, but you can do everything you need for end-to-end testing with the free product.

JavaScript - are you serious?

2007-07-19T07:15:00.001+01:00

One of the sessions at Monday's miniSPA conference was Serious JavaScript led by Peter Marks and David Harvey.

I missed the original session at SPA2007, and I was very glad to get a chance to see it second time round. I found it fascinating and alarming.

It was fascinating because the presenters brought out some of JavaScript's real strengths as a development language. One example: JavaScript supports functions as first class objects, which allows smart developers to do all kinds of clever and useful things.

The session was alarming because it showed how seriously JavaScript is marred by arbitrary and counter-intuitive semantics.

Time after time the session leaders asked the audience to predict the results of simple JavaScript expressions. Time after time we predicted incorrectly (and sometimes the presenters did too).

The language seems to have been designed around the principle of most surprise.

People are suggesting that JavaScript is going to be the next big language, and it looks like a fair few people at Google are in that camp, including Steve Yegge. If so, heaven help us all. Power and unpredictability make a very dangerous combination.

miniSPA and jMock2

2007-07-18T20:40:00.000+01:00

I spent yesterday at miniSPA. It's a selection of popular presentations from last year's SPA conference; we run it to attract new participants and new session leaders.

This year's miniSPA was brilliantly organised by Ivan Moore and Andy Moorley. Ivan is programme chair for SPA2008. Andy runs the conference, organises miniSPA, and generally keeps the whole world turning as it should.

I was at miniSPA in two capacities; as next year's conference chair, and as a presenter from last year. Nat Pryce and I ran a tutorial on Test Driven Development with jMock2, which went really well. We've offered to run it again in November at XPDay7.

I'm about to analyse the feedback and first impressions are very positive. SPA is a remarkable conference. If you haven't been, take a look at the website and think about going next year. If you want to go next year, think about submitting a session proposal.

Unit testing builders

2007-07-11T07:05:00.000+01:00

Should one unit-test test Builder classes? If so, how?

A well-designed application is a composed network of simple components. Many developers use one or more builder classes to construct applications.

Builders can get quite complex.

In many cases we want a single component to be shared by several client components. It's easy to get this wrong. If we do so, we may build a network on which all dependencies appear to be satisfied, but we have actually created multiple copies of components when a single instance should be shared.

Builders often use lazy initialization, in which case they have state. This introduces additional opportunities for error.

Since it's easy to get builders wrong, we need to test them. End-to-end tests will usually tell us if we've build out application without all the necessary parts, but they don't pinpoint the problem, and they may not tell us if we've mistakenly introduced multiple objects instead of sharing a single one.

This suggests that we need to write unit tests for builders, but that's often difficult. The classes that are built will hide their implementation (as they should) so unit tests cannot easily verify that the right internal components are there inside them.

I'm coming to the conclusion that builder tests need to break encapsulation. That's normally a major code smell, but the nature of builders seems to leave no alternative.

Is measurement evil?

2006-12-02T09:22:00.000Z

A while back Dave Snowden of the Cognitive Edge (formerly Cynefin) posted a well-aimed comment about the UK Government's folly in trying to fix their outcome-based measures.

I first met Dave Snowden when he presented a keynote at XPDay5. I was bowled over by his talk, as were many of the audience.

Not so Tom Gilb, who was sitting next to me in the talk. Perhaps I shouldn't have been surprised: most of Tom's professional life has been focused on helping organisations to improve thorough quantification and measurement.

Both Dave and Tom are brilliant thinkers and I have been pondering ever since how two such clever people could hold such diametrically opposed views.

I had a delightful talk about this with Clarke Ching at XPDay6. which helped to clarify my ideas. I've tried to summarize my understanding below.

If I am trying to manage others, it's useless for me merely to set them measures. My team will game them. I'll get what I measure. It will not be what I want. (I think that's at the core of Dave Snowden's position. It's a sufficiently common view that Duncan Pierce and Jason Gorman ran a session about it and XPDay5.)
If I am a customer trying to define my requirements, it is sensible to quantify the qualities that I require of my supplier's product or service. (I think that's at the core of Tom Gilb's position).

In order for 2. to work, I must be able to:

quantify the qualities that I require
agree scales of measure for those qualities
agree realistic and achievable targets based on those scales of measure
monitor progress towards those targets (weekly if possible)
and negotiate changes to both the measures and the targets as appropriate.

This process relies on certain norms of behaviour by both parties. If either party seeks to cheat the other the process will break down. I think that this leads to the final precondition for Gilb's approach to work:

Either party must be able to terminate the relationship at any time without direct financial penalty.

Dave Snowden can produce plenty of examples where these conditions were not met and outcome-based measures were disastrous.

Tom Gilb can produce plenty of examples where these conditions were met and the results were excellent.

Conclusions:

both Dave Snowden and Tom Gilb have something valuable to tell us!
when we use measures to manage a supplier relationship, we must make sure that either party must be able to terminate the relationship at any time without direct financial penalty.

XPDay6 - better than ever...

2006-11-28T16:17:00.000Z

XPDay6 is nearly over, and it's been great.

One of my favorite sessions was Clarke Ching's Embracing Change: An Introduction to Agile Software Development. Clarke has given similar presentations at XPDay for several years. They've all been good and they keep getting better.

Another was a goldfish bowl session Facilitated by Nat Pryce and Jonathan Clarke asking Why Is Simple So Difficult?

It's hard to define simplicity; it's much easier to define or detect complexity (and then avoid it).

Lots of good ideas in the session; I liked Joe Walnes' comment that you shouldn't do the simplest thing - you should have the simplest thing (but it may be very difficult creating that simple thing).

I introduced Roy Sykes' 3am rule. This defines what I call simple code. It's the sort of code you'd hope to find if you have been called out of bed at 3am to fix a critical defect in production code.

My favorites session today was on LiFT - a framework for literate testing. It can be used as an alternative to FIT, but it can also be used to implement readable FIT fixtures. I'm hoping to give it a spin on my current project.

I really enjoyed the pub session afterwards. (Thanks, Google). Quote of the day - to Angela Martin, as she was taking photos of us all:

"I see you're taking a picture of me while I'm sober. I hope you have a fast film."

Ah well - off to the Old Bank Of England in Fleet Street for the post-conference Extreme Tuesday with more free drinks - thanks, Kizoom!

Factory methods can improve readability in embedded DSLs

2006-11-20T13:34:00.000Z

Nat Pryce has just got rid of lots of explicit object creation from the interface to GetInLine; the embedded Domain Specific Language now reads much better.

No new()s is goood news, in fact!

Facts and Fallacies of Software Engineering

2006-11-15T07:56:00.000Z

I've just read Facts and Fallacies of Software Engineering and I love it.

My mate Bryan Dollery lent it to me after a chat in the pub about developer productivity. (Don't look so surprised. I do go the pub occasionally).

The book confirms many long-held opinions, and offers a lot of hard evidence ato support them. In particular, it spells out why a few skilled (and therefore expensive) developers can deliver business value much faster than a larger, less skilled team.

I've started to write a couple of short articles on themes raised by the book. Watch this space.

And thanks, Bryan, for an excellent read.

Ruby and Amazon Web Services

2006-11-14T17:05:00.000Z

I've been ill at home for the last few days, but today I felt well enough to play a little. I built a really useful little command line application in Ruby. It's already saved the time it took to write it.

It expects to find an ISBN in the clipboard. It checks that the ISBN is valid, uses the Amazon Web Service interface to get book details from an ISBN, and pastes the details to the clipboard, complete with an Amazon affiliate link. (If you buy the book after clicking on the link, I get a little money).

A variant does the same thing, but copies the book details in a format ready to paste into TiddlyWiki which I use as an personal blog/wiki.

I found out how to parse XML in Ruby by reading Enterprise Integration with Ruby. (You can guess how I created that link, can't you?)

Here's the Ruby code. To use it you will have to sign up for Amazon web services (if you haven't done so) and insert your access key into the code.

Ruby rocks.

GetInLine at SourceForge

2006-11-05T18:51:00.000Z

The GetInLine file processing DSL is now hosted at SourceForge. Code is in cvs and I've released an initial version (0.0.9).

Here's a sample test case that will give you a feeling for how to use it.

Brain Quadrants updated

2006-11-03T07:51:00.000Z

I'm still fascinated by Ned Herrmann's Brain Quadrant Theory.

My colleague Chris Pollard points out that my earlier blog post never listed the four core types of thinking in Herrman's model.

Here they are:

logician
organizer
communicator and
visionary.

For a good introduction, read Herrman's overview (pdf).

DSL for file processing - GetInLine

2006-10-23T13:32:00.000+01:00

I've been playing with an embedded Domain-specific language for file processing. It grew out of a requirement from my current project, but I've already found a use for it in a couple of personal tasks.

Tha language is growing and changing fast, but here's a code snippet that may give you a feel for what it can do already:



public void testIncludingFiltersOutRecordsThatDoNotMatchACondition() throws Exception {
 new Source().
 fromFile("docs/sample-data/flights3.txt").
 upToTheEnd().
 counting(allRecords).
 withoutBlankLines().
 counting(nonBlankRecords).
 allowing(Processing.rule().
  accepting(1).
  including(terminalRecordPattern).
  populatingList(
  terminals,
  terminalFactory,
  regexParser),
   Processing.rule().
  including(detailRecordPattern).
  populatingList(
   flights,
   flightFactory,
   commaDelimitedParser))
  .read();
  
 assertEquals(1, terminals.size());
 assertEquals(4, flights.size());
 assertEquals(6, allRecords.getCount());
 assertEquals(5, nonBlankRecords.getCount());
 checkEntry(0, "BA123", "Glasgow", "09:45");
 checkEntry(2, "BM435", "Bradford", "10:40");
}

which will read this file and create objects from it:


Terminal 1

BA123,Glasgow,09:45
IB236,Barcelona,10:20
BM435,Bradford,10:40
KL326,Amsterdam (Schipol),11:00

Callisto appears on schedule

2006-07-01T11:29:00.000+01:00

Yesterday the Eclipse project released Callisto. Callisto is a simultaneous release of 10 related Eclipse projects, including Eclipse 3.2.

One of the projects is GMF, the Eclipse Graphical Modelling Framework. GMF is a very high-level tool for building Graphical Editors for Domain-Specific Languages (DSLs). GMF is what Martin Fowler calls a Language Workbench. I'll be reporting on my experiences with the latest release of GMF over the next few days.

I've also been following two similar projects, one from JetBrains and one from Microsoft.

JetBrains have just announced an Early Access Programme for their Meta Programming System. I've not yet started to play with it, but I will. Given the outstanding usability of IntelliJ Idea, this is likely be an outstanding product.

Steve Cook and Alan Cameron-Wills are busy building DSL tools at Microscoft. As you'd expect, they are based on VS2005, and tie in with Microsoft's Software Factory strategy. I'm much less productive when using VS2005 than I am using Eclipse or Idea, but Microsoft initiatives can't be ignored and the authors are world-class experts.

Back to Callisto. The team at Eclipse turned the Callisto release into a real cliff-hanger. The relase date had been announced well in advance, and for several days the Callisto web-page showed a ticking countdown.

Yesterday I watched the release clock tick down to zero, to be replaced by a banner reading "You're too late!". This was (fairly) quickly replaced by a banner saying Callisto was coming soon; then an additonal "no really, we promise"; then a banner saying "packing up the bits"; "distributiong to mirrors"; and finally, late yesterday evening UK time, the download was released.

Someone at Eclipse has a fine sense of the dramatic!

Brain Quadrants and Thinking Styles

2005-11-11T14:04:00.000Z

Have you wondered why some of the people you work with just don't seem to get things that are blindingly obvious to you?

One possible explanation is that their preferred style of thinking is diametrically opposed to yours.

The Whole Brain Business Book by Ned Herrmann suggests that there are four core thinkng styles based on four distinct parts of the brain.

You've probably read about Roger Sperry's Nobel prize winning experiments which showed that the left and right sides of the brain have distinct and complementary functions.

Herrmann takes Sperry's idea further by also distinguishing between brain functions located in the cortex and and those based in the limbic system. The cortex is a relatively new part of the brain in evolutionary terms; the limbic system evolved much earlier.

Herrmann classifies brain functions within a four part (quadrant) model depending on whether they are located on the left or right, and in the cortext or limbic system.

According to Herrmann, we have personal preferences for one or more quadrants. These preferences have profound effects on how we think, learn and interact with others.

Our preferences can be modified by practice, and the book suggests activities that you can use to develop the functions of each quadrant.

The book includes a simple guide that helps you to establish your own preferences, and explores the consequences of different thinking styles.

I found something of value in each chapter but I was most excited by Whole Brain Products and Whole Brain Training and Development.

If you're interested in how you think, learn or work with others, this book offers a fascinating and thought-provoking model for you to explore.

A neat improvement to JUnit testing

2005-09-03T08:51:00.000+01:00

Joe Walnes has posted a neat way to simplify and improve tests with JUnit. His technique borrows classes from JMock, but doesn't require you to use Mock Objects.

SableCC v3 and Domain Specific Languages

2005-09-03T07:46:00.000+01:00

If you're trying to build a external DSL, you'll need to write a compiler for it. If the language is non-trivial, you will probably want to use a parser generator. One of the best is SableCC, which deserves to be better known.

Nat Pryce (of JMock and OO-Matron fame) has written an excellent introduction to SableCC version 3.

I've been using v2 of SableCC for a couple of years to compile the DSL for the SmartRec project.

At the time I started work on SmartRec, v3 of SableCC was still unstable and the documentation was minimal. I decided to stick with V2, which was recommended for production applications.

I got off to a very quick start. Sadly, as the DSL got more sophisiticated my compiler got more brittle. I've been looking for a good solution for a while. Nat's article has given me one. Thanks, Nat!

XP version 2, Mocking and more at the Agile Academy

2005-09-01T17:00:00.000+01:00

I forgot to this before my holidays, so its kinda late in the day - but there is a really good series of courses on in London next week.

The Agile Academy Summer School runs from Monday 5th to Friday 9th September.

Agenda

Day 1: Introduction to programming with XP version 2

Day 2: Putting XP into practice with RoboCode

Day 3: Advanced Test-First with Mock Objects

Day 4: Acceptance Testing with Fit and Fit Library

Day 5: Automation of Builds and Deployment

They may still have some places free (but at this notice you may have to bring your own sandwiches).

The presenters are leading lights in the AgileCommunity and the courses will be very hands-on. Strongly recommended.

Apologies for posting this so late!

Better ways to find and satisfy requirements.

2005-09-01T16:28:00.000+01:00

If you're interested in better ways to build better systems, you should take a look at the courses that Tom Gilb and Kai Gilb will be teaching in London this October.

Titles include Requirement Specification (two day), Evolutionary Project Management (two day) and Agile Inspection (one day).

Tom's new book on Competitive Engineering has had glowing reviews (not just from me :-) and the courses will give you a chance to learn how to apply key techniques directly from the author.

You can find out more at the testing solutions group website.

Saxon rocks! My XSLT transformations are now 20 times faster.

2005-09-01T08:16:00.000+01:00

I'm currently developing a tailored course for one of my major clients. I use an XSLT pipleline to convert a huge Mind Map into pdf and html student notes .

As the course takes shape, the Mind Map has been getting larger and the conversion has been taking longer. Yesterday evening the publishing task ran for four and a half minutes - too slow for an agile developer like me!

Today I switched from XALAN to Saxonica's Saxon 8. The publishing task now takes 43 seconds, of which more than half is spent outside XSLT. The XSLT processing now runs 20 times faster!

I'm opening a lot of output documents in the slowest stylesheet, which may be the problem. You may not get the same speed-up in your applications. I'll do some more timings and report them here.

Saxon is developed by Michael Kay, the author of the XSLT 2.0 Programmer's reference. Michael has also played a key role in the development of the XSLT 1.0 and 2.0 specifications. Saxon comes in two versions; I'm using the free open source variety, but there is also a commercial schema aware version.

I also use the XSLT 2.0 engine/debugger in Altova's XML Spy. It's not open-source, but there is a free home edition and you can download the Altova XSLT engine separately (also free).

Saxon was the first widely available implementation of XSLT 2.0. It's functional and fast. If you're doing heavy-duty XSLT transformation, or need an open-source XSLT 2.0 engine, I strongly recommend it.

Topic Maps and .Net

2005-08-24T13:21:00.000+01:00

Take a look at An Introduction to Topic Maps on msdn. It's written by Kal Ahmed and Graham Moore of Networked Planet. It's more informative and more up-to-date than the introduction on my website :-(

Kal Ahmed has been heavily involved in TM4j, an open source implementation of the Topic Map API. Graham and he have now created a (commercial) TMCore topic map engine for .NET

I've been playing with Topic Maps for a while now. Topic Mapping has incredible potential as a technique for knowledge creation and dissemination, but uptake has been slow. Let's hope that interest from the Microsoft developer community will be the trigger for liftoff.

MindJet, .Net and the Mac

2005-08-23T15:23:00.000+01:00

I'm not given to conspiracy theory, but I know that well-known companies sometimes have big secrets. Like me, you probably like to guess those secrets, and sometimes get them right.

You're about to read my latest wild guess.

First, though, I need to set the scene. For years I've used a program called MindManager to create Mind Maps on my computer. (Thanks to Rachel Davies for the latter link). Mind Manager is not the only Mind Mapping Software, but it's probably the most widely used. It's well integrated with Windows, but is currently only available on that platform.

A few days ago a message on the MindManager Yahoo Group mentioned that MindJet, the makers of MindManager, were recruiting developers with Mac experience.

I can only see two plausible explanations for this,and one of them is mindblowing.

The boring one is that MindJet are really worried about NovaMind, which is a Mind Mapping program that runs on the Mac and will soon be available for Windows. The folks at MindJet might be willing to create a second codebase for their product, but I doubt it. I don't think they are that dumb.

The second explanation is that they know something that we don't. That's not unlikely, as MindJet have a very close relationship with Microsoft.

When the Mac makes its move to Intel hardware, is it possible that we'll also see .Net for the Mac OS?

It's not XML - it's not YAML - it's SPaML!

2005-08-22T18:10:00.000+01:00

I'm currently working on a project that requires a lot of XSLT 2.0 transformations. Since I'm doing Test Driven Development, I am relying heavily on XMLUnit. (Bless you, Tim and Jeff).

However, I've been spending more time than I want to writing XML test documents.

So I created SPaML - the Simplest Possible Markup Language. It's very concise, and allows me to specify a complex document in a short line of text.

Here's a SPaML document (quoted, as it would be in a JUnit TestCase):

"(map file=foo(branch id=0(branch id=1(text{bar}))(branch id=2)))"

The corresponding XML is about three times as long, with escaped quotes. Horrid to read, tedious to write, and very error prone. Don't you just love it when you've got bugs in your test data!

I've written a SPaML converter in Java, and will probably port it to C#.
If anyone is interested, drop me a note: romilly(at)cocking.co.uk

Competitive Engineering hits the bookshops!

2005-08-12T17:29:00.000+01:00

Tom Gilb skyped me this morning to tell me that his latest book "Competitive Engineering" has hit the bookshops.

There's a (five star) review on Amazon.co.uk; you can view my (five star) review on Amazon.com.

New SmartArrays website

2005-08-12T17:16:00.000+01:00

SmartArrays, Inc. have just finished a remake of their website. There's a lot of new information there, including a new interactive tutorial.

eXtremely enjoyable Tuesday

2005-08-10T10:03:00.000+01:00

One of the things I'll miss most if I ever move out of London is the Extreme Tuesday Club.

It's a goup of great people who get together when they can (on Tuesdays, as you might have guessed) at the Old Bank Of England pub in Fleet Street.

If you're a regular you can be sure of meeting old friends and making new ones. If you're a first-timer you can be sure of a warm welcome. Everyone seems to share a thirst for learning and a love of sharing what they already know. The regulars include some very experienced developers.

It's a great way to find out what's going on in the Agile community. Last night Peter Brown announced XPDay5 - a two day international conference about Agile Software Development for software developers, project leaders, IT managers, testers, architects, and coaches.

I went to XPDay4 last year and loved it. This year XPDay5 will take place on Monday 28th & Tuesday 29th November, 2005 in London, U.K. Recommended!

Fun with XSLT - no, really!

2005-08-09T15:07:00.000+01:00

I've been having a lot of fun with XSLT recently.

I'm currently working on a customised course for a banking client. (No, not the French one). Over the last couple of years I have been gradually refining a course publishing framework that takes a MindMap and turns it into courseware. The framework works pretty well, but I still need to create lots of graphics.

Trouble is, I hate using drawing editors.

I'd much rather just describe the picture that I want in text and then have a piece of software draw the diagrams for me. Fortunately there are several Open Source tools that can help; one of my favorites is dot (part of the GraphViz package) . Dot reads a text file that describes a directed or undirected graph in the dot language and then lays out the graph for you. It can export a variaty of formats, inclding svg which is what my framework prefers.

The dot language is pretty intuitive, but it's a bit repetitive if you are creating lots of similar diagrams. Since I hate un-necessary repetition, I decided to create a little language for some of the most common diagrams that I needed.

Problem: how should I turn my little language into a dot file? Normally I would use a compiler compiler to do this - fun, but not quite trivial. A few days ago I stumbled on an article by Michael Kay which showed how to use regular expressions and the new unparsed-text capabilities of XSLT 2.0 to turn text files into XML. It's then very easy to get XSLT to generate a new text file in dot format.

Here's a short sample input file:

data xml "XML\ndocument"
text txt "Unparsed text\ndocument"
data out "XML, HTML or text\ndocument"
program xsl "XSL\nStylesheet"
process XSLT "XSLT 2.0\nProcessor"
input from xml to XSLT
unparsed from txt to XSLT
code from xsl to XSLT
output from XSLT to out

Here's the dot language file that's generated:

digraph process{
node[shape=Mrecord]
xml[label="XML\ndocument" shape=box]
txt[label="Unparsed text\ndocument" shape=box color=brown]
out[label="XML, HTML or text\ndocument" shape=box]
xsl[label="XSL\nStylesheet" shape=invhouse]
XSLT[label="{{XSLT|ANT task}|XSLT 2.0\nProcessor}" ]
xml -> XSLT
txt -> XSLT[color=brown]
xsl -> XSLT[color=gray]
XSLT -> out
{ rank=same; txt; xsl; }
{ rank=same; xml; out; XSLT; }
}

And here's the output as a jpeg image:

Why agile development and outsourcing don't mix

2004-11-11T07:50:00.000Z

If you're involved in IT you must be aware of two trends in the market - the move to more agile methods of application development, and the move by large enterprises to outsource more and more.

I believe that there is a fundamental conflict between these two trends, as do many of the gurus of the agile world. (For example, read Scott Ambler's article Outsourcing Examined in the April 2003 issue of Software Development Magazine ).

The most important factor in agile IT projects is the quality of communication. (See Alistair Cockburn's Agile Software Development).That includes communication within the team and communication between the team and outside stakeholders.

We can assess the quality of communication in terms of a few key measures. These measures are the same as those you would use to assess an electronic communications network. The measures are

Bandwidth
Latency
Reliability
Availabity

Of these measures, we want latency to be low and the others to be high.

Let's look at these in relation to a spectrum of application development styles.

At one end we have an XP team.

All four measures are likely to be very good. The team members are physically close, so bandwidth is high and latency is low. Reliability of communication is high, not least because because the team is likely to develop a common culture. By culture, I mean the customs, conventions, codes and values that are shared by the team. These all help to make communication faster and more reliable.

At the other end of the spectum we have an outsourced team that's located in another continent, and from a different company.

All four measures are likely to be very bad.

Bandwith is low. If we use email to communicate, we lose about 95% of the information our brains get when we interact with someone within touching distance.

Latency is high. If the person we want to interact with is available only via email and is seven time zones away, we may need to wait for 18 hours for a reply even if the person we're interacting with replies as soon as they see our email. If they are busy, or want to avoid the issue we've raised, we may have to wait much longer.

Reliablilty is low, both for technical reasons and because our cultures are different.

Availability is low. In the most extreme case, some countries pretty much close down for two weeks or more in the summer!

While this is an extreme example, any outsourced development will suffer from a significant degradation of all four measures of communication.

Conclusion: agility and outsourcing are incompatible.

Anyone who tells you otherwise is probably trying to sell you something.

Wot - no SVG?

2004-11-09T18:04:00.000Z

I've recently started using SmartDraw for some architectural diagrams in my current project. It's a nice piece of software - intuitive, simpler than Visio, and with a huge and useful library of templates. But why won't it export scalable vector graphics (svg)?

I do most of my graphics in SVG these days. Because it's an XML language I can easily transform it or generate it from other XML documents. Because SmartDraw won't import or export SVG, it's hard to merge SmartDraw diagrams with documents that I've created previously. It's a surprising limitation in what is otherwise an excellent, if commercial, program.

In my copious free time I'm planning to take a look at an alternative called Inkscape. I've heard good things about it. It uses SVG, and it's open source.

The plot thickens.

2004-11-08T07:25:00.000Z

Yesterday I followed my own advice and installed Firefox. Things went pretty smoothly, except for two interesting problems.

One became apparent immediately. Firefox had trouble importing my cookies, favourites and passwords from IE. Since IE was causing such intolerable problems after I installed XP SP2 that I decided to press on regardless.

The other became apparent this morning when I tried to access my wish list at Amazon. (After all, Christmas is coming, and so is my birthday!). The XP SP2 firewall appears to block SSL from Firefox (but not Netscape or IE). As soon as I turned the firewall offI could access my account at Amazon again.

Could SP2 be targetted at Firefox, designed to stop non-technical users from adopting an alternative to IE?

Of course not. A moment's reflection on Microsoft's unblemished track record of concern for fair competition and openness tells us that they would never stoop to such a trick.

OK - I give in..

2004-11-07T15:51:00.000Z

Over the last few weeks I've felt that the whole world has been telling me that I need to blog. Today I gave in.

Setting one up has taken longer than it should do. That's in part because Mr Gates has blessed my laptop with XP Service pack 2 and now nothing with scripts in it works with IE.

I've now created more mis-configured blogs on blogger than you can shake a fist at, and I cannot currently delete them. (I'm not worried, though, becase the blogger site tells me that "engineers are aware of the problem" and will be fixing it Real Soon Now).

I finally realised that I should have used Netscape, not IE, to set up the blog acount; as soon as I did everything worked fine.

Has Microsoft shot itself in both feet with XP SP2? It stops so many things from working that even MS diehard fans (not to mention non-technical computer users) will consider migrating to Nestcape or Firefox. All we need now is a similarly disruptive service pack for another widely-used product and Linux will positively fly onto everyone's desktop.

Memo to self - download and install Firefox today, and make it my default browser.

Second memo to self. How hard would it be to migrate all our desktop machines to Linux? Our three production servers already run Linux, and we could use VMWare to keep any non-portable Windows apps available.

There are plenty of signs that Microsoft is really concerned about the threat from Linux. If their comparison of TCO for Linux and Windows doesn't convince you, take a look at their annual filing with the SEC.

Then try the simple test below.

Question: In how many of their main markets does Microsoft indicate that they face significant competition from Open Source Software (OSS) , or software derived from OSS?

Answer: In 5 out of 7, including the client (desktop) software market.

How sad.