I have now got a relatively decent workflow for configuring both a vagrant instance and a server using chef.

I’m using a vagrant instance configured (initially) to use chef-solo. Because I will eventually be using chef with the opscode platform, I’ve acquired a couple of hacks that help me use data-bags when using vagrant.

I could of course use the vagrant instance with chef-server linking to the platform, however I find I make too many mistakes in my config in the early work-up of my config and the additional time to (knife cookbook) upload my recipes to the server and then pull them down as well makes the development loop somewhat frustrating. Chef solo provisioning allows the vagrant instance to just link across my recipes directly into the VM and run them from there.

When I’m happy with config working in vagrant (and I destroy the vagrant and re-provision) then I try the same on my rackspace (or ec2 or whatever) instance. Usually I find I need to modify one or two things (forgotten “depends” in cookbook metadata files).

I’m doing mostly quite trivial configs which require only a one server (times two if there’s a staging and a production server), so I don’t typically have the reworking which moving from a single vagrant instance to a multi-tier server architecture would require.

Here are the hacks I use:

1. I don’t like writing json unless I can avoid it, so like to write my roles in ruby. Chef-server translates them automatically (mantra from opscode training: “no ruby is run on the server”). I’ve created a rake task to convert roles from ruby to json: https://gist.github.com/834890
2. A monkeypatch for Vagrant’s chef-solo provisioner to allow it to link in your data_bags folder onto the server https://gist.github.com/867958
3. A monkeypatch for Chef-solo to allow it to manipulate data_bags very basically: https://gist.github.com/867960

However, reading at the interesting pairwithus:

When we first heard of the “Given When Then” structure for writing acceptance tests, we knew instinctively that it was a better way of writing executable examples (wrapped in the structure of an automated test to reduce ambiguity). It was presented by Dan North and Joe Walnes at XP Day 2006

Since then a number of frameworks have risen up using that structure. Many of these can be written in plain text. Despite this, there are still many teams using FitNesse and the new table parsing Test System called SLIM.

Unfortunately, the way FitNesse is used is, too often, as an .automated testing. tool rather than as a tool that documents desired software behaviours in a customer-friendly way (using automated tests). There is a substantial difference in practice, despite such subtle difference in the words.

http://www.pairwith.us/current-project.html

I notice that they are emphasizing BDD-style notation (Given/When/Then) into customer readable rather than customer writable tests. Even if “customer readability” is still a hard target to actually achieve (i.e. can I imagine any of my customers actually reading my tests?) it is a lot more plausible than getting them to write the tests. And anything that focuses on test-readability can’t be a bad thing.

Okay, so sounds like it’s time for me to focus on getting my tests more readable, and learning Given/When/Then style together with some explicitly BDD toolset or other (guess that’s cucumber for a Ruby/Rails context, right?). And I figure I better throw in Selenium or WebDriver into the mix too, cause my ajax testing strategy is just creaking at the seams…

Wouldn’t it be nice?

Some more thoughts we had:

It would be easy for people working across multiple organisations (as so many do) to aggregate their todo lists (domestic, projectA, projectB) – and also to aggregate shared todo list (ProjectA official) with any personal or confidential note (ProjectA e.g. – “speak to X about problem with day rate”)

Provided that the API was rich enough, any two webservices that supported enough of the API would be able to migrate tasks between them – in other words you could transfer your data from one to the other. That would be a compelling demonstration of trust in their own quality (and a user-attractor, I would think).

The API would be likely to need to be able to move very quickly – as the definition of what an “adequate” or even “minimal” API is likely to be quite fluid.

The first API that would be specified and implemented would be a “capabilities” API – listing a list of capability ids, together with a list for each of versions (“ticking: v1, v2″, “basic-editing: v1″, “reordering: v1″, “categorizing: v2″)

But it would be possible for clients to work in a read-only fashion with “remote to-do lists” without even having the API: two possibilities suggest themselves:

• RSS: You could specify one or more RSS 2.0 feed for any  category / project in your client (e.g. in Things a  ”project” or an “area of responsibility”). Any RSS item (with a mandatory guid and remote address) that appears in the feed is a ‘new’ (according to the guid) task to be listed in the feed. Any time the guid disappears from the feed, the task is marked as ticked, and if the guid reappears in the feed it is re-marked as unticked. Of course this means your task-list webservice provider has to support an RSS feed for your project…
• Plain old HTML: If your task-list webservice provider doesn’t support RSS, then of course the client could support a url and look for relevant links – the relevant links could be identified by some geeky means such as looking for a specific pattern of url (e.g. for a particular nokahuna that might be %r{http://nokahuna.com/projects/1324/tasks/[0-9]+} using ruby’s RE syntax)
• In both cases, the display of the task could show that it is remote and (most likely) uneditable – to make the change, you could visit the remote url.

Garry and I would like to start working on API spec together (with a reference implementation) and implement a test plugin starting with minimal RSS/PO-HTML support , but not sure how to get clients (like Things’ manufacturers Cultured Code) interested… any suggestions?

(Have no idea what I’m talking about? Permatime.com is a micro webapp aimed at making it easier to share a point in time with people across different timezones. A sample (all the bells and whitstles) permatime link might be: http://permatime.com/US/Central/2008-12-04/11:00/37signals_live?link=http://live.37signals.com. More samples on the homepage.)

The first change was to give people who click on a permatime link for the first time an “guess” of the time according the users’ current GMT offset (difference from GMT / UTC).  The reason that is a “guess” is that timezones aren’t as simple as a GMT offset. Just because you are in, say 4 hours earlier than UTC/GMT in April (like, say New York or Santiago), doesn’t predict that  you are 4 hours earlier in February (No, in fact New York will be 5 hours earlier then, and Santiago 3 hours earlier).

So we do need to encourage first time visitors to enter a timezone to be sure, but we want to give them some idea of what the permatime they have clicked on actually represents for them, without a further click. What we have now is a balance between these two design principles. Let us know what you think.

A few small additional bits of crafting: every permatime gives you relative times (3 days away, 2 hours ago, etc). Plus, for the geek in all of us: there’s now a “Unix timestamp” timezone. Yay!

Also, (it was last week actually, but as it was ‘just’ a bug fix, I never mentioned it) ironed out complications with  “Today”  on the create permatime form: it was originally being calculated as of “Today” in GMT, rather than the basis of your home (first in the list) timezone. Of course I rarely saw that… and in fact never guessed that I needed to test for it. Problems like this would be much more obvious if one of the development team was based in, say Kiritimati (GMT +14). Wonder if anyone would like to fund me to trial that for a month or two (particularly when it’s so cold here in GMT-land).

I had a busy weekend being a social innovator. Well being social at least (let’s keep the claims reasonable): I think I met more people in one weekend than I have done in the whole of last year. Enough  great ideas to give me a headache too (or was that post-sicamp pub session?).

It started off, I have to say, inauspiciously with a crowdvine site (a group/conference supporting micro-social site), which, for me, didn’t really work. Why not? Maybe because of some specific social dynamics – some of the people knew each other, others didn’t. Plus there wasn’t a clear purpose to why we were using the crowdvine… yes it aggregated our tweets (but do I want to introduce myself to strangers through my tweets?) but I found judicious use of  twitterific on iphone more useful. But maybe I was just nervous… looking foward but not sure what to expect…

So this thing, Sicamp December 08,  kicked off with a Friday evening  of frenetic socialising –  trying to meet as many people as possible, and work out (if you didn’t know already) what  project team to work with. For the first time in my life, I managed to ‘mingle’ properly; that’s to say, I managed to move on from people with whom I’d have been very happy to spend the rest of the evening chatting with (about Erlang’s relevance to web technologies … or … music in Sao Tome … or…  how  “volunteering” and charitability can be the cause of holding back the development of the developing world …).

On Saturday morning I still had no clue what project I was going to be working on, and came in to the Young Foundation offices to find out what clever process they were going to use to divide us into teams. They stoked us up with coffee and  croissants then pointed out the labels with arrows to each of the project’s spaces in the buildings … and then got out of the way as people started running to get to into their chosen project. We-need.com (that had rechristened itself already) and Going postal (that later was to becom post post) were both very fervent teams, and had already organised what from the outside looked like a very disciplined structure from the off.

I had decided that being on a small team would be a good idea (not just an idea -  Denise Stephens had pointed out to us that  her Enabled By Design winning team last year had been the smallest); somehow I found myself gravitating to what was turned out to be the largest team, Useful Visitors. The idea was either about something I was skeptical of – getting business people to donate pieces of time to charitable works when they have extra time on trips to the developing world, or was about something I found more compelling – harnessing the drive (and guilt!) of any country’s diaspora. Very different kinds of visitors, very different kinds of visits, very different kinds of network/site that would need to be built.

As expected by all, the size of the team was almost our downfall… Femi, whose idea and project Useful Visitors is, was keen to get as many ideas as possible – bring it all on. For him the weekend was as much about driving the concept forward for the long term as about building a proposition and prototype for the Sunday show and tell. But then again we had to have something.

We split into groups, working very effectively on different ideas, but when we were trying to all pull together a consistent vision of the site / service and the user journey through it, it all felt like it was getting bogged down in treacle. Despite having initially asserted that I couldn’t help with the content/context, just the tech (and really I was right, I have no experience in these areas), I none the less became passionate advocate of this and that. In particular I loved an idea which we we called the Useful Lunch – an idea of a free exchange between a visiting business person (probably a high flier) with a local business person – over a local business lunch, that gives the visitor an insight to the country and context (and some company),  and the local access to some external points of view that may differ from what everyone around them are saying. A nice idea, but who knows if it would actually fulfil any existing social need.

But other more general ideas held sway (probably a good thing) my passions about particular ideas subsided (ditto). I was not 100% clear if the slightly abstract user journey that emerged on the whiteboard was intended to be a site that might take months to make, or what we were going to do in the weekend. Could be either. Too many people in the room perhaps. Too much design up front perhaps. Not enough coffee maybe. I looked over at Tom’s laptop  (Tom Ten Thij, a rubyist with way more rails & tech cred than me)… and noticed that he was coding up what he saw on the laptop into a database model and basic rails scaffold. To fill in the time. Yay! let’s just get stuck in. I often feel I just want to see something to get a reaction as to what it will be.

So we split into groups and Tom and I, together with  Tim Jackson standing in as our on-site customer,  made a basic construct of the a very simple arc of interaction working getting another . Funny things (I can say that, it wasn’t me doing the ‘fun’) like setting up deployment were more time consuming than we guessed (isn’t it always trickier than you guess) – a good idea to get that out of the way in advance, but we got it sorted and some decent work done before poor tired Sicamp leaders had to chuck us out of the building (that didn’t stop us coding of course).

There were a lot of other people working in separate teams and we occasionally crossed over, and sometimes stepped on each others toes (both metaphorically and literally – there were a lot of people in a smallish space).  I was aware of their work and had to dip in and out of my coding mind (which added to the brain ache slightly), to see what others were doing – pulling together stats and business cases,  getting video endorsements from high-powered contacts in Nigeria, creating graphics and designs which  required only subtle changes.

The next day we were fired up doing lots and more of the same. Somehow my enthusiasm for adding more and more to the site waned as it became apparent I was going to demo the site as part of the presentation team. Hmm, invited guests and cameras (The People Speak were on hand documenting and being video provocateurs throughout the wekekend) made me a little more nervous about adding in edgy features at the last minute. Tom managed to sneak in a cap deploy of what he had been working on a few seconds after I’d closed my laptop to leave for the big room at the Museum of Childhood round the corner. Chuckle.

I was really proud of what we’d done with the prototype http://uvisitor.com. At least until I saw what everyone else had done. Ouch! Now that stuff is impressive. AccessCity did four APIs to the webservice they’d built and an iphone app on top of one of them (ouch ouch). We-need did novel UI stuff (appropriate for their target market) and clever information visualization. Good gym actually ran after runners on the canal and got video of them signing up to join the website. OwnGrown seemed to have implemented a full trading system, as well as doing their first delivery during their talk. Do I have to go on?

The weekend was so packed with ideas and people (and code!) it seemed to pass by in a whizz, at the same time as feeling like it went by  in slow motion. I have a feeling it’s a weekend that will last for a long time.

UPDATE: I wrote this late at night, so I forgot the really obvious… a major thank you to all at SICamp for setting it all up, and all at and in the Young Foundation for putting us up and putting up with us (and our aftermath)!

UPDATE: Filled in Tim Jackson’s last name

After all, doesn’t the technique just guarantee an easier point of entry to the practice of screen-scraping (which mysociety’s sites themselves rely on)? A technique “with no coding required”, “an easy way for people to get into this sort of thing if they don’t have the coding skills of others”.

But. the two questions – technique and usage – are related, because if no shining example of the technique can be found, is it a valid technique? Perhaps one might argue, some people should be focused on technique (this is part of the domain of, say, software development), whereas others (statisticians, information designers, authors) should be focused on usage… and thus a technique might be neutral.

While described as “no coding required”, a glance at the original post, shows that, while no specific programming language is being used, there is coding going on (see Bonnie Nardi’s description of excel as a user-programming language in “A Small Matter of Programming”). The coding that’s going on is a form of re-presentation – taking data from one format (a textual table) and translating it onto a map.

The usage of similar forms of screen-scraping by mysociety websites is in the form of taking multiple sources of data and rather than just representing each, finding new value by aggregating (and/or re-partitioning them) in novel ways. This requires all the trappings of typical “coding” – hard work development involving persistent databases, coping with long-term changes in data and managing servers and server-loaded code.

All screen-scraping inevitably involves some stripping out of data (even if it is only of original context). There must be someway of adding back some new value for any screen scraping to

And it is precisely this “coding” that enables the value in mysociety sites. Is it possible that the technique, which doesn’t add value in the form of aggregation or repartitioning (filtering) but just “re-presents” the data, might be used in a context where radical value was added simply through a new presentation form, but I wonder if that’s possible… with ‘no coding required’.

We work with the latest technologies and with only the highest standard of multi-disciplinary developers and designers. These are both ways of cutting costs, but can also present risks (latest technologies carry unknowns, highest standard of developers delivery quality, but are expensive). For this reason we find that risk is the highest factor affecting cost, and we dedicate our working practices to controlling and reducing risk within the project, aiming to have a smooth curve in the decline in risk over the course of project’s timeline.

To do this, we use best practices & methodologies from the agile methods of software methodology (see Martin Fowler on the New Methodology and The Agile Manifesto). Overall this means focusing on interactions amongst the team, working software, and collaboration with our customer to achieve the needs of the project over its lifetime.

This leads to practical impact on what we don’t do, what we do do, and then how this reduces cost…

What we don’t do
In particular, we do not believe that it is worth spending a large tranche of time at the beginning of a project, writing up long specification documents, and engaging in back and forth between client and ourselves ironing out these documents until they are perfect. This is because these documents are often a major source of cost drain. Not only do they take time themselves, but they (perhaps somewhat counter intuitively) add to project risk. They seem to promote good understanding  – but with a focus on description of software rather than delivered software – and there is still a large gap (wherein a large risk) between the words of description and the delivery. Until software is actually coded, understanding what is achievable (at a high quality and within a reasonable time span) cannot be known. By fixating on a given specification, easy wins that come up during development cannot be capitalized on, and resources may focus on one hard immovable task. With so much communication and work gone on in the specification phase, during development, teams have a tendency to go uncommunicative. In order to deliver to a fixed feature set contract, punitive change control charges encourage everyone to minimize. Teamwork across client and supplier are discouraged.

What we do do
Of course it is necessary to establish common ground and focus.  We aim to first have a clear shared understanding of the context of the commissioned  project, including its aims and the outcomes and targets for our software. With this high-level view understood, the team (commissioner and supplier together) pick the most critical, small, achievable piece of functionality, and we progress that as soon as possible to working software, and deliver it to the customer. Things that are included in this software are working production quality code. This is not a prototype or mock up, or a rough draft. It is version 1 (of many version). All sorts of things can be added, improved, but the basic code quality is fixed at an initial (high) level. Code is subjeced to automatic tests (the (separate) body of test-code grows and grows through the lifecycle of the project -  a large body of “regression tests” ensuring that new functionality added doesn’t unintentionally remove or corrupt previous functionality).

This means risk has gone down. The gap between expectation and actual delivery has been lessened – so any misunderstandings or wrong directions can be corrected. The next step is to collectively choose – what is missing – why couldn’t this be used as it is? Usually, although many things may come up, one or two are in some way more critical or core to the project. These are then discussed and simply described, coded (and tested) and delivered. This then becomes of cycle of progressive improvement of the technology.

As time passes (in day or in weeks, depending on the agreed schedule of the project), the project team sees the software approach its delivery targets – quality being of a consistent standard, and more functionality being delivered to ongoing specification.

How this reduces cost
Time is not wasted specifying things that cannot or should not or will not be built. Communication gaps are kept to a minimum and forced out into the open as soon as possible. Risk is lowered over the course of the project. Budgets can be fixed and most high priority functionality focused on. Ongoing communications and delivery helps encourage transparency, trust and teamwork.

I tried the ‘Hipster PDA‘ index-cards-as-organizer thing – at different times in my life (yes, even before they blogged about it), but only ever for a while. It’s fun while it lasts (till my backing cardboard gets too bent and all my rubber bands snap ).

I’ve tried doing CRC cards, too (though never really caught on with me).

And then there’s story cards. We had a great experience with running sodaplay development with these for a while – until some of us needed to work remotely. Since then we’ve had some good experience of transferring the principles to Trac, but the mystique (and discipline!) of story cards (that derives in many ways from the physical features (let’s say, affordances, even) of index cards) is missing.

Enter Mingle, perhaps?

This is a rather exciting* looking offering from thoughtworks (which seems to be crossing the consultancy-product gap in interesting ways) which seems to combine the virtuality of a trac-like system with some of the power of index cards. Plus oodles of optional process-y stuff (that trac largely eschews).

*’exciting’: perhaps only understood in context, and maybe only if you are a process-anorak (oh dear, that must make me one). 

In some ways, I like the look of Mingle (I have to admit I haven’t tried it out, but only looked through the tour). However as a micro-business, with tiny (1-5 person) teams, all it does for me is increase my frustration with trac – don’t get me wrong, I do love trac, but it is aiming at a slightly different audience -  opensource and product-management, I’d say, rather than project iteration management.

But Mingle’s not aiming for my segment of the market either, as at least evidenced from the price point. It is just too steep a hike away from even the most luxurious of hosted trac offerings. And in any case that price is license-only (unhosted).

The per-user pricing also hits if you are project-focused (which red56 is) – because the team needs to include one or more customers, each of which need to be a team member (interestingly the customer isn’t really a part of most of the tour – are index cards meant to be only part of the development ‘back room’?)

This probably adds up to a request for a different product – hosted, aimed at smaller companies, and probably with some of the issue tracking features of trac as well (after all a bug/enhancement ticket/issue is part of the pipeline towards becoming a story  – though some (notably 37signals) disagree). Perhaps someone will write this application, launch a hosted service, sign me up and start billing me for it – or just tell me about it (send me one of those index cards with pictures on the back).

One odd thing that strikes me about Mingle – a ‘project collaboration and management tool for Agile software development’. Isn’t agile meant to be about “Individuals and interactions over processes and tools”? How do we measure that against promotion in Mingle of  features that “make it easy to ensure adherence to project compliance”?

Simple definitions:

The basic story is, that a URI (Universal Resource Indicator) is the general term, and that URLs (L for locator) are things that locate them (in simple language you can paste a URL into a application like a browser/email client/ftp client and it should know where to find it. One might think – well what else could a URI be? An example could be, a unique way of referring to a book by its ISBN number (isbn:0714844403). NB: The http:// or isbn: at the beginning is part of the URI-scheme (a format for defining a recognizable URI). If something is only a name it is ocassionally called a URN (N for name). However URL and URN are now viewed as being unnecessary partitionings – instead a URI may be a locator, and/or it may be a name.

A locator can be (simplistically) a “link” (the target of a hyperlink). This makes it very tangible and graspable by most internet users.

A name can be something that uniquely identifies/represents or corresponds to something or someone. This is a more abstract concept which is in the past has only been important largely for programmers and the digerati/webnoscenti.

Sources/References:

The most common source for defining URIs and URLs is the W3C page Naming and Addressing: URIs, URLs, … However the three dots are a give away, that nothing is actually resolved on this page, and within the jumble of links and metadata it is hard to find an authoritative source. Elsewhere for example, Tim Bray argues that, officially, “There is no such thing as a URL”. This is definitely wrong: According to the IETF (who really are the authoritative source)  RFC3986 section 1.1.3 URI, URL, and URN is the current ‘truth’. If you need more detail, there is in fact an entire RFC dedicated to the topic of the terms, and why the confusion has arisen (RFC3305). The word URL  still means what it always did – the location of something within compter-findable space (a web page, an ftp location, an email address….) – URLs are a specific subset of URIs – but the term is not that useful for definitive documentation as the “name” and “locator” aspects of URIs are not disjoint subsets, but overlapping.

Implications

This disposal of the terms URL and URN is useful – because it allows one to see what happens when the “locator” and “name” aspects of URIs overlap, or not…

Locator + Name  = Good

However it gets more intersting for everyone when a URI is both a name and a locator – particularly when this is the canonical form of the name. This name can then be a token which can be passed around, but also inspected for its meaning (by typing into a web browser’s location field). This is one of the bases of many effective and lightweight web phenonmena – lightweight APIs, blog permalinks, and (relatively) new phenomena like the socialgraphing, and XFN. As web users increasingly have to manage their own online identities through multiple intertwinglable social networking sites, the idea of the link that is also a name (as is implicit in XFN) will become important. URLs are great when they are names as well as locators, but are even better when their form (the words and format they use) is expressive of their “namefulness”, and explain their own contruction.

Locator – Name = Broken links

When you have a locator which is not a name, you have basically some arbitrary text string giving you a page, which often has its form of technology embedded in it (URIs ending in .jsp, .asp, .do, .action or encoding data in the querystring ?id=nnnn). The linkage is an arbitrary result of the creation of the site, and it is highly likely that these old URIs will not and cannot be preserved when the site is migrated to a new technology.

Name – Locator = Confusion

The converse is scary, when a URI looks like a locator (e.g. it is a http URI) but it is NOT a locator only a name. Most of the time this doesn’t happen. A good case where it does come is in XML Schemas: in namespacing for example – in which the namespace is defined by a URI with an http URI-schema, but that doesn’t demand that the URI works as a locator but is just a unique name (it could (and often does) generate a 404 when typed in a browser). However it is much more confusing with the XML Schema schemaLocation attribute:

The schemaLocation attribute value consists of one or more pairs of URI references, separated by white space. The first member of each pair is a namespace name, and the second member of the pair is a hint describing where to find an appropriate schema document for that namespace. [W3C Schema Primer]

This means that a schemaLocation contains a http-URI that is not a locator (not a URL!) followed by a space followed by a http-URI that is a locator. Hmm. Makes sense to me now (after six or seven years of battling with XML Schemas…).

I keep re-reading Joel Spolsky (whose last name is not, I discover, “on Software” – not, then, a member of some kind of minor german nobility) emphasizing that good programmers must be good writers. Well, actually when I google for him saying it, I find he actually says:

…years of experience working with programmers have taught me that programmers who can communicate their ideas clearly are going to be far, far more effective than programmers who can only really communicate well with the compiler.

and elsewhere goes so far as to say:

If you’re a software development manager and the people who are supposed to be writing specs aren’t, send them off for one of those two week creative writing classes in the mountains.

I don’t always agree what Joel says but it’s always strongly argued, and he does have a bit of experience. Maybe there’s something to it. Besides, I’ve been experiencing what I might call a writing gap – the gap between feeling I ought to be writing and actually having written something. But that gap (what, in university, would be called an “essay crisis”) has extended for me, for some twenty years – since thinking, in the midst of an essay crisis, that I really didn’t enjoy writing essays, and I would be better off doing something else.

This flight from writing lead me to working in software, where of course the acts of pitching and then speccing (and worse, delivering ‘specs’) necessarily involves much writing, although more and more effective speccing (I discovered) could be done with less and less (although some may beg to differ).

Inevitably, I found myself, like Alice through the looking-glass, getting involved in writing papers.

And above all, I can’t hide in my code – Joel tells me (and he’s so right) that it’s all about communication.

My main aim then, in ‘rebooting’ this blog (kickstarted with the posts from being Technical Director at Soda Creative), is to get myself writing again on subjects that connect my work in software with the world of people – groups, users, other developers. I’m not really worried about the outside audience yet (I’m mostly talking to myself), so don’t be surprised if (by some strange quirk of fate you’ve arrived here) that the posts are in a perhaps rough & ready state.

A word about dates: I’m planning to write things up on this blog, as of the date they were written (in whatever form) – this blog is then a form of record, rather than a publishing medium.