In two previous post ("When Basket Checkout isn't RESTful" and "What a RESTful Basket Checkout might look like") I have referred to the question of where basket state should be stored in a RESTful system. A number of conversations have ensued from these posts that I feel warrant a deeper post specifically about this question.

I will start by describing the typical basket / checkout implementation (which isn't RESTful).

The vast majority of web sites require that you authenticate (or are 'authenticated' as a new anonymous guest). A token is then passed to the client which is a reference to session state on the server. The user then browses the site, adding to the basket as they go. When the time comes to checkout, the client simply passes the token back to the server for it to retrieve the session state internally:

From the perspective of REST, this breaches the Statelessness constraint. From a more general perspective, the basket implementation is tightly coupled to the checkout implementation which leads to a monolithic Store Application. Serendipitous reuse is reduced or prevented. If a requirement is received to implement a rich client, for example, then either the client has to carry out site-specific work to orchestrate the basket and checkout or significant server changes must be carried out.

Moving away from this model requires that the client maintains the basket session state. For a browser, this might be in a local cookie or by using a browser extension such as Google Gears. For a rich client, this might be simply kept in memory or in a local data store. The specifics do not matter from the perspective of the client/server interactions:

The client does not need to interact with the server until the basket is ready for checkout. Where does the client get the data to build up a basket? It doesn't matter to the Checkout Application, which is no longer monolithic. Instead it is entirely decoupled from the rest of the store. Serendipity is enhanced because clients only need to know how to construct a valid basket representation and how to authenticate. In a RESTful implementation, the basket representation will be rendered using a standardized media-type and authentication will use a standard authentication mechanism such as WWW-Authenticate.

In "What a RESTful Basket Checkout might look like" I discuss, in passing, what a RESTful implementation of basket persistence might look like. Many online stores allow the user to add an item to the basket and then to come back in 6 months time and still see the item in the basket. Alternatively, perhaps the user should be able to see the items in the basket when they use a new machine. Clearly, a RESTful implementation should allow for such functionality.

REST has a Layered System constraint. So long as this is not breached (by, for example, the checkout knowing about the basket persistence mechanism) the client is allowed to compose multiple RESTful services. In this case, the client would compose Basket Persistence with Checkout:

The client is free to use whichever persistence implementation is desired. Perhaps one client implementation will use Amazon S3. Another might use the data capabilities of the new Azure platform or employ a custom persistence solution. The checkout application doesn't care.

I hope that this clarifies some of the questions and answers arising from my previous posts.

This is the talk I will be giving at DDD7:

In my previous blog post, I discussed "When Basket Checkout isn't RESTful" so here are some thoughts on what a RESTful implementation might look like.

This is what I want to have as the final step of transitioning to checkout (where application/basket+xml is a standard media type):

POST /checkout
Content-Type: application/basket+xml

<basket xmlns="http://example.org/basket">
<item id="1" count="1" product="http://example.com/product/green-ham" />
</basket>

This is deliberately trivial xml as the format itself is not especially important. What is noteworthy is that the basket contains all of the items that the user wants to purchase. The basket state might have been constructed without ever previously visiting the server, perhaps in a rich desktop client.

Upon receipt of the basket xml, the server sends a WWW-Authenticate challenge to authenticate the user. If the authentication succeeds, it continues with whatever workflow it has for checkout.

Note how this request fulfils the REST constraint to manipulate resources through the exchange of representations (in this case, of a user basket).

After checkout, the server would likely expose the current state of the order to the authenticated user, for example:

GET /order/abc123
Content-Type: application/order+xml

<order xmlns="http://example.org/order" total="12.34" xml:base="http://example.com/order/abc123" currency="GBP">
<item id="1" count="1" product="http://example.com/product/green-ham" price="12.34" />
<payment>...</payment>
<delivery>...</delivery>
</order>

OK, so now you can see what the end objective looks like, it makes the client side journey a little easier.

If the client is a rich client, life is simple: just store the basket state in memory or persist it to a local data store as desired. When the time comes to checkout, the necessary basket xml is crafted and then sent to the server using something like HttpWebRequest. All runtimes expose the functionality to send HTTP messages these days; read your documentation for guidance.

If you are in the browser then you will need to store the data locally as well. There are a number of alternatives. The most obvious is in a client cookie, but you could also use Google Gears if it is installed.

In my previous post, I discussed the proposal by serialseb for a 'personalized resource'. Purely in context to the client needing to store the basket state, this makes sense. You might, for example, have requirement to remember the last basket state in order to highlight previous purchases. What is imperative, however, is that only the client has knowledge of this 'personalized resource'. At no time does the server which will checkout the basket ever know about where the basket state is persisted. In fact, this is an excellent example of meeting the Layered System constraint of REST and applying information hiding.

For my money, that is a RESTful Checkout.

Following on from my twitter chat, here is why I think that the following two basket checkout styles are not RESTful.

GET /checkout

When the server keeps session state about a user basket (cart in the US) you will tend to see checkout as a regular GET. This is because the server keeps track of all the "Add to Basket" choices of the user. Somewhere, a token is being kept that links the user to the session state on the server; this may be a token in the URI, in a server cookie or in ViewState. Whichever method is chosen, the significant fact is that the server has knowledge of the application state of the client.

This is in breach of the hypermedia as the engine of application state (HATEOAS) constraint of REST. It also breaches the more general Uniform Interface constraint by failing to manipulate resources through the exchange of representations.

Let me be perfectly clear. This is effectively the standard pattern for implementing basket and checkout across the whole of web. It is perfectly valid as an architectural style: it is simply not RESTful, that's all.

Content-Type: text/uri-list

During the twitter chat, serialseb proposed an alternative approach whereby a 'personalized resource' of the user basket is hosted somewhere. In one version, this can be RESTful (see next blog post) but in another, it isn't.

If the client sends a message (where the provided link is a representation of the basket state):

POST /checkout
Content-Type: text/uri-list

http://example.com/username/basket

Then I don't consider this to be RESTful either because there is no representation being exchanged with the server by the client.

Again, to be clear, it's perfectly valid HTTP - it's just not REST.

In addition to the non-RESTfulness of the style, it has a couple of 'difficulties':

1. Which Content-Type should the server Accept from the provided URI?
2. The application state is inherently confusing. Should the server check if the basket is updated? If it does update, should the server change the order? To the original implementers of such a system, no doubt the answer is obvious but later implementers may not find life so easy. An example of this kind of problem can be seen with OpenID where sites tend to work with a select set of OpenID providers but break with others.
3. Access Control:
   • Does the URI provided require authentication? If not, private data is available for anyone to view. If so, how does the server hosting the checkout obtain the credentials?
   • How is the server hosting the checkout granted authorization to read the basket state?

These problems can clearly be overcome. For example, by complex redirects such as we see with credit card authorization today but the complexity is unnecessary and typically ends up with a situation of bespoke systems integration.

In short, this is another breach of the hypermedia as the engine of application state (HATEOAS) constraint of REST.

Finally, because of the lack of RESTfulness I consider that neither of these options are serendipitous.

I've saved the best bit of news about Windows 7 to last: it comes with native support to mount VHD images. This makes me happy. In the keynote it was announced that there will also be support to Boot to VHD. That makes me unnaturally happy.

I am guessing that the boot support isn't in the bits we got at the PDC because I can't see how to do it. But here are some screenshots of mounting a VHD natively using Disk Management. First, just right-click and select "Attach VHD" and enter the path to the file (not shown):

The disk image then shows up as an ordinary drive:

Explorer sees it as a regular drive:

You can then browse into the file system directly:

In the long-running arms race between Microsoft and software vendors, the notification area has become the latest arena for third-party applications to intrude on the user when they should know better. To combat this, the user will be given more control over which icons they allow to be displayed:

Furthermore, a new Message facility is being made available to that the user can deal with messages when they choose to rather than when the application intrudes:

Oh, and that little vertical box in the far bottom corner is the new "Show Desktop" button. It's clear when there are no windows to minimise. The clock is identical to Vista so far as I can tell.

Microsoft are bringing "Libraries" to Windows. I haven't played in depth yet, but they represent groups of similar files such as documents, pictures and so on. A world away from the promise of Longhorn but I am curious to find out how usable non-expert find this new functionality:

Windows search also gets some attention, with highlighting and a new style of showing results. Maybe it will grow on me but I found it visually cluttered and confused:

Microsoft have updated the functionality, usability and look of those common, everyday Windows applications. This includes adding the Ribbon to Paint and WordPad:

Media Player gets what looks like a visual tweak (the Aero effects on the control bar are gone):

Media Center appears unchanged at first glance:

Control Panel gets a slight tweak, still following the (highly usable) Vista style:

I don't know if the PowerShell ISE (Integrated Scripting Environment) is included in the install because of the developer audience or if it will be included in the RTM bites, but here it is:

I've made a (not terribly interesting) video of the installation of Windows 7 bits given out at the PDC. You can clearly see how closely related to Vista it is.

Unfortunately, I can't embed video on this blog so please visit it here.

Apologies for the poor quality - it's a camcorder recording (couldn't figure out if or how to get a direct video feed to the camera from the monitor output)

Last weekend I trawled through all the published sessions to separate out those sessions I was interested in attending from those that I wasn't (it turned out to be a long list of interest).

During the week, the Microsoft PDC site switched on the timeline planner so last night I did a second round of filtering to select the sessions that I intend to go to:

Sunday

• 10:00 - 17:45
  Agile Perspectives, Industry and Microsoft

Monday

• 11:00 - 12:15
  A Lap around Cloud Services Part 1
• 12:45 - 13:30
  "Dublin" and .NET Services: Extending On-Premises Applications to the Cloud
• 13:45 - 15:00
  The Future of C#
• 15:30 - 16:45
  A Lap around Cloud Services Part 2
• 15:00 - 18:30
  Agile Development with Microsoft Visual Studio

Tuesday

• 12:45 - 13:30
  WCF: Zen of Performance and Scale
• 13:45 - 15:00
  Project "Velocity": A First Look
• 15:30 - 16:45
  Project "Velocity": Under the Hood
• 17:15 - 18:30
  ASP.NET and JQuery

Wednesday

• 10:30 - 11:45
  Panel: The Future of Programming Languages
• 12:00 - 12:45
  Panel: The Future of Unit Testing
• 13:15 - 14:30
  Improving .NET Application Performance and Scalability
• 15:00 - 16:15
  SQL Server 2008: Deep Dive into Spatial Data
• 16:45 - 18:00
  Windows 7: New Shell User Experience API's

Thursday

• 08:30 - 09:45
  Dynamics Online: Building Business Applications with Commerce and Payment API's
• 10:15 - 11:30
  Microsoft .NET Framework: CLR Futures
• 12:00 - 13:15
  "Oslo": Building Textual DSL's
• 13:45 - 15:00
  An Introduction to Microsoft F#

At Charteris Day this week, we had a series of presentations discussing Agility. The various speakers covered both Business Consultancy (with a focus on our 'Customer Centricity' capability) and from the Technology Consultancy (with a focus on Agile software development). My contribution was to give a 20 minute talk explaining why I felt that the presentation ("Future Directions for Agile") by David Anderson at Agile 2008 in August was so important.

At the head of my presentation I discussed something that I have been articulating for some time now (including during the Park Bench session at the Alt.Net Conference) and I thought it would be worth repeating here to see if others agree with me or not.

First, a little background. During the late 90's I was developing shrinkwrap software using RAD. Whilst I was working for the IVIS Group on Tesco.com in 2002, I was introduced to eXtreme Programming. Since that time I have considered myself an extreme programmer, rather than an Agilist (for various reasons that I won't bore you with right now) and have acted as an agent of change at a number of organisations.

In hindsight, I think that at the beginning of the Millennium we genuinely believed that our principles and practices would prove to be viral and that the meme would spread. After all, we could demonstrate how effective the practices were. We could measure productivity and velocity in a far more transparent manner than  hitherto. The team members felt more empowered and gained more satisfaction from their work. Sponsors felt that, often for the first time, that they actually knew what their development staff were doing and how well. Sponsors especially appreciated the visibility of functional gain and the ability to direct the functional implementation. These characteristics were (and still are) real. It is not an illusion.

Given the reality of these powerful advantages, how could it not be viral? Why would the meme not spread? Yet it has not. At least, it has not in the way that we envisaged.

It is true that the term Agile Development has gained wide currency now, although perhaps not wide understanding. Many, possibly a majority, of development teams aspire to be Agile. It is also true that there are more agile teams now than there were. My observation about a lack of virality is separate to this.

So what do I mean? Over the years, I have engaged with a number of teams to, amongst other objectives, bring in Agile development practices; to act as an agent of change. To a greater or lesser extent, these engagements have proven successful and the teams grew demonstrably more Agile and effective. But then, the time comes to move on to the next engagement. After the departure of the agent of change, the team reverts to the status quo ante. Not immediately, but visibly and certainly. It isn't a deliberate, conscious decision to do so - it just happens, gradually. Six months or a year later, it is as if you had never been there.

Remember, this is a scenario where everyone was actually happier, more contented and fulfilled when they were Agile. Astonishing.

There have been times when I have questioned my own ability to effect change because of this phenomenon. It was very refreshing to hear David Anderson articulate that he has encountered the same problems.

If we accept that Agility is not viral then we are bound to start asking the question: why?

I will start by saying that I have no potted answer to this question, I am still in the process of seeking to gain understanding, but my first candidate explanation is this: "people are intrinsically lazy".

Ouch. That sounds contentious! Well, it shouldn't be. After all, there is a reason why we prefer to buy labour-saving devices. There is a reason why people walk across the grass along the hypotenuse of the triangle when the path follows the other two sides.

So why might laziness be an issue? I suspect it is because Agility is hard work. The practices we espouse are difficult and require us to work at a more challenging pace. Even if we feel better for doing it, it is easier not to do it. It is much the same as exercise. Why do gyms offer 'new joiner deals' in January? It is because they know full well that only a fraction of people will actually use their facilities all year and after the Christmas break people feel guilty about all that excess consumption.

Sponsors love the transparency of agile functional gain but it takes effort from them to achieve it. Sponsors are busy and in the absence of that agent of change hounding them to set aside time to contribute to the development process they start to miss meetings, they stop engaging fully with the development team and so agility is lost. The development team is just as guilty though. They too are busy people and it takes effort to ensure that the quality control practices are maintained. Knowing that the agent of change isn't around anymore to catch them out, they start cutting corners on their TDD. Before long, the team reverts to being less agile; possibly even entirely ceasing TDD for example.

The direction that David is taking in his talk is to look again at the CMM and he certainly marshals some strong arguments. He also challenges some of our practices by asking if Lean and Kanban are Agile. I am not yet wholly sure that these are the right answers but he is certainly asking the right questions.

I have a feeling that perhaps we have historically had a very mechanistic view of people and process. Perhaps this is why we assumed that agility would be viral, because we assumed utility was inherently compelling. When I use the term 'we' here, I suppose that I am really talking about alpha geeks. The same people who naturally gravitated to extreme programming, agile and now to alt.net. This is why there are so many discussions about "is alt.net elite.net?" and so on. If so, we need to recognise that most people are not like us. Even most software developers, I suspect. We therefore face the very real challenge of how to make agility sticky.

My suspicion is that we need to become sociologists and learn how to enculture organisations with agility.

I am very happy to say that I think that the agility presentations at Charteris Day look like they have succeeded in sparking the debate I believe is needed to grow our capability further as a consultancy; to better leverage the depth of knowledge in the company and to harness this for our clients. I'm sure that I will have more to say on this subject in the future.

P.S.
I will be attending the PDC Pre-conference "Agile Perspectives, Industry and Microsoft" session so if you would like to have a chat then or during the PDC just let me know.

I received the news this morning that my proposed REST session will be on the agenda for the upcoming DDD7 Developer Day.

For those who are not familiar with DDD, speakers are asked to propose sessions (community speakers only - not Microsoft) and then the community at large votes on what they want to see so it is always gratifying to get a session accepted.

DDD really is a great, fun event. It's held on a Saturday so you don't have problems with work commitments and it's totally free. I hope to see you there!

First question: "what stable, mature frameworks for REST are there?". Naturally, I challenge the basis of the question as I feel that this is the wrong thinking. Seb starts talking about his feelings around frameworks (he is more in favour than me, being the creator of OpenRasta). He describes OpenRasta as a 'REST enabler'.

Moved on to discuss the issue of RESTful authentication and touched briefly on why cookies break REST. Next, discussion about what, really, is service orientation. I put forward the divisions in the SOA community and comment that SO is simply the REST null style.

Discussion next of the importance of MIME types to REST, also discussing serendipity of reuse. Seb starts talking about HTML that isn't XML. really interesting discussion about the use and value of IANA - concern expressed that standardisation requires buy-in (I would argue not). Questions around who should start the IANA process asked.

Seb mentions best practice for conneg to specific resource representations.

Long conversation about auth & cookies (and why they are not RESTful) - quoted Roy.

Finished up on a discussion around HTML5 and the possible inclusion of PUT and DELETE from HTML <form> elements.

Something that I care deeply about. I give some background, referring to David Anderson's talk at Agile 2008. Discussion then moves onto concerns about 'ratchet decisions' that are so expensive to reverse that they are effectively one-way.

Interesting question: exactly what are the differences between anarchy and agile (for me, it is rigour). Which leads to the question of why hasn't Agile proven to be a viral idea. Ten years ago, I genuinely thought that simply demonstrating the value, training and mentoring would gain acceptance. But it didn't. I strongly suspect that the problem is that we have collectively failed to enculture organisations to agility.

Conversation moves on to discussing the management perception of agile practices, notably pair programming. Also discussing developer impediments, such as the lack of true shared ownership of code.

Next, I spout off a bit about my thoughts on 'tiger teams' (I will have to blog in more detail at a later date on my opinions about this). I also discuss my concerns about the 'cult of the amateur' in some of the management in the UK. Then, I am asked my opinion about where I think that Agile will be in the future (again, more later on this).

Finished up with an excellent, deep discussion about how, where and why Agile fails. (and we overran ...)

Discussion largely revolving around a comparison between NServiceBus and BizTalk. Interesting conversation going through the usage sceanrios of the two. My take-away is that NServiceBus is positioned as an 'easier way to do MSMQ', with a focus on long-running transactions.