Saturday, February 13, 2010

Source Code Management Friendly Design

Good software design, especially proper modularization, brings many benefits such as single points of change, encapsulation and a reduced event horizon, just to name a few. This is clearly a Good Thing at the source code level, but it also has important source code management (SCM) implications. Let's look at an example.

Suppose your code needs to process some data in several different ways. When I say processing think about things like manipulating the data, reacting to it, and so on. The simplest thing that could possibly work would be something like this:
public class DataProcessingEngine {

 public void process(Data data) {
  processOneWay(data);
  processAnotherWay(data);
 }
 
 private void processOneWay(Data data) {
  // ...
 }
 
 private void processAnotherWay(Data data) {
  // ...
 }
}
This code has many issues. Purely at the source code level we have a class that just has too many responsibilities. At the SCM level, this class could become a merging nightmare. Imagine multiple development branches each adding new ways to process the data. Merging these branches back onto the trunk will almost certainly result in merging conflicts.

Of course we can do a lot better. Let's factor out the different ways to process the data into seperate DataProcessors:
public interface DataProcessor {
 void process(Data data);
}
Each different way of processing the data would have its own DataProcessor implementation. The DataProcessingEngine now becomes:
public class DataProcessingEngine {
 
 private List<DataProcessor> dataProcessors;
 
 public DataProcessingEngine(List<DataProcessor> dataProcessors) {
  this.dataProcessors = dataProcessors;
 }

 public void process(Data data) {
  for (DataProcessor dataProcessor : dataProcessors) {
   dataProcessor.process(data);
  }
 }
}
Of course we still need to configure the DataProcessingEngine with the appropriate DataProcessors somewhere:
public class DataProcessingEngineFactory {

 public static DataProcessingEngine create() {
  List<DataProcessor> dataProcessors = new ArrayList<DataProcessor>();
  dataProcessors.add(new OneWayDataProcessor());
  dataProcessors.add(new AnotherWayDataProcessor());
  return new DataProcessingEngine(dataProcessors);
 }
}
You could also use something like Spring to do this for you. In this case you would end up with bean definitions equivalent to the code above. At the code level this refactoring has pretty much solved the problem. We now have a few small classes, each with its own responsibility. However, at the SCM level, the DataProcessingEngineFactory class (or equivalent alternative configuration) still sits in a single file causing merge conflicts.

To solve this problem, we have to make the system a bit more dynamic. If the DataProcessingEngineFactory could automagically detect all available DataProcessor implementations, adding a new way of processing the data would be as simple as adding a new DataProcessor to the classpath. As a result, there would be no need to change the DataProcessingEngineFactory every time, an no more merge conflicts!

In Java you have quite a few options to implement such a dynamic discovery mechanism:

Using the Java 6 ServiceLoader, the DataProcessingEngineFactory would end up looking something like this:
public class DataProcessingEngineFactory {

 public static DataProcessingEngine create() {
  List<DataProcessor> dataProcessors = new ArrayList<DataProcessor>();
  for (DataProcessor dataProcessor : ServiceLoader.load(DataProcessor.class)) {
   dataProcessors.add(dataProcessor);
  }
  return new DataProcessingEngine(dataProcessors);
 }
}

It's interesting to note that annotation based configuration systems, which are all the rage, typically don't hold all configuration information in a single location, which helps ease your source code management as I've shown. XML based configuration is more problematic in this respect.

Tuesday, February 2, 2010

OSGI Dependency Resolution is NP-Complete

Apparently, the resolution problem in OSGI is NP-Complete, as are similar dependency resolution problems like apt package installation on Debian Linux systems. If you think about it it makes sense, but who would have thought!

I still remember learning about P and NP problems, and the famous P = NP question at university, and being really intrigued by all of it. Lance Fortnow provides an excellent overview of the P = NP problem in Communications of the ACM. It turns out that this is no longer a computer science problem, but one of the fundamental questions in all of science. Fascinating stuff! If only I could find a simple proof :-).

Friday, January 1, 2010

REST Reservations

I'm currently reading RESTful Web Services by Leonard Richardson and Sam Ruby, and happened to come across an interesting QCon talk by Mark Nottingham on the status of HTTP. To me, this talk carries an interesting critical undertone on REST.

I've followed the whole REST and RESTful Web Services hype with some interest the last few years, and genuinely like several aspects of REST:
  • You can't go wrong with a KISS approach to software development.
  • Most everybody is familiar with how the Web works, so exploiting that familiarity makes a lot of sense.
  • The HTTP underpinnings of REST make it very well supported by a multitude of tools.
  • By leveraging HTTP, REST has excellent support for intermediaries (i.e. proxies and caches) and all the cool things they can bring to the table.

On the other hand, I also have some reservations about the whole REST thing, and Mark Nottingham touched on several of these topics in his talk:
  • As always, there is a lot of dogma to go around, and everybody has his own flavour. Is using this or that HTTP method in a particular way RESTful? Mark, who's obviously intimately familiar with HTTP, seems to have a much more pragmatic view on how you can or should use HTTP (i.e. listen to his comments on POST at the very end of the talk).
  • HTTP is deceptively simple. The spec is BIG, and full of ambiguities. The entire first part of Mark's talk deals with this, and I would argue that most people (me included) only have a superficial familiarity with HTTP.
  • Another thing I dislike about REST is that it's not generally applicable. For instance, suppose you're developing a search service, like Google (a typical text-book RESTful Web Service example). Most everybody in the REST community agrees that you should use HTTP GET for the search and encode the query in the URI: http://www.google.be/search?q=jellyfish. Imagine a similar search service that allows you to search for images that look like a given image. Logically this is completely in-line with the simple search service, but because of technical limitations of HTTP GET you're forced to implement this in a different way (WS-* is more consistently applicable this way).

For me, it's all about sane software engineering. If you have a well designed service, using established standards and conventions where relevant, people are going to be able to use your service without much problems. Things like REST and WS-* all have there place as tools in a developers toolbox, and it's up to us developers to make informed decisions on when to use what to build the best possible software we can.

Sunday, December 13, 2009

Concurrency Hides Latency

To paraphrase Brian Goetz's excellent The Concurrency Revolution: The Hardware Story talk at Devoxx09: "concurrency hides latency". In other words: use concurrency to combat latency. This really seems to be the common theme in current-day software development.

In his talk, Brian explained how CPU speeds have increased at a much faster pace than memory speeds the last few decades. This has resulted in a situation where going to main memory is now extremely expensive in terms of CPU time. It could take several hundred clock cycles to fetch some data from main memory. Brian captured the problem in anther very quotable statement: "memory is the new disk". Hardware designers have been trying to minimize the impact of memory latency on computer performance by clever tricks like speculative execution and extensive caching. However, we've now come to a point where these techniques are breaking down because the gap between CPU and memory speed is just too big. As a result we're seeing more and more multi-core CPUs: slower in absolute terms but designed for concurrency so latency is less of a problem.

In web applications, a round-trip to the server involves a lot of latency, so web applications are doing more things on the client in JavaScript and hide this latency by concurrently doing other things: think AJAX. In a typical back-end business processing system, a round-trip to the database also involves a lot of latency, so we use techniques like event driven architectures to process several transactions concurrently, again hiding the latency cost.

More and more performance is about data, not code (another quote from Brian's talk)! I've seen several situations where the efficiency of a particular piece of software is completely determined by it's data access strategy. Relatively speaking, fetching data from the database is so expensive that it doesn't matter that you process that data in a naive or unoptimized way. All of this of course also implies that a key design question is the data access patterns and data structures used by the application. If you want a fast application, data is your key concern, and you can fight the latency of getting to the data by using concurrency.

Saturday, November 28, 2009

Hidden Benefits from Commenting Your Code

In his famous book Refactoring: Improving the Design of Existing Code Martin Fowler mentions comments as a possible code smell: if you need comments to explain what your code does, try to refactor it so that the comments become superfluous. Uncle Bob also mentions this same principle in his Clean Code book: most comments are bad comments, typically redundant or misleading.

I largely follow this reasoning. However, I've also seen an interesting benefit from trying to comment your code, especially with higher level comments, for instance JavaDoc comments on classes. Writing these kinds of comments forces you to explain things like the purpose and responsibility of the class. Personally, I've had several occasions where explaining things in plain English highlighted problems. If it's hard to explain what the class's purpose is, or what the reasoning behind the class or method names is, this typically means you have code problems. Maybe the class has unclear or too many responsibilities? Maybe the class name doesn't communicate its purpose well? Maybe the class's role in the system is confusing, and so on.

Nowadays, I try to keep comments in internal code to a minimum, but I do try to write a brief JavaDoc style comment for each class, forcing myself to go through that explaining exercise I just mentioned. I do the same thing for methods that are part of a public API: they also get brief JavaDoc style comments.

Tuesday, November 24, 2009

Page Flow Challenge Clarification

The Page Flow Challenge in my previous post warrants a bit of clarification. A few readers had questions about the usefulness or sanity of certain steps in the test scenario. I'll go through the scenario and highlight what is being tested in each of the steps.
  1. Start the page flow
  2. On Page 1, enter "foo" and click Next
    You should end up on Page 2 where "foo" is displayed as input from page 1
  3. On Page 2, enter "baz" and click Next
    You should end up on Page 3 where "foo" and "baz" are displayed as input from the previous pages
  4. Click the browser back button
    Without any browser warnings you should end up on Page 2, where "foo" is still displayed as input from page 1
The first part of the scenario sets up a basic set of data associated with the page flow: "foo" and "baz" at this point. At step 4 we're using the browser back button to backtrack. This tests POST-REDIRECT-GET behaviour: if the application does not use POST-REDIRECT-GET, the browser will show a warning before displaying Page 2 again. This is so because Page 2 is the result of posting the HTML form on Page 1, and modern browsers never automatically resubmit post data. Essentially we're protecting the user from accidental resubmits. Not having the browser show ugly warnings about resubmitting also makes the application feel stable and trustworthy.
  1. Right click the Back link on the page and do Open Link in New Window
    The new window should show Page 1 where "foo" is displayed in the input field
  2. In the new window, enter "kit" (replacing "foo") and click Next
    You should end up on Page 2 where "kit" is displayed as input from page 1
  3. In the new window, on Page 2 enter "kat" and click Next
    You should end up on Page 3 where "kit" and "kat" are displayed as input from the previous pages
  4. In the original window, click the browser refresh button
    Without any browser warnings you should see Page 2 again with "foo" as input from page 1
Step 5, admittedly a contrived step, simulates the user branching off the page flow into two separate windows. The point of doing this is testing whether or not the two branches are isolated. In steps 6 and 7 new data is entered ("kit" and "kat") that potentially interferes with the previously entered data ("foo" and "baz"). Step 8 verifies that no interference happened: the page flow in the original window still has "foo" as data.
  1. In the original window, enter "bar" and click Next
    You should end up on Page 3 with "foo" and "bar" as input from previous pages
  2. In the original window, click Finish
    You should end up on the start page of the application: the page flow has finished
  3. In the original window, click the browser back button
    You should either receive an error immediately, or when you try to resubmit by clicking Finish again
In step 9 and 10, we complete the page flow and finally submit values "foo" and "bar". At this point the page flow has finished. The state and navigation management of the framework should have invalidated the page flow at this point, making it impossible to resubmit. Step 11 verifies this: if you go back using the browser back button, you should not be allowed to resubmit. This part of the scenario tests intentional double submit protection (classically solved with something like a synchronizer token).
  1. (For extra bonus points) In the new window, (where you still are on Page 3) click Finish
    You should receive an error of some sort (potentially a automatic page flow restart)
The final step of the scenario tests whether or not the framework tracks the two branches of the page flow as being part of the same original conversation. This is an advanced form of double submit protection. In most cases this is a beneficial extra protection. In some cases this might not be what you want.

To make all of this a bit more tangible, let's suppose this is an airline ticket booking flow. The scenario would go something like this:
In step 2, on Page 1, the users fills in basic traveller information (name, age, seating class) and advances forward. In step 3, on Page 2, he selects a seating preference. Page 3 in step 4 shows an overview of the captured information and asks for confirmation. The user notices that he made a mistake and backs up.
Steps 5, 6 and 7 has the user thinking he might want to try a seat in a different seating class and compare the prices. Page 3 in step 7 displays an overview of the price with the new seating class.
In step 8 and 9 the user completes the page flow with the original seating class. On step 10 he compares the prices of the two alternatives and decides to submit the first of the two alternatives.
Steps 11 and 12 verify that the user cannot accidentally book two tickets, either by resubmitting the first alternative or submitting the second alternative. If he really wants another ticket, he'll have to start the ticket booking process (the page flow) again.

I hope this clarifies things a bit and shows that the test scenario is actually somewhat realistic.

Sunday, November 22, 2009

A Page Flow Challenge

In this post I setup a page flow challenge to find out if the new or up-and-coming web frameworks handle page flows well.

Why am I doing this? Over the last several years I've seen quite a few blogs and discussions where people talk about how web framework X handles page flows, typically comparing the page flow functionality in framework X with Spring Web Flow. The problem with most of these comparisons is that they focus on just two parts of the page flow puzzle: expressing page flows through some kind of DSL, and reusing page flows as modules throughout the application. There is an important third piece: managing navigation and the associated state. The reason why this is often overlooked is probably because the navigation and state management is completely automatic in Spring Web Flow, so many people don't even realize that it's there.

While attending a session on the Lift framework last week at Devoxx09, this issue came up again. Somebody (not me :) asked whether "Lift has page flow functionality like Spring Web Flow?". The answer was as expected: the Lift developers have a page flow DSL in the works but seem to have missed the navigation and state management part (update: this appears to be incorrect, Lift does have navigation and state management.).

It's not enough to be able to define a page flow, the framework also needs to manage it! To help users and framework developers realize this, I've setup a page flow challenge. The idea is that you develop a simple 3-page page flow in your framework of choice and test whether it can correctly run a somewhat contrived scenario that tests things like POST-REDIRECT-GET to avoid accidental resubmission, double submit protection to avoid intentional resubmission, and correct state management. In essence the page flow should behave correctly whatever the user does and protect the user from potential mistakes.

Let's look at the page flow:



Easy peasy. This could for instance be a flow to submit a payment in an electronic banking application, or a flow to handle checkout in a web shop.

The page flow challenge scenario is as follows (note: I've clarified this scenario in a follow-up post):
  1. Start the page flow
  2. On Page 1, enter "foo" and click Next
    You should end up on Page 2 where "foo" is displayed as input from page 1
  3. On Page 2, enter "baz" and click Next
    You should end up on Page 3 where "foo" and "baz" are displayed as input from the previous pages
  4. Click the browser back button
    Without any browser warnings you should end up on Page 2, where "foo" is still displayed as input from page 1
  5. Right click the Back link on the page and do Open Link in New Window
    The new window should show Page 1 where "foo" is displayed in the input field
  6. In the new window, enter "kit" (replacing "foo") and click Next
    You should end up on Page 2 where "kit" is displayed as input from page 1
  7. In the new window, on Page 2 enter "kat" and click Next
    You should end up on Page 3 where "kit" and "kat" are displayed as input from the previous pages
  8. In the original window, click the browser refresh button
    Without any browser warnings you should see Page 2 again with "foo" as input from page 1
  9. In the original window, enter "bar" and click Next
    You should end up on Page 3 with "foo" and "bar" as input from previous pages
  10. In the original window, click Finish
    You should end up on the start page of the application: the page flow has finished
  11. In the original window, click the browser back button
    You should either receive an error immediately, or when you try to resubmit by clicking Finish again
  12. (For extra bonus points) In the new window, (where you still are on Page 3) click Finish
    You should receive an error of some sort (potentially a automatic page flow restart)
UPDATE: I've introduced a new step 11 in the above scenario, and what used to be step 11 is now step 12.

The only extra rule to keep in mind is that the HTML forms should use POST, to follow proper web design rules.

I've made reference implementations of the page flow challenge in Spring Web Flow 1 and 2. The source code is available in Subversion:

Spring Web Flow 1: https://svn.ervacon.com/public/spring/samples/trunk/pageflowchallenge-swf1/
Spring Web Flow 2: https://svn.ervacon.com/public/spring/samples/trunk/pageflowchallenge-swf2/

If you have Subversion and Maven installed, getting things up and running is as simple as:
$ svn export https://svn.ervacon.com/public/spring/samples/trunk/pageflowchallenge-swf2/ pageflowchallenge-swf2
$ cd pageflowchallenge-swf2
$ mvn jetty:run
You can now point your browser to http://localhost:8080/pageflowchallenge-swf2/ to test things out.

This challenge is solely focussed on page flow functionality, so it's not meant to reflect on the quality of a particular framework in other areas. For instance, SWF1 and SWF2 handle this scenario flawlessly, but probably have a harder time dealing with some other use-cases.

If you implemented the challenge in your web framework of choice, please let me know in the comments! I'm especially interested to see how Wicket, Seam, Lift and GWT fare. If you think this is irrelevant for your web framework because it handles things differently, I'd love to learn why you think this is the case, and how a use-case like this would then be implemented.