Distributed Compensation with REST and jBPM

September 18, 2007

billburke jbpm, REST, transactions, Webservices 9 Comments

Over the past few months I’ve been leading up to something. Although I was researching and diving into specifics of compensations, jbpm, and REST, I had a much grander purpose in writing these specific blogs:

These blogs were specifically written as a thought exercise in what it would take to write a coordinated distributed application. What pieces would we really need? Where does WS-* fit in? How about REST and BPM? Do we really need a business activity framework? How can we write scalable and maintainable distributed systems? I want to pull together my thoughts on the answers to these questions in this blog entry as well as ask a few more.

Maintainability: REST over WS

One of my concerns is the complexity of the WS-* layer and how it affects maintainability and evolvability of a distributed system. WS-* requires both the client and server to have the necessary middleware components and libraries installed to work. A distributed application usually means that it is an integration project. You are probably integrating with systems you do not control. There is probably a good chance you would not be able to bring down these systems to install or upgrade their middleware architecture. They may be poorly maintained or even not maintained at all. This is where the idea of REST comes in to simplify things and make things more usable over time. Since it has constrained, well defined interfaces over a mature and stable protocol, it has a higher chance of letting your distributed applications evolve as the WWW has evolved.

What I continually like about REST over HTTP is that there is little to no requirement on either the client or server side for additional libraries or generated stubs other than a http client library and a http server. This doesn’t mean that you cannot use a client and/or server side framework to implement your RESTful services or connect to an existing service. You are free to use whatever you want to make yourself productive, but not at the expense of requiring the client or server to have anything installed other than HTTP support.

Flexibility: BPM vs. WS-Business Activity

Earlier this year Mark Little introduced me to the idea of compensating transactions and a framework that could manage such an application. My first thought was that this stuff was interesting, but my intuition was telling me that compensations were something that might have to be modeled rather than managed. That compensations were more of a business process than something that could be coordinated automatically. My feeling is that compensation could easily become too complex for a transaction manager to handle generically. That if you modified a transaction manager to handle these more complex scenarios it would quickly start to look like a bpm framework.

Take an abort event for example. The coordinator in a compensating framework like WS-Business Activity has to make decisions on what order to perform a compensatory action on the resources participating in the activity. A default LIFO ordering seems very appropriate on the surface, but it falls apart in even the simplest of examples. For instance, let’s revisit the example I gave in my previous blog of the ordering of a computer at an online retailer. The business process might look like this:

  1. Bill credit card
  2. Pull parts from inventory
  3. Assemble computer
  4. Install software
  5. QA
  6. Ship

Let’s say the customer canceled the order after it had reached the QA state, #5. Having the customer wait for a LIFO compensation to receive refund on their credit card is just not acceptable. They will not want to wait for installed software to be wiped, the computer disassembled and parts sent back to inventory before they receive their refund. Therefore a 1-4-3-2 ordering of compensations might be more appropriate. Let’s expand this scenario even further. What if the customer paid in cash? The cash refund process might take just as long as disassembling the computer would. Maybe it would make more sense to do the refund in parallel with the other compensations? So, state #1 could be done in parallel with 4, 3, and 2, but, 4-3-2 still has ordering dependencies. You can’t disassemble the computer before you’ve wiped the hardrive. You can’t send parts back to inventory before you’ve disassembled the computer. So, you see, in even the simplest of business activities, you can have variable and complex compensation ordering. This is why I think a bpm engine like jBPM might be a better solution to manage compensations.

When I had discussed this with Mark, he made a good point that you are just replacing one coordinator(BA) with another (jBPM). You still need a coordination engine. Although he is semantically correct, I think the solutions are very different. With BA you have something generic and decoupled from the business process. In a distributed system, it is also a defined contract that both the client and server must obey. Which brings me to my favorite quote in the O’Reilly RESTful Web Services book:

The client should be in charge of managing its own path through the application

While WS-BA puts the onus on both the client and server to have WS libraries that are of the same version and can interact with one another. This all works beautifully in controlled environments. You have a very nice decoupling between compensations and business logic in both client and server side code. But…. What happens when BA versions are out of sync, there’s interoperability problems, or even worse, one or more of the services you have to coordinate does not support WS-BA? With jBPM, success and failure paths are a part of the business process. There isn’t necessarily a contract between the client and server as the client is responsible for knowing how to compensate and to perform the compensation. You still have strong decoupling in a bpm driven activity. The difference is that instead of compensation being hidden by the server, it is visible, and in your face, embedded directly in the business process. With jBPM, you can look at the process diagram and know exactly the flow of success and failure conditions of your distributed application. With BA, you would have to dive into the documentation and code to determine how your application functions.

In his blog in late August, Heuristics, one-phase commit and compensations, Mark Little talks about his early compensation framework and its interaction with heuristics:

In both of these problem scenarios what typically happens is that someone (e.g., a system administrator) has to get to grips with the inconsistent data and figure out what was going on in the rest of the application in order to try to impose consistency. One of the important reasons this can’t really happen automatically (at the TM level) is because it required semantic information about the application, that simply isn’t available to the transaction system. They compensate manually.

Until then. What we were proposing was allowing developers to register compensation transactions with the coordinator that would be triggered upon certain events, such as heuristic outcomes or one-phase errors. And to do it opaquely as far as the application developer was concerned. Because these compensations are part of the transaction, they’d get logged so that they would be available during recovery. Plus, a developer could also define whether presumed abort, presumed commit or presumed nothing were the best approaches for the individual transaction to use (it has an affect on recovery and failure scenarios).

Reading this fed more fuel to my idea that compensations really belonged as part of a process model. Transactions that have heuristic outcomes many times require human intervention as well as action by the framework itself. Yes, there’s a lot a transaction manager could do to make it easier to record business activities, but you’re still going to have interact with a real person to resolve many issues. jBPM is great at modeling work that requires a human.

But what about reliability?

Great, so we’re going to use something like jBPM as both our coordination and compensation engine and REST to expose our services on the network. Can we trust these things to provide us with the reliability and consistency that we expect out of something like WS and WS-BA? If you’re modelling compensations in jBPM, you’re gonna need the ability to reliably perform these compensations. Luckily, I’ve already answered how you can guarantee execution and failure handling in jBPM. But what about REST, what does it add? If you have truly written RESTful services, you have a lot of reliability and consistency built into your application. HTTP GET, PUT, and DELETE are supposed to be idempotent, meaning that it doesn’t matter how many times you’ve invoked them, the result is the same. While jBPM can guarantee you execution and failure handling, it can’t guarantee that these events won’t be duplicated. Since your RESTful services should be idempotent, you won’t have to worry about it! (I know, easier said then done.)

Where does that leave WS-BA?

I did discuss my ideas a bit with Mark. One of his comments that struck home was “Imagine if you had to explicitly tell all of your XA resource managers to prepare and commit?” I’ll admit that in a WS-BA world, writing a client that integrates a bunch of distributed services into one business activity would be uber simple and clean. The work being done with the JBoss Business Activity Framework simplifies the server side greatly as well. A compensation framework like JBoss Business Activity, might work really really well in the controlled environment of a web application. Until I actually write one of these applications or talk to a bunch of users and customers that have used WS-BA to model their integration project, I will remain skeptical. I just hope that I can get the feedback I need to further formulate my thoughts on this subject. Your thoughts and experiences??????

REST vs. SOAP: Ops vs. Dev?

August 22, 2007

billburke REST, Webservices 5 Comments

I’ve been reading a lot about REST, REST vs. SOAP, and security lately and came across this email thread post about how SOAP is so unsecure:

REST is not a security panacea. There is no silver bullet. All we can do is evaluate each layer’s *contribution* to security. SOAP’s is negative.It detracts from the security of the protocols that it runs on top of. If that were not the case then you would not be defensive about the idea that SOAP is designed to bypass firewalls.

I think I agree. SOAP does detract from the security of the protocols it runs on top of. Its trying to tunnel through port 80 and sneak past operations aversions to things like CORBA traffic. But is this such a bad thing? As far as this particular aspect of the SOAP vs. REST debate goes, what is the real underlying issue here? What I think it is is a matter of control. Ops doesn’t want CORBA/SOAP like requests because they can’t control what’s going back and forth over the wire. Control of the application has passed from operations to engineering.

From a developer point of view, we usually hate operations. You know the people I’m talking about. The control-freak paranoid dweebs that take at least a day to respond to any of our requests? They are the ones enforcing 10 character passwords that must be both alpha numeric, contain punctuation, upper and lower case, and that change once a month. They are the ones we have to sit in 3 hour meetings on whether to upgrade a certain library. It is like pulling teeth to get an operations guy to do anything beyond what they are trained to do. They are change resistant. They usually end up being blockers to our productivity. So its only natural that us developers would want to tunnel over HTTP to bypass these clowns. We have problems to solve, deadlines to meet, management breathing down our backs. They are the bureaucracy. Nobody is gonna come down on them when we can’t meet our deadlines.

On the flip side, operations is used to handling security issues. They have pre-existing tools and experience to deal with securing HTTP resources. Plus, managing the security aspect of an application is a tedious task. As developers, don’t we want ops handling this? Also, do we really want to waste the time training these idiots to a new technology? I guess it really depends how tech savvy and un-unionlike your ops guys are.

Side note:

Man, seems like I’m falling more and more into the REST camp. Its simpler. I want it to succeed. I’ve always disliked WS-*. But really, its probably because I’ve been focusing on pro-REST propaganda lately. I need to read some pro-SOAP articles.

RESTless, RESTFul, or RESTlike?

August 20, 2007

billburke REST, Webservices 8 Comments

A couple of weeks ago I ran into my old mentor from my Iona days, Steve Vinoski. I was in shock. The guy I looked up to (and still do) for years, the guy who used to have sideburns the size of Elvis’s turned all “queer” on me. How could the guy who taught me everything I know about distributed systems go all RESTful on me? Telling me basically that he’s hooked and will never go back. I had read a little bit about REST earlier and was a little skeptical, but now I really had to take a closer second look.

Its funny, REST seems to be a bit like learning AOP. With AOP, one or two sentences just doesn’t articulate exactly what it is, or how you are supposed to use it. REST seems very similar in this regard. I’m almost getting it, but am still a bit fuzzy on how to do certain things. Since REST is more of an architectural style rather than a concrete API, I guess my slowness and confusion is a little justified. Or maybe its because RESTful articles and blogs seem to spend so much time trashing SOAP and saying how wa-wa-wa, “the Web is proof of why REST is so great”, in other words, “REST is great because of REST”, and very little time on actual examples is why it takes so much reading to get to the heart of it. I don’t know…. Maybe its all my CORBA baggage? All my Java EE baggage? Anyways, I don’t want to go into yet another, what is REST blog. If you want information on the definition of REST, the REST Wiki is a better source, found this link too. No, what I want to do is go through my own thought process to see if the REST way of things is the golden path. To do that I want to walk through some of the basic cornerstones of REST.

Use of URIs

One of the cornerstones of REST is to have resource location modeled as a URI. This seems like a no brainer. When we modeled our EJB 3.0 container on top of JBoss Remoting, which is URL based, our design became much simpler and things like request failover were much easier to write. Yeah, yeah, not every EJB has its own URI, but moving to URI based endpoints made implementation easier. So +1 for REST here. Its interesting though. How many times do we run into issues in Java EE land because it does not define unified addressing of its services? Its been a big roadblock in writing portable frameworks. WS-* has to have a whole spec on the subject, funny eh?

Uniform Interface

RESTful articles make a good analogy between SQL and the RESTFul architecture. I won’t paraphrase, but instead, directly quote:

It is nevertheless imperative to question our basic assumptions. Consider a similar debate taking place in the 1960s. A radical researcher within IBM states that it is incredibly inefficient for each application to determine for itself how to store, index and search its data. He proposes the concept of a “generalized database”. Clearly this “database” would not be as optimized for a particular problem domain as a specialized data store. But then imagine the interoperability benefits if many applications could access the same data in a uniform fashion. Then, later, another IBM researcher makes an even more ridiculous claim. He states that the vast majority of the structured information in the world can be manipulated with only four operations: SELECT/INSERT/UPDATE/DELETE. Ludicrous! What about AddCustomerRecord, DeleteCustomerRecord, InsertCustomerRecord, UpdateCustomerRecord?

The idea is, that in a RESTful architecture, everything is modeled in terms of resources accessed through a URI, and only 4 CRUD HTTP operations of GET, POST, PUT, and DELETE. From my perspective it is very counter-intuitive to express object interfaces in terms of 4 methods. Doesn’t make much sense as an object designer of local models, but for distributed systems, it does make a lot of sense I think. The CORBA in me balked at this idea, but then I thought about it a little more. How many of us had interoperability problems between different vendors or even different versions? Not only that, but in WS-* land most of the time you need to generate and compile stubs so that you can actually invoke on a web service. With REST, all you need is a HTTP client library. With WS-* you need the HTTP client library plus all the other junk. While HTTP is a mature, well-known, agreed-upon protocol, WS-* is such a moving target.

The question is, with the uniform interface of GET, PUT, POST, and DELETE, aren’t we just pushing the complexity to the content type? To the data that is being transferred and received? I had this very same argument with Vinoski. After thinking about it for awhile, i’ll say, that SOAP has the exact same problem. With SOAP, you still have to define and agree upon the API. The REST approach seems simpler because the protocol is set and simple, it doesn’t have the baggage of SOAP which is a protocol tunneled over another protocol (HTTP).

With the uniform interface approach, it does seem like it would be a lot easier to model security roles. It would just be assigning CRUD permissions really instead of assigning individual permissions to each method like you do in Java EE land. Caching is obvious because GET should mean the operation is idempotent. I don’t know…uniform interface and protocol seems pretty compelling.
Content-type

Don’t have a lot to say here other than, doesn’t XML schema sort of make SOAP obsolete?

Statelessness

I remember one of the students in the JBoss advanced training a few years ago talking about holding the state of a SFSB in the client proxy and transferring it to the server with every request. This would make sessions really failure proof and make the server fully scalable. An interesting idea. I was thinking that this could be done with Seam conversations as well, i’ll get into that in another blog in a few weeks or so. This idea of the client maintaining conversational state and passing it to the server is compelling, at least in theory. My last blog I talked the JBoss Business Activity service. What if the client was the coordinator for business activities and managed all compensations? The idea focuses around a quote I read in O’Reilly’s RESTful Web Services book:

The client should be in charge of managing its own path through the application

I’d like to dive into this more in future blogs on both Seam and BA. But all these ideas sound pretty RESTlike to me, at least in terms of statelessness.

That’s all for now

Well, I pretty much asked more questions than I answered here. Probably didn’t enlighten you, dear reader, at all about REST. Maybe confused you a little more? Or maybe I just showed how much I am confused? 🙂 Like AOP, I think I have to actually implement something RESTful before I actually understand if it is the right approach. Right now I’ll say my leaning are neither RESTless nor RESTFul, but rather RESTlike. Who knows, I’ll probably end up getting angry at both approaches(SOAP and REST) and design my own BURKEful way of doing things. Such a burkeful architecture would definately only work on JBoss though 😉 Let’s see what happens….

P.S.

It is fate. A day after I wrote this blog (I write a few days or weeks before I publish), this TSS article came out. I’m also taking a look at O’Reilly’s Restful Web Services book. I’ll let you know sometime how it goes.

Compensating Transaction Framework

August 14, 2007

billburke jboss, transactions, Webservices 5 Comments

Mark Little’s team has put together a nice annotation framework on top of the JBoss TS, Business Activity service. What?!? JBoss has a Business Activity service? Yup, we do. Have had it for a long while, accept nobody but Mark (did he even?) made any noise about it within the JBoss community. What is the Business Activity service? Mark does a better job in an article he wrote a few years ago, (his book is even better), so I won’t get into a lot of detail. To summarize in my own words, the Business Activity service looks very much like XA and traditional transactions, except they are really meant for long running conversations or activity. Traditional transactions are not good in that scenario because they hold onto resources too long. Particpants in BA look a lot like resources in XA. The register themselves with an activity. An activity is very similar to a transaction in that you begin it, and either cancel (rollback) or close (commit) the activity when you are finished. Participants can interact at closing with a 2 phase protocol, much like 2PC in regular XA. The difference is that the rules are much more lax. Participants can choose to leave the activity whenever they like as well as some other neat things. See Mark’s book or article for more details. Oh, and I forgot, this is a WS-* specification.

So back to the annotation framework. Its pretty cool. It allows you to tie in a compensatory action to an executed method. So, if the business activity is cancelled, the coordinator will automatically callback to this compensation. Here’s an example:

@Stateless
@SOAPBinding(style = SOAPBinding.Style.RPC)
@HandlerChain(file = "jaxws-ba-handler-server.xml")
public class HotelBAImpl implements HotelBA
{
	@WebMethod
	@BAService(BAServiceType.MODIFY)
	@BAAgreement(BAAgreementType.PARTICIPANT_COMPLETION)
	@BACompensatedBy(value="cancel",type = BACompensationType.PARAMETERS_MATCH)
	public Integer book(String user, String pass, Integer room)
	{
		// ... business logic
		return reservationNumber;
	}

	@WebMethod
	public void cancel(String user, String pass, Integer room)
	{
		// ... business logic
	}
}

So, in the example, if you invoke the book() method of the Hotel service, it will interact with the coordinator to register a compensation action. This action will be to invoke the cancel() method with the exact parameters that were passed into the book() method if the activity is aborted. Alternatively, you can map the book() method’s return value and parameters to the compensation action using the @BAResult and @BAParam.

{
	@WebMethod
	@BAService(BAServiceType.MODIFY)
	@BAAgreement(BAAgreementType.PARTICIPANT_COMPLETION)
	@BACompensatedBy(value="cancel",type = BACompensationType.CUSTOM)
        @BAResult("reservationNumber")
        public Integer book(@BAParam("user") String user, @BAParam("password") String pass, Integer room)
        {
           // ... business logic
          return reservationNumber;
        }

        @WebMethod
        public void cancel(@BAParam("reservationNumber") Integer reserverationNumber,
                                 @BAParam("user")String user, @BAParam("password") String pass)
        {
           // ... business logic
        }
}

There’s also a CompensationManager object that allows you to programatically put data into the activity that can be referenced by the @BAParam annotation:

{
        // injected field
        @CompensationManagement CompensationManager cm;

	@WebMethod
	@BAService(BAServiceType.MODIFY)
	@BAAgreement(BAAgreementType.PARTICIPANT_COMPLETION)
	@BACompensatedBy(value="cancel",type = BACompensationType.CUSTOM)
        public Integer book(String user, String pass, Integer room)
	{
		// ... business logic
               cm.put("reservationNumber", reservationNumber);

               return reservationNumber;
        }

        @WebMethod
        public void cancel(@BAParam("reservationNumber") Integer reserverationNumber)
        {
           // ... business logic
        }
}

There’s a few more ways to define a compensation. You can have the method compensated by another different EJB method, Web Service, or even a plain POJO. I won’t get into further detail. You’ll have to read the documentation and take a look at the demo app that comes with the distribution.

Client interface

The client interface is very simple and similar to UserTransaction:

public abstract class UserBusinessActivity
{
    public static final int ATOMIC_OUTCOME = 0;
    public static final int MIXED_OUTCOME = 1;

    /**
     * Start a new business activity with atomic outcome.
     * If one is already associated with this thread
     * then the WrongStateException will be thrown. Upon success, this
     * operation associates the newly created transaction with the current
     * thread.
     */
    public abstract void begin()
        throws WrongStateException, SystemException;

    /**
     * Start a new BA with atomic outcome and the specified timeout as
     * its lifetime.
     * If one is already associated with this thread then the
     * WrongStateException will be thrown.
     */
    public abstract void begin(final int timeout)
        throws WrongStateException, SystemException;

    /**
     * The BA is normally terminated by the close method. This signals to
     * all registered participants that the BA has ended and no compensation
     * is required.
     */
    public abstract void close()
        throws TransactionRolledBackException, UnknownTransactionException, SystemException;

    /**
     * If the BA must undo its work then the cancel method is used. Any
     * participants that can compensate are forced to do so.
     */
    public abstract void cancel()
        throws UnknownTransactionException, SystemException;

    /**
     * If participants have registered for the BusinessAgreementWithComplete
     * protocol then they will be expecting the application to inform them
     * when all work intended for them has been sent (and responded to). The
     * complete method is used for this purpose.
     */
    public abstract void complete()
        throws UnknownTransactionException, SystemException;

    public abstract String transactionIdentifier();
}

Here’s an example of using the activity service:

UserBusinessActivity businessActivity = UserBusinessActivityFactory.userBusinessActivity();
businessActivity.begin();
try {
   // do some work
   businessActivity.close();
} catch (Exception ex) {
   businessActivity.cancel();
}

There is of course some initialization to do. I’ll leave that to the documentation, but overall its a very simple API. What I like about this way of doing compensations is that the client is decoupled from the knowledge of whether something should be compensated for, and how it is compensated. The knowledge of this is encapsulated in the services the client interacts with. Services automatically register themselves for the appropriate callbacks. User code does not have to perform commits or rollbacks. Very similar in ease of use to what we’re used to with JCA combined with JTA transactions.

I really think this stuff would be even more useful in plain web applications, not just coordinated remote web services. In my previous blog, Andy Oliver commented how Hibernate application transactions with FlushMode NEVER isn’t always a great solution for long running web apps that interact with your browser. This new framework could provide a very simple mechanism for registering compensatory callbacks so that browser interactions with a database don’t have to be held in memory.

In my next blog on this subject, I want to dive into some ideas I have for improving this framework. Until then, have fun.

Newer Entries