Possible iteration on avoiding duplicates

1 Comment

Draft 5 of REST-* messaging talked about iterating on the reliable posting protocol.  Recently, I was arguing with Jan Algermissen on a completely unrelated subject.  As a result of this conversation, I ended up re-reading Roy’s post on hypermedia for the 4th time in 2 years. Re-reading Roy’s blog got me thinking a bit about improving the message posting protocol of REST-* Messaging so that it is driven more by in-band information, rather than out-of-band information.

Firstly, I want to remove the post-message-once and post-message link relationships.  Instead, a destination resource would only publish the create-next link.  When a client wants to post a message to a queue or topic, it will use this link to create a new message resource on the server.    The type of this link would be “*/*” meaning it accepts any media type.

The key change to the protocol would be for the client to be aware that responses to creating messages through a create-next link may contain a new create-next links.  The client is encouraged to use these new, on-the-fly, create-next links to post additional messages to the topic or queue.  An important point of this change is that the server is not required to send a create-next link with its responses.  How, and if, the server protects itself from duplicate message posting is up to the server.

So how could the server protect itself from duplicate message posting?  One implementation could be that the server returns a 307 response code, “Temporary Redirect” for the initial POST to the static top-level create-next link published by the destination resource.  This 307 requires the client to re-POST the request to a URL contained in a response Location header as defined by the HTTP 1.1 specification.  The Location header would point to a one-off URL (like the previous protocol defined in Draft 5).    If a network failure happens, then the client re-POSTs to this  URL.  If the messages was previously successfully processed by the server, the server would respond with a 405, Method Not Allowed.  If no network failure happens on the re-POST from the 307 redirection, then the server would just return a success code.  In either response, the server would also return a new create-next link as a Link header within the response.  The client would use this new create-next link to post new messages.  Subsequent posts to these new links would not have to go through the re-direct protocol because they would be newly generated one-off URLs.

I have been reading a bit that some weird or undesirable behavior may be experience with some user agents when using 307 and POST/PUT.  So, I think that if the REST-* Messaging specification leaves it undefined how a server implementation handles the initial response of a duplicate-message protection protocol, we can let it evolve on its own.  The key here is that the client should be encouraged to look within the response for new create-next links even from error responses.  For example, if instead of 307, the initial POST return a 415, Preconditions Failed and that error response contained a create-next link header, the client should use that link to re-post the request.  NOTE!  I think 307 is probably the best implementation, but maybe its best to give flexibility to implementors.

Keep the *-batch links

I still want to have separate links for submitting batches of messages.  Specifically rename post-batch to create-next-batch (and remove post-batch-once).  I want the distinction so that the server knows that it is receiving a collection of messages vs. the server just forwarding a message to message consumers that just happens to be a collection media type.

Links instead of PATCH

28 Comments

Here’s some random thoughts I’ve had about REST while interacting with commenters on my previous blog.  Please, please, they are just thoughts, not rules of thumb I’m taking on.  Some things to think about and ponder…

Use Links instead of PATCH

The idea of using the new PATCH HTTP operation makes me squeamish.  I’m not sure why, but intuitively I don’t like it.  Maybe its because so many semantics can be hidden behind it?  Maybe its because a user is going to have to read a lot of documentation to understand how to interact with it?  I think I’d rather use links instead.  Let me elaborate.  Consider this customer XML document:

<customer>
   <first-name>Bill</first-name>
   <last-naem>Burke</last-name>
   <address>
       <link rel="edit" href="/customers/333/address" type="application/xml"/>
       <street>...</street>
       ...
   </address>
   <billing-address>
       <link rel="edit" href="/customers/333/billing-address" type="application/xml"/>
       ...
   </billing-address>
...
</customer>

If you were using links instead of PATCH, doing a GET on a specific customer resource returns you a document that pretty much describes to you how to interact with it.  The “edit” link elements under address and billing-address let you know that you can partially edit the document without having to refer to any user documentation (or yuck, and WADL document) on whether PATCH is supported or how you use PATCH on the resource.

RESTEasy 2.0-beta-2 released

1 Comment

I don’t usually blog about beta or RC releases, but people have had a few problems with Apache Client 4.0 integration with RESTEasy, specifically, a bunch of connection cleanup bugs.  I have fixes those bugs reported for this.  Also, this release ran successfully against the JAX-RS 1.1 TCK.  I had to make a bunch of encoding fixes here.

You can download from the usual places.  Go to our home page for more info.

Modeling operations in REST

23 Comments

A Red Hat Colleague of mine, Bryan Kearney,  recently solicited my advice about a RESTful interface his team was creating for an entitlement system being used for our products.  Here’s what he wrote to me:

The project is called Candlepin, and it is an entitlement engine. You can see it at https://fedorahosted.org/candlepin/ and the code at http://git.fedorahosted.org/git/candlepin.git/. We are using RESTEasy as the engine, and I would say our API is at gen 2. We are no longer doing too much RPC style calls.. but are not yet doing a HATEOAS API.

Our current quandry is around how to model state trnasitions on resources. We have seen a couple of possible approaches, and none leaps out as a best practice yet.

Asssume I have a Consumer who has entitlements. The entitlement resource is really the relationship between the consumer and a product they have purchases. Today, we model this as:

GET /consumers (Gets all)
GET /consumers/{UUID} (Gets a Consumer)
GET /consumers/UID/entitlements (Gets the entitlements for a consumer)
GET /entitlements?consumer={UUID} (Gets the entitlements for a consumer)

if we terminate a consumer, we want to terminate all his entitlements. We also think that we will not want to delete objects from the DB, and instead just filter out the terminated.

Approach 1
———-
First approach is to PUT /consumers/{UUID} the full consumer object with a new state. The method would then look for state changes and do the needful. This seems the most Hateoas-ey

Approach 2
———-
Since we know we wont want to delete objects, we could hijack

DEL /consumers/{uuid}

And not really delete, just terminate. This seems kinda hackey.

Approach 3
———–
Approach 3 would create “action” links. We would therefore support

POST /consumers/{uuid}/terminate

and do the termination logic there. This would mean that the POST to /consumers/{uuid} would be pure object update only. This seems very RPCish

Approach 4
———–
The final one would be to support both

/consumers/
and
/inactiveconsumers/

So.. a

DEL /consumers/{uuid}

would allow me to now do a

GET /inactiveconsumers/{uuid}

We had a bit of an email exchange, but here’s the advice I gave him summarized.

Have your clients consume links, not a URI scheme

Modelling URI schemes is an implementation detail.  You need to do it when designing your application and then finally implementing it because the URI scheme of your restful web services has a huge affect on how they work.

BUT, URIs are an implementation detail of your restful web services.  URIs should be almost completely opaque to your clients.  Instead only one top-level resource’s URI should be published to the outside world.  Representations of this root URI should have links (or link headers) embedded within them to publish other entry points into your suite of web services.  Why is this important?

  1. Location Transparency.  Lets give an analogy.  If you were writing a CORBA, Java RMI, or SOAP based application, would you publish each and every endpoint to the client?  No, you would not.  You would use a Naming service or UDDI to register a logical name for your endpoints.  A client would go to this naming service to obtain the location of your object/service.  Another analogy is, how do you interact with your browser?  You read an HTML page and click on a word that is underlined as a link to go to a new document.  You don’t cut and paste the actual URL in your browser window to go to another page.  The same thing should happen with your web services.
  2. Clients become immune to URI scheme refactorings. Over time as your services evolve, you may want to change how the URI schemes are modeled and defined.  If your clients are using a harded coded URI scheme and you change it, you’ve broken all clients that are using the older scheme.  Links protect clients from refactorings and allow server developers the freedom to refactor to their hearts desire.

Avoid changing the meaning of an HTTP verb: specifically DELETE

I would like to preface this with I used to think the approach Bryan preferred was the best approach.  His preferred approach was to have DELETE change the state of a consumer to make it inactive.  IMO, this is changing the meaning of delete.  Instead of removing a resource, which is what DELETE was designed to do, DELETE (in his example) is actually changing the state of a resource.  This is bad, IMO.  So what approach did I recommend from his list?  Approach 3.

Use links to model operations (Approach 3)

I prefer Approach3.  While it smelled RPCish to Bryan, IMO, it is not, if you model it correctly using link relationships.  For example:

GET /consumers/{UUID}
would return.

<consumer id="333">
   <name>Bill</name>
   ...
   <atom:link rel="terminate" href="/consumers/333/terminate"/>
</consumer>

The href here is *IRRELEVANT*.  URI’s should be opaque to the client (just as they are opaque when a human surfs the web through a browser).

Think of a link almost as a form.  When you get the representation of the consumer, the atom:link elements act like forms.  They tell client of possible state transitions, what representational information is expected, and what URL to post the representation too.

Having terminate as a link gives you a lot of flexibility.  1st iteration of the link might be an empty POST.  2nd iteration you might want simple form parameters to specify the parameters of the termination.   3rd iteration you might define a actual media type for termination.  Because termination is now a resource you can store this type of information there and link it within your consumer representation.

Another interesting thing that can be done is to publish (or not publish) different links based on the state of the resource.  For example, if a consumer in Bryan’s service has been terminated, a GET of that consumer resource might publish a “reinstate” link and remove the “terminate” link.

But isn’t this RPCish? No, it is not.  Why?  Well, think how you would implement something like this through an HTML/browser-based UI.  How would you model terminate in this situation?  From the consumer HTML page you would either have a link that brought you to a page with a HTML Form you had to fill out to terminate the consumer, or, the consumer HTML page would have a mini form that terminated the consumer.  In each case, the form would point to a specific URI that handled the state transition.

There’s some other interesting aspects that came out of my dialogue with Bryan, but I’ll save those for another blog.  Thoughts?  Do you think it is ok to model operations as links?


Webinar on REST, RESTEasy, JBoss – March 23rd

Leave a comment

I’m doing a webinar tomorrow on REST, JAX-RS, RESTEasy, and REST-*.  I only have 40 minutes, so it will be a brief overview of all those subjects and how they fit into our EAP product.  I’ll be giving it twice:

9am – EST

2pm – EST

For more information, click here

REST-* Messaging Draft 5: new post and subscribe patterns

1 Comment

I’ve made some small changes to REST-* Message Draft 5.  First is to the reliable posting of messages to a message destination.  The second is to the push model default subscription creation method.

New post-message-once protocol

Previously, the post-message-once link used the POE pattern to avoid duplicate message posting.  I asked around and it seems that the POE pattern isn’t used a lot in practice.  I’m glad because it kinda breaks the uniform interface (unsafe GET) and isn’t really consistent with the other protocols I defined.  It is also very inefficient as you have to make two round trips to post each message.  Nathan Winder, on the reststar-messaging list suggested using a one-off link generated with each message post.  Here’s how it looks:

The post-message-once link URL provided by this link is not used to actually create a message, but rather to obtain a new, one-off, URL. An empty POST should be executed on the post-message-once link. The response provides a new “create-next” link which the client can then post their message to. The link is a “one-off” URL. What that means is that is that if the client re-posts the message to the create-next URL it will receive a 405 error response if the message has already successfully been posted to that URL. If the client receives a successful response or a 405 response, there should be a Link header returned containing a new “create-next” link that the client can post new messages to. Continuously providing a “create-next” link allows the client to avoid making two round-trip requests each and every time it wants to post a message to the destination resource. It is up to the server on whether the create-next URL is a permanent URL for the created message. If it is not permanent, the server should return a Content-Location header to the message.

post-message-once example

  1. Query the destination root resource for links.Request:
    HEAD /topics/mytopic HTTP/1.1
    Host: example.com
    

    Response:

    HTTP/1.1 200 OK
    Link: <...>; rel="post-message",
          <...>; rel="post-batch",
          <http://example.com/topics/mytopic/messages>; rel="post-message-once",
          <...>; rel="message-factory"
    
  2. Client performs a POST request to the post-message-once Link to obtain a create-next link.Request:
    POST /topics/mytopic/messages
    Host: example.com
    

    Response:

    HTTP/1.1 200 OK
    Link: <http://example.com/topics/mytopic/messages/111>; rel="create-next"
    
  3. Client POSTs message to create-next LinkRequest:
    POST /topics/mytopic/messages/111
    Host: example.com
    Content-Type: application/json
    
    {'something' : 'abitrary'}
    

    Response:

    HTTP/1.1 200 Ok
    Link: <http://example.com/topics/mytopic/messages/112>; rel="create-next"
    

Change to push model subscription

I also added a minor change to the push model’s subscriber registration protocol.  In the previous version of the spec, the client would post form parameters to a subscribers URL on the server.  The form parameter would define a URL to forward messages to and whether or not to use the POE protocol to post this message.  I changed this to simple require the client to post an Atom Link.  Since links define protocol semantics, the server can look at the link relationship registered to know how to interact with the subscriber when forwarding messages.  So, if the client registers a post-message-once link when it creates its subscription, the server knows how to interact with the link.  This gives the client and server a lot of simple flexibility in describing how messages should be forwarded.  For example:

This example shows the creation of a subscription and the receiving of a message by the subscriber.

  1. Client consumer queries topic resource for subscribers link.
    Request:

    HEAD /mytopic
    Host: example.com
    

    Response:

    HTTP/1.1 200 OK
    Link: <http://example.com/mytopic/subscribers, rel=subscribers, type=application/atom+xml
          ...
    
  2. Client does POST-Created-Location pattern to create subscriber
    Request:

    POST /mytopic/subscribers
    Host: example.com
    Content-Type: applicatin/atom+xml
    
    <atom:link rel="post-message-once" href="http://foo.com/messages"/>
    

    Response:

    HTTP/1.1 201 Created
    Location: /mytopic/subscribers/333
    
  3. A message comes in, the message service does a POST to this subscriber based on the interaction pattern described for post-message-once
    Request:

    POST /messages
    Host: foo.com
    

    Response:

    HTTP/1.1 200 OK
    Link: <http://foo.com/messages/624>; rel=create-next
    

    Request:

    POST /messages/624
    Host: foo.com
    Link: <http://example.com/mytopic/messages/111>; rel=self,
          <http://example.com/mytopic>; rel=generator
    Content-Type: whatever
    
    body whateve

Mapping response on client side

8 Comments

I’ve been prototyping a bit lately for the REST-* effort, specifically for BPM.  I rely heavily on Link headers to pass links around.  RESTEasy has become pretty decent at handling links on the client side.  Here’s an example of client request/responses and link following via link headers:

      InputStream jpdl = Thread.currentThread().getContextClassLoader().getResourceAsStream(file);
      ApacheHttpClientExecutor executor = new ApacheHttpClientExecutor();

      ClientRequest request = executor.createRequest("http://localhost:8081/bpm/definitions");
      request.body(mediaType, jpdl);
      Link definition = request.create();
      Assert.assertNotNull(definition);
      ClientResponse response = null;

      response = definition.request().head();
      Assert.assertEquals(200, response.getStatus());
      Link instanceFactory = response.getLinkHeader().getLinkByTitle("instances");

      MultipartFormDataOutput form = new MultipartFormDataOutput();
      form.addFormData("order", "$199.99", MediaType.APPLICATION_XML_TYPE);
      response = instanceFactory.request()
              .body(MediaType.MULTIPART_FORM_DATA_TYPE, form)
              .post();
      Assert.assertEquals(201, response.getStatus());
      System.out.println(response.getLinkHeader().toString());
      Link instance = response.getLocation();
      Assert.assertNotNull(instance);

      Link next = response.getLinkHeader().getLinkByTitle("continue");
      Assert.assertNotNull(next);

      Link variables = response.getLinkHeader().getLinkByTitle("variables");
      Link newVariables = response.getLinkHeader().getLinkByTitle("variable-template");

      response = variables.request().head();
      Assert.assertEquals(200, response.getStatus());
      System.out.println(response.getLinkHeader().toString());
      Link order = response.getLinkHeader().getLinkByTitle("order");
      String xml = order.request().getTarget(String.class);
      request = newVariables.request();
      response = request.pathParameter("var", "customer")
             .body(MediaType.APPLICATION_XML_TYPE, "bill")
              .put();
      Assert.assertEquals(201, response.getStatus());
      response = request.pathParameter("var", "customer")
             .body(MediaType.APPLICATION_XML_TYPE, "bill burke")
              .put();
      Assert.assertEquals(204, response.getStatus());

      response = next.request().post();
      Assert.assertEquals(204, response.getStatus());

The thing about this code, it is a little hard to read.  Because HTTP is being treated like a messaging API, the code is a conglomeration of simple API calls.  The RESTEasy client proxy framework provides a nice way to map Java method calls to HTTP requests.  It also allows you to map automatically, a response body to a Java object.  Unfortunately though, this doesn’t work that well for my usecases.  Because I’m relying a lot on Link headers to pass information around in REST-*, I need something that can represent an HTTP response as a whole in a nice way.

A POJO Response Mapping

So, I thought, why not define (or reuse JAX-RS annotations) to map an HTTP response to a Java POJO?  It would kind of be the opposite of the RESTEasy @Form feature (where form maps an incoming request to a POJO).  It could look something like this:

@ResponseMapping
@ExpectedCode(200)
public interface MyResponse {

   @Body
   public Customer getCustomer();

   @LinkHeader
   public Link getNext();

   @LinkHeader("last")
   public Link getLastCustomer();

   @Header("ETag")
   public String getHash();

}

In this example, the client code would be expecting a response code of 200 (OK), a message body converted to a Customer object, a Link header named “next”, and a HTTP response header “ETag”. Using the RESTEasy proxy framework, you could then return this as a method return value, i.e.

@Path("/customers")
public interface CustomerClient {

   @Path("{id}")
   @Produces("application/xml")
   public MyResponse getCustomer(@PathParam("id") int custId);
}

What about errors?

For responses where the response code does not match, you could define similar mappings on an exception class.

@ResponseMapping
@ExpectedCode(404)
public class NotFoundException extends RuntimeException {}

You’d integrate it with the RESTEasy proxy framework by putting it within the throws clause.

@Path("/customers")
public interface CustomerClient {

   @Path("{id}")
   @Produces("application/xml")
   public MyResponse getCustomer(@PathParam("id") int custId) throws NotFoundException;
}

What do you think?
Maybe I’m going a little overboard here. Maybe not? I don’t know. Let me know what you think.

Speaking at new Boston JBoss User Group 2/9

Leave a comment

Jesper Pederson has created a Boston JBoss User Group.  Our first meeting is next Tuesday, February 9th.  I’m the first speaker and will be giving an intro to REST, JAX-RS, and, if I have time, some of the stuff that we’re doing at REST-* (rest-star.org).  Please click here for more details.

Polling styles show difference between Coakley and Brown

2 Comments

I don’t know if you’ve been following Massachusetts politics, but there is a run-off election for the deceased senator Edward (Ted) Kennedy between Martha Coakley and Scott Brown.  I’ve been telephoned twice, once by each candidate’s supporters.  I think the difference in how these polls were conducted tell you a lot about the candidates.

Scott Brown’s poll:

It was automated and voice-recognition driven.   A long series of questions were asked before the final “Who will you vote for?” question.  Questions like “Does abortion affect who you will vote for?”, “Do you support a health bill that will add trillions in debt to our country?” “Do you believe in fiscal responsibility for our govt?”.  Very slick…

Marth Coakley’s call:

It was a person who first states “I’m from the Martha Coakley campaign”.  She asked if I knew there was an election on Tuesday and if I was going to vote.  She asked if I knew where the poll was.  She asked if any voting members of my family needed a ride to the poll and gave me a number to call if I didn’t have a ride.  That was it.  There was no “Who will you vote for?”  Nothing else.

If you’re a MA resident, I hope you vote on Tuesday.  Its a pretty important election no matter what side of the fence you sit on.

dm to Eclipse: Does it really matter?

14 Comments

I was just reading on Adrian Coyler’s blog that Spring’s dm Server is moving to Eclipse.org.  Cutting through all the blah blah blah, this is what I came up with as the reason for this move:

“At SpringSource we know that open source development and community involvement can play a huge role in evolving simple, pragmatic solutions that enable a technology to bridge from early adopter to mainstream usage. We know because it is a path we have successfully taken many times. In creating the Virgo project at Eclipse.org, we seek to accelerate the journey of the dm Server and of enterprise OSGi along this path.”

What I extrapolate from this is that they want to accelerate adoption.  I have a simple fundamental question:

Does it really matter if you host your project at an OSS organization like Eclipse.org or Apache?

I’ve been living in the Java OSS community for about 9 years now.  IMO, based on my experiences with JBoss, the only thing that Eclipse.org or Apache really gives you is an established brand to promote your project.  I’ve always felt that a move to Apache or Eclipse, while may be beneficial in the short run with a bump in your adoption curve, in the long run it is bad for both your project and ultimately your business.  Why?  Because you lose control of both your brand and the governance of your project.  Brands cost money and time to build.  Governance introduces the inefficiencies of any bureaucracy (see my previous blogs on problems at Apache.org).  So, with both of those disadvantages, I question the advantages of moving dmServer to Eclipse.org.  JBoss has never had a problem starting a new project and building new communities on our own.  Neither has Spring for that matter.  Our brands are strong enough so that we don’t need the bump of an Apache or Eclipse to drive adoption.

The only real benefit I see to move to Eclipse or Apache is if you’re trying to build a consortium of companies that build off of a base core technology.  I really don’t see that happening with dmServer, even if that is there goal.  I know we wouldn’t use the technology.  Would Oracle?  IBM?  SpringSource is no longer the little guy.  VMWare is a serious competitor to all of us.

Standardization more important

IMO, a better route to grow adoption is standardization.  Hibernate got a huge bump in adoption by aligning itself under the JPA banner and EE 5.  We are seeing the same with Seam->CDI->Weld and Bean Validation.  Standardization drives adoption because it frees competing companies from having to depend on a competitor’s code base.  Since the APIs are public, vendors can provide their own implementations and value add on their own terms.  At the same time, standardization gives an easier entry point for vendors that want to enter into the space and gives users the peace of mind that they have less vendor lock-in.  Just because something is open source doesn’t mean that it is immune to lock-in.  The lock-in here is to the implementation.  Standardization breeds multiple implementations.

With dmServer, yeah, it is built upon the OSGi specification (and OSGi implementations), but that is just not the same.  You have to bring your innovations to a standards body to create the ecosystem.

What do you think?

So please help me optimize my personal algorithm for professional open source.  Do you think it really matter if you bring your project to an established organization if you yourself are already established?  I say no…


Older Entries Newer Entries