A RESTful Core for Web-like Application Flexibility

Sky-Tiger

Conclusion
This article continued the discussion about building application software using the Resource Oriented Computing approach; a flexible, Web-like architecture based on RESTful principles. In our discussion, we moved upwards from the physical code level to examine those capabilities which lie at the boundary between the physical and logical levels of the architecture. We saw how ROC features, such as logical URI addressing, logical address spaces, address mappings, transports, and modules support clean and flexible application design. Hopefully, the reader is beginning to see how one might utilize these ROC features to program at the logical level within a resource-oriented computing environment.

A subsequent article will explore design and architectural patterns that become available at the logical level. Just as patterns of object relationships and interactions were important for object-oriented programming and design, logical level patterns allow one to think and reason about system design in a powerful and concise way.

Sky-Tiger

Introduction
In the first three articles of this series [1] [2] [3], we described an approach to application software architecture that is based on a core set of "RESTful" principles. This Resource Oriented Computing (ROC) approach is motivated by the desire to see if the flexibility found in the World Wide Web can be incorporated into application software.

Previous articles examined the physical level of such an architecture and the boundary between the logical and physical levels. This article describes programming at the logical level and discusses some logical level design patterns available for ROC application design.

Programming at the Logical level
Hello World
The classic "Hello World" program is useful to show the minimal structures required to create an application which responds to requests from a client. For the ROC implementation we describe, the following must be in place:

A transport to detect the external event that is the request for the "Hello World" resource,
The "Hello World" resource,
A module containing the "Hello World" resource,
Mapping definitions to route the transport's internal request to an endpoint that will return the resource representation.

These basic components have been introduced in our previous articles, so here we concentrate on explaining how the components work together to create the "Hello World" application.

Remember that a transport is an event detector that resides at the edge of the system. It is responsible for detecting and handling external events while hiding the details; including the transport protocol and all protocol-specific issues. We intentionally leave the transport unspecified for this example and presume that the chosen transport detects the external request event and issues an internal root request for the resource located at the logical address "resource:/helloworld".

A transport is typically implemented in its own module and imports one or more modules from applications which wish to receive requests via the transport. Each module within the system has a unique URI identifier [4] and this is used in the import statement. Module version numbers (min, max) can be specified in the import statement or, if unspecified, the latest available module version is imported.

   <import>
      <uri>urn:com:mycomp:demo:helloworld</uri>
      <version-min>2.0</version-min>
   </import>
The implications of using logical identifiers and module version numbers are significant. Because module relationships are defined with logical addresses, the same resolution process that locates a physical endpoint for a logical URI address is also used to resolve the relationships between modules. This means that modules can be added, updated, and rolled back while an application is running. Different versions of a module can also operate simultaneously within a single system without conflict.

For example, imagine that an existing module at version 1.0.1 exhibits a defect and an updated module (version 1.0.2) needs to be deployed. A module deployment manager could upload the module's updated JAR file and perform a "Hot Restart". This would allow the microkernel to dynamically load the new module into memory and then resolve all subsequent requests to it. If desired, the original version of the module could even continue to operate in parallel with the updated version. This would, however, require that requests to the original version specifically target the original module by including the desired version number (1.0.1).

In order to expose the "Hello World" resource, our "Hello World" module must export the address of the resource:

   <export>
      <uri>resource:/helloworld</uri>
   </export>
It must also map the logical resource address to a physical endpoint that returns the resource representation. It does this within its internal private address space as follows:

   <map>
      <match>resource:/helloworld</match>
      <class>com.mycomp.HelloWorldEndpoint</class>
   </map>
With this setup for the "Hello World" example, we are ready to follow the flow of a request/response cycle. When the external client request arrives at a transport, the transport detects the event and issues a root request for the resource with the address "resource:/helloworld". The microkernel attempts to resolve this request within the address space of the transport's module and finds the import statements identifying our "Hello World" module. Since the "Hello World" application module has been imported by the transport, the resolution process will check our module's export statements and, seeing a match, it enters the module to continue the resolution search. Inside our module, the microkernel finds a mapping to a physical endpoint. The microkernel will then schedule the endpoint processing on a worker thread. When the endpoint returns a representation, the microkernel will pass that representation back to the transport which, in turn, delivers it back to the external requesting client.

Sky-Tiger

Mashups
Instead of mapping the address "resource:/helloworld" directly to a physical endpoint (as we did above) we could map it to a templating micro-service; which could compose or "mashup" a representation:

   <map>
      <match>resource:/helloworld</match>
      <to>active:template-service+operand@resource:/template.html</to>
   </map>
The template micro-service requires a single argument: the logical address of a template resource. The template resource, in turn, can contain logical addresses for the resources to be composed into the final resource representation:

    <html>
       <body>
          <template:include uri="resource:/menu.xml"/>
          ...
          <template:include uri="resource:/helloworld.xml"/>
          ...
          <template:include uri="resource:/footer.xml"/>
       </body>
    </html>
As the template micro-service processes this template, it issues sub-requests back into the logical address space to obtain the constituent resource representations. Each sub-request is sent to the microkernel and will be resolved starting in the address space of the module providing the template service. If a particular resource has been requested before, then its representation is probably in the system cache, in which case the cached value is returned by the microkernel. In fact, the resource representation for the top level request ("resource:/helloworld") might already be stored in the cache, too. If so, the the templating micro-service will not be called at all (unless one of the dependent resources changes or expires).

Notice that a developer (or graphic designer) working on the template only needs to know the logical addresses of the resources needed. Knowledge of APIs, datatypes, objects, protocols, and physical locations is not needed as those details are hidden in the physical layer of the application.

Our "mashup" operation can also compose remote resources located at any addressable location on the Internet. With an endpoint implementing the http: scheme, we can issue requests for resources located across the Internet using the HTTP protocol. For example, simply change a URI to a familiar http: scheme URL:

   <html>
      <body>
         <template:include uri="resource:/menu.xml"/>
         ...
         <template:include uri="http://www.1060.org/upload/readme.xml"/>
         ...
         <template:include uri="resource:/footer.xml"/>
      </body>
   </html>
and the content for the second logical address will be retrieved from a remote web site [5]. Notice that the only difference is a change in the logical address of a resource. The templating service need not know that a resource is located remotely.

An even better application design would isolate the remote address information within a lower logical layer. To create this layering we use a set of mappings from one layer to the other. In our example, the following maps the original resource address to the remote logical address:

   <map>
      <match>resource:/helloworld.xml</match>
      <to>http://www.1060.org/upload/readme.xml</to>
   </map>
This kind of logical-to-logical mapping creates a layer in which mashup components have names that make sense to those creating the mashup application and allows the addresses of the resources to be relocated (local, remote, etc.) without impacting the mashup level.

Sky-Tiger

Layers and Mappers
Good systems design often relies on distinct layers to separate different conceptual levels and to segregate design and implementation details into (mostly) disjoint sets. The use of layers to maintain a "separation of concerns" is observed not only in well-designed applications but in many other complex systems; such as hardware chip designs and operating systems. Through the use of the ROC components we have discussed (modules, logical-to-logical mappings, micro-services, etc.), it is easy to create layered logical systems in a Resource Oriented Computing architecture.

With ROC, each layer in an application design can be given an address space and each of these address spaces can be defined and managed by a module. If a design calls for a layer A to reside above a layer B, then the module for layer A (the upper layer) will import the module for layer B (the lower layer). A logical-to-logical mapping will often be defined to connect their address spaces. The mapping will define the subset of requests within the upper layer which will be routed down into the lower layer for handling. A common design for ROC applications uses three layers; a presentation layer, a domain layer, and an information access and integration layer.

Sky-Tiger

A Layered Application
The forum at 1060 Research (http://www.1060.org/forum/) is an example of an ROC application which uses three layers. The top layer performs presentation "mashups" (composition); the middle layer understands domain concepts such as "topics", "groups", "users", and so on; and the lowest layer accesses a relational database to persist the forum content.

Let's trace one request through an application, such as the forum, to see how layering works [6]. The external request "http://www.1060.org/forum/topic/374/1" is a RESTful request for a specific topic. This request arrives at the HTTP transport and a root request is issued by the transport for the resource at address "resource:/forum/topic/374/1". Several applications are hosted at this web site so the forum's top level module uses an export statement to declare that all requests starting with "resource:/forum/" should be routed to it:

   <export>
      <uri>resource:/forum/.*</uri>
   </export>
The presentation layer of the forum application maps the request address to remove the "forum/" portion:

   <map>
      <match>resource:/forum/(.*)</match>
      <to>resource:/$1</to>
   </map>
The request is then mapped to a templating service with the resource address provided as the "operand" argument:

   <map>
      <match>(resource:/topic/.*)</match>
      <to>active:template-service+operand@$1</to>
   </map>
The templating service creates the HTML presentation frame and then issues a request for the resource whose address it received as its "operand" argument: "resource:/topic/374/1". While resolving this request, the resolution process enters the module containing the forum application layer. Within this middle layer, the request resolves to a micro-service whose job is to handle requests related to topics:

   <map>
      <match>(resource:/topic/.*)</match>
      <to>active:topic-service+operand@$1
   </map>
The topic micro-service parses the RESTful URI and then issues a request for a database access to retrieve the stored topic information. The database access request might look something like this [7]:

   active:sqlQuery+operand@resource:/topic-query+topic@374+entry@1
This request will eventually be resolved in the lowest layer of the application; implemented by the persistence module which returns the representation of the requested topic.

It is important to note that the forum application is designed and composed solely in the logical level using logical level constructs. No Java objects, data types, cache lookups, transport protocols, or database drivers are exposed at the logical level. They do exist within the ROC system, of course, but only within the physical level.

Sky-Tiger

Mapping Between Layers
Mapping between application layers can be handled by a micro-service located between address spaces. For example, the following logical-to-logical mapping definition

   <map>
      <match>(.*)</match>
      <to>active:mapper+operand@resource:/map-definitions+uri@$1</to>
   </map>
routes all requests to a "mapping" micro-service. To perform its mapping functions, the mapping service uses a resource (whose address is the "operand" argument) containing mapping definitions. The mapping micro-service will map the request's URI address (passed as the "uri" argument) to another URI address, as specified by the mapping definitions. Using a mapping micro-service, instead of the microkernel's Regular Expression based mappings, is advantagous for a number of reasons:

The routing of requests to the mapping service is simple to configure and understand; requiring only a single map rule,
The mapping micro-service can perform more specialized and varied kinds of transformations, and
The mapping definitions are contained in a logically addressed resource.

This last point has interesting implications for application design, as it means that the set of mapping definitions can be dynamically defined by application code. In this example, the logical address "resource:/map-definitions" could be mapped to an endpoint which returns a different set of mappings depending on the time-of-day, the mode of operation, the number of users, or any other desired criteria. And, again, since resources are cached, the system will run at full speed until the resource containing the mapping definitions changes.

Sky-Tiger

Patterns
In addition to layering, many interesting architectural patterns are available within the logical level.

A Pattern [8] is a named description of a set of related architectural ideas; a template for solving a problem which recurs in many different contexts. Patterns in object-oriented system design are a powerful means of succinctly describing an overall application design. Patterns are also evident in the logical level of ROC application designs. Some ROC patterns will be familiar to OO programmers and some are unique to the ROC architecture.

In this section we describe several patterns commonly found in ROC application designs: Mapper, GateKeeper, Toolbox and Golden Thread.

Mapper and GateKeeper
We have already seen the Mapper pattern, which uses a micro-service to map one address space into another. A variant of the Mapper pattern is the GateKeeper pattern which, instead of making changes to the URI address of the request, decides whether or not to admit the request into the second address space. For example, a security gatekeeper service can be used with the following map:

   <map>
      <match>(.*)</match>
      <to>active:gatekeeper+operand@resource:/policy.xml+uri@$1+credentials@...</to>
   </map>
The gatekeeper service works by either re-issuing the incoming request into the mapped address space ("admit") or throwing an exception ("reject") if the security policy is somehow violated (e.g. wrong time of day, credentials, etc.). In this example, credentials are passed as a resource referenced by the "credentials" argument.

Sky-Tiger

Toolbox
The Toolbox pattern is interesting because it uses only address space mappings. A toolbox combines micro-services, resources, and other logical-level capabilities into a single module and then exposes them to users as a "standard" set of tools. A toolbox module accomplishes this by importing the modules that contain the various tools and then exporting their capabilities in a single, combined address space.

A variation of the Toolbox pattern is the Filtered Toolbox. It has a more restrictive export statement, which may not "pass on" access to all of the tools and services it imports.

Golden Thread
Another pattern, the Golden Thread, takes advantage of the dependency cache managed by the microkernel. In normal operation, any resource representation can be cached and will remain in the cache as long as its dependencies remain valid (not changed and not expired). Normally, the dependency mechanism tracks physical level changes (such as a file change) and notifies the cache when it should re-examine dependent resources.

It is also possible to base dependency tracking on a logical virtual resource. The Golden Thread pattern takes advantage of this ability by creating named virtual resources and associating them with other logical resources. For example, one could create a virtual resource named goldenthread:customer-table and associate it with all resources created from database queries on a customer table. Then, when an update is made to the customer table, the update process simply invalidates the goldenthread:customer-table virtual resource. This will cause all cached resources associated with the goldenthread:customer-table resource to be flushed from the cache.

The Golden Thread pattern works well within a system where all database access occurs through an ROC system. However, in cases where other applications update the database, there is still an elegant solution based on the same pattern. In such a case, a trigger or stored procedure can issue a RESTful request (such as "http://www.mycomp.com/gt/customer-table") back into the system. This request would be mapped to an internal endpoint that would then invalidate the virtual resource in question ("goldenthread:customer-table" in this example).

The Golden Thread pattern can also work across multiple computers running ROC applications. Shared information, such as the time of day, could cause the invalidation of resources. Or, a high-speed, lightweight messaging protocol could explicitly coordinate cache information between multiple computers,

Sky-Tiger

Example ROC Applications
Resource oriented computing systems are not hypothetical. Many applications have been or are being built using ROC, including:

Commercial and Government applications:

DeltaXML built an Ajax-based demonstration of XML difference processing at http://deltaxml.com/free/sandbox/
The next generation of the permanent URL system (PURL) will be based on ROC http://purlz.org/ and http://zepheira.com/news/releases/20070802.html
An Enterprise Service Bus for a large university: http://www.infoq.com/articles/netkernel-casestudy
An embedded telecom system: http://1060research.com/customer ... chnology/index.html
Los Alamos Laboratory built their digital library using ROC for service discovery and resource fulfillment.
The 1060 Research suite of applications:
A Forum at http://www.1060.org/forum/
A Blog at http://www.1060.org/blogxter/
A Wiki (in beta testing)
A Bug Tracking System at http://www.1060.org/bugxter/
Summary
This article concludes the exploration of a RESTful core for application software development. We started at the lowest level (binding) and have ended at the logical level; discussing application programming, designs, and patterns. The authors hope that this series has been informative and that it will motivate discussion, debate, and exploration of this innovative approach to building application software.

It is still early in the life of the ROC approach but, so far, the feedback from users has been very positive. The results of real-world application development projects have shown that application software can be assembled quickly and exhibit the desirable economic properties found in the World Wide Web. We and the team at 1060 Research are looking forward to discovering what long-term benefits Resource Oriented Computing can provide for software system architecture, design, development, and maintenance.

Sky-Tiger

Learning More
As mentioned, the system we have been describing is not hypothetical. Research on RESTful systems, which lead to ROC, started within HP Labs in 1999. When HP decided to exit the middleware business in 2002, 1060 Research was formed to continue the research and development of ROC. NetKernel is the result of this work and is currently at release version 3.3.1.

If you are interested in learning more about ROC and NetKernel, you can download a working system (including documentation, an application server, scripting language support, development tools, and many functional modules) at http://www.1060.org/download/. Detailed documentation (including instructional videos) is available at http://www.1060research.com/. The NetKernel user discussion forums are available for review and participation at http://www.1060.org/forum/.

This year, the second annual NetKernel Architects' Weekend will be held September 25-27, 2008 at the University of Mary Washington in Fredericksburg, VA, USA. At this event, 1060 Research will disclose information about the next generation of ROC, under development for the last three years, to be released as NetKernel 4.0.

References
[1] A RESTful Core for Web-like Application Flexibility - Part 1
[2] A RESTful Core for Web-like Application Flexibility - Part 2
[3] A RESTful Core for Web-like Application Flexibility - Part 3
[4] A module's identifier should be globally unique; we recommend using a URN based on a reverse DNS name (such as urn:com:mycomroject1:workbench).
[5] The resource representation for this remote address might also be found in the local cache; as long as it is still valid, as specified by its HTTP headers.
[6] This trace of the request is through a prototypical forum application. The full details of the actual forum application are described in the 1060 Forum Guide, which includes details specific to NetKernel 3.3.
[7] To be accurate, the syntax of the active: URI scheme requires that its arguments are also URI schemes, so the database request would be written more like this: active:sqlQuery+operand@resource:/topic-query+topic@data:text/plain,374+entry@data:text/plain,1
[8] See the Wikipedia entry for design patterns, which originated from the concept of architectural patterns.