How to get a list of Extensions in WebComposition/DGS

If you are not sure if a DGS instance provides a certain extension, just have a look at the metadata of the service, e.g. http://www.restful.ws/datagridservice/meta. You will find the all the available extensions with the predicate http://www.webcomposition.net/2008″02/dgs/meta/extension (we say “has extension”). In the example below we find the SA-REST extension I have introduced yesterday.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:ns="http://purl.org/dc/elements/1.1/"
         xmlns:crud="http://www.webcomposition.net/2008/02/dgs/crud#"
         xmlns:meta="http://www.webcomposition.net/2008/02/dgs/meta/"
         xmlns:sarest="http://lsdis.cs.uga.edu/SAREST#">
  <rdf:Description rdf:about="http://www.restful.ws/DataGridService">
    <!-- ... -->
    <meta:extension>WebComposition.Dgs.Extensions.SaRest.SaRestHandler</meta:extension>
    <!-- ... -->
  </rdf:Description>
<rdf:RDF>

WebComposition/DGS Extensions and SA-REST Support

The last few evenings I spend writing an extension concept for the WebComposition/DGS approach. Initially, I was looking for a semantic description for the RESTful interface of the DGS. There are plenty approaches and a lot of research is going on for extending the WSDL interfaces with semantics. OWL-S and WSMO might be the most important examples within this field. However, it appears to be a bit more tricky for REST-driven approaches. For the DGS, I decided to try the SA-REST approach of the Kno.e.sis Services Science Lab of the Wright State University, Ohio.

<rdf:Description rdf:about="http://localhost/datagridservice/DataGridService">
    <SAREST:input rdf:resource="http://lsdis.cs.uga.edu/ont.owl#Location_Query" />
    <SAREST:output rdf:resource="http://lsdis.cs.uga.edu/ont.owl#Location" />
    <SAREST:action>HTTP GET</SAREST:action>
    <SAREST:lifting rdf:resource="http://www.restful.ws/lifting.xsl" />
    <SAREST:lowering rdf:resource="http://www.restful.ws/lowering.xsl" />
    <SAREST:operation rdf:resource="http://lsdis.cs.uga.edu/ont.owl#Location_Search" />
</rdf:Description>

However, I did not want to make this as a internal component of the DGS as is a research project, and no recommendation or standard is out there yet. So I came along with the extension concept of the DGS that allows dynamically loading of additional extensions in addition to the core components of the DGS.

The IExtension interface provides hotspots were own code can be executed right before and after one of the CRUD (Create, Read, Update, Delete) events.

namespace WebComposition.Dgs.Extension
{
    public interface IExtension
    {
        bool CreateStartUp(ref ExecutionContext context);
        bool CreateTearDown(ref ExecutionContext context);
        bool ReadStartUp(ref ExecutionContext context);
        //...
    }
 }

The ExecutionContext contains the request URI, the corresponding data adapter and (if already available) the current content to be returned with the response.

The extension can modify or create the content to be returned. In this case the return value must be set to false. Returning this dirty flag the DGS knows that the extension provided a new content to be returned. Any further internal functionality by the DGS is thus skipped and the newly provided content from the extension is returned.

To keep the WebComposition/DGS approach as flexible as possible, extensions can be deployed at runtime. The DGS is instructed to use extension by specifying them within the web.config file.

<webComposition>
    <dataGridService>
    <extensions>
        <extension type="WebComposition.Dgs.Extensions.SaRest.SaRestHandler,
                   WebComposition.Dgs.Extensions.SaRest,
                   Version=1.0.0.0,
                   Culture=neutral,
                   PublicKeyToken=null" />
      </extensions>
      ...
    </dataGridService>
</webComposition>

Based in this extension mechanism I created my very first extension to provide SA-REST support. If you have a closer look to SA-REST you might realize a drawback of the approach (one that is directly related to the REST principles). When using RDFa or any other microformat in HTML/XHTML the description of the service is rather static. The DGS on the other hand provides a highly flexible solution. Here, the full power of the DGS turn up. Analyzing the SA-REST annotation you will realize that you have RDF triples, e.g. metadata. This metadata can be stored within the DGS of course.

If we have a closer look on this metadata we can add this metadata by performing an updated on your service’s meta URI. In my case it is http://localhost/datagridservice/DataGridService/meta. Below we use some RDF based on the originally SA-REST example.

The SA-REST extension now extends the URI scope of the DGS – which is actually a very cool feature. Once deployed, the /meta scope is extended with /meta/sarest. If you now perform a GET request to http://localhost/datagridservice/DataGridService/meta/sarest. The extension will return the corresponding SA-REST metadata we used above.

To round up this exercise I’ve also created a set of XSL transformations that create XHTML to be used within any Web page. E.g. the SA-REST annotation mixed with the content, again based on the original SA-REST example, would look like below. Keep in mind, the XHTML snippet you see here was dynamically created by the DGS itself using a transformation.

<p about="http://localhost/datagridservice/DataGridService" xmlns="http://www.w3.org/1999/xhtml" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:sarest="http://lsdis.cs.uga.edu/SAREST#">
    The logical input of this service is an
    <span property="sarest:input">http://lsdis.cs.uga.edu/ont.owl#Location_Query</span>
    object. The logical output of this service is a list of
    <span property="sarest:output">http://lsdis.cs.uga.edu/ont.owl#Location</span>
    objects. This service should be invoked using an
    <span property="sarest:action">HTTP GET</span>
    request.
    <meta property="sarest:lifting" content="http://www.restful.ws/lifting.xsl" />
    <meta property="sarest:lowering" content="http://www.restful.ws/lowering.xsl" />
    <meta property="sarest:operation" content="http://lsdis.cs.uga.edu/ont.owl#Location_Search" />
</p>

The extension concept allows to add new custom components and to customize the DGS even more while the SA-REST extension provides a very first capability to describe the DGS’ RESTful interface in a semantic way.

I am going to do a second proof of concept using another approach describing RESTful services soon to show the flexibility of the extension concept.

WebComposition/DGS Core Architecture

Ok, how would you describe the core architecture of your current project? Inspired by Ralf’s idea, I really thought about the WebComposition/DGS architecture. What are the e core concepts and the key ideas of it? If everything has to fit on a napkin, you really have to restrict yourself in drawing different components. The interactive NapkinNotebook thus cam in very handy. Let’s look at the first try of the architecture sketch?

WebComposition/DGS Core Architecture

What are the core components we find in the WebComposition/DGS? Well, there is the Data Grid Service (DGS) as central entity? What else? Some storage solution for data and some other storage solution for metadata. Ok, here we have to think the first time? Two of them? Definitely yes. Metadata might be saved a completely different way than the data itself. You don’t think so? Let’s see, think about our images as digital photos and the metadata created from the EXIF data extracted from the imaged. The EXIF data is stored as RDF in some triple store, the images are stored on the file system. Consequently, we need two storage solutions.

OK, next component on the diagram: Users; do the interact directly with the DGS? No, usually the use a Web-based application. The user also might be another application or a service – but that’s no information to be on this sketch. The important thing is: there is usually only interaction through some other component – usually a Web-based application.
After some feedback from Ralf, I’ve updated my sketch using another software cell for the Web application. Makes much more sense though, what do you think?

RSS for CRUD Events

With the last update on the WebComposition/DGS, we now provide RSS feeds for CRUD events. Therefore, the Meta-URI /meta is extended by the additional path segments /meta/crud, /meta/crud and /meta/crud/rss. The Meta-URI /meta/crud can be extended to /meta/crud/create, /meta/crud/read, /meta/crud/update and /meta/crud/delete. Each URI points to a certain set of events. The event URIs in form of http://www.example.org/meta/crud/abab1c07-9262-4e6a-9f52-3dc497ef92f1 point directly to the RDF of the corresponding event.

CRUD Event RSS Feed

Following this HTTP-URI will lead directly to corresponding description of the events as seen below. Therefore, we take care of our carefully chosen URI concept within the WebComposition/DGS approach.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:meta="http://www.webcomposition.net/2008/02/dgs/meta/"
         xmlns:ns="http://purl.org/dc/elements/1.1/"
         xmlns:crud="http://www.webcomposition.net/2008/02/dgs/crud#">
    <rdf:Description rdf:about="http://www.foo.bar/DataGridService">
        <meta:event>
            <rdf:Description rdf:about="http://www.foo.bar/DataGridService/meta/crud/abab1c07-9262-4e6a-9f52-3dc497ef92f1">
                <ns:date>2008-05-15T19:20:13.7+02:00</ns:date>
                <ns:creator />
                <crud:read rdf:resource="http://www.foo.bar/DataGridService/foobar" />
            </rdf:Description>
        </meta:event>
    </rdf:Description>
</rdf:RDF>

A more visual depiction of the event can looks like below. Following the RDF data you are pointed to the resource that was affected by the event.

CRUD Events

<pre>

&lt;rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot;
xmlns:meta=&quot;http://www.webcomposition.net/2008/02/dgs/meta/&quot;
xmlns:ns=&quot;http://purl.org/dc/elements/1.1/&quot;
xmlns:crud=&quot;http://www.webcomposition.net/2008/02/dgs/crud#&quot;&gt;
&lt;rdf:Description rdf:about=&quot;http://www.foo.bar/DataGridService&quot;&gt;
&lt;meta:event&gt;
&lt;rdf:Description rdf:about=&quot;http://www.foo.bar/DataGridService/meta/crud/abab1c07-9262-4e6a-9f52-3dc497ef92f1&quot;&gt;
&lt;ns:date&gt;2008-05-15T19:20:13.7+02:00&lt;/ns:date&gt;
&lt;ns:creator /&gt;
&lt;crud:read rdf:resource=&quot;http://www.foo.bar/DataGridService/foobar&quot; /&gt;
&lt;/rdf:Description&gt;
&lt;/meta:event&gt;
&lt;/rdf:Description&gt;
&lt;/rdf:RDF&gt;

</pre>

WebComposition/DGS Update

After some discussion about the WebComposition/DGS configuration this week, we changed the usage of schema validation with the Web.config file. The validation scope is now not bound anymore to data adapters directly. Rather we introduced a default validation defined in the Web.config such as

 <webComposition> <dataGridService> <defaultValidationscope="None"/> ...

Valid values are ‘None’, ‘List’ and ‘Element’.This schema scope represents the default schema validation scope whenever an information store is created and data is added. If set to ‘None’ no validation is performed at all. Using ‘Element’ the added XML is validated against the schema (a-priori validation) before the element is added to the information store while the usage of ‘List’ causes the data to be validated against the schema after being added to the information store (a posteriori validation).

The settings can be easily overwritten by using RDF meta data e.g. when using the N3 filter on a information store http://www.foo.bar/myStore:

<http://www.foo.bar/myStore>
http://www.webcomposition.net/2008/02/dgs/meta/validation
"Element".

That way, the default value can be always adapted to the specific need for a single information store.

Distributed Version Control at the DSS Evaluation Phase

This post will give some insight into the distributed version control ad the Distributed and Self-organizing Systems Group. Usually, I am keen about trying out new things, e.g. distributed version control systems such as GIT, Mercurial or Bazaar. However, currently the group’s infrastructure is set up with Subversion as internal revision control system.

For the next few months we will start an evaluation phase with the students, working on the WebComposition/DGS approach, based on a distributed version control concept. Core components of the WebComposition project are branched into a dedicated student branch where all students can commit changes. Based on this branch each student creates his or her private branch for local changes.

The coordinator (usually some DSS staff) takes care of regular forward integration of the core components into the student branch. Each student in turn is responsible for forward integration into his or her own branch.

Changes can be submitted after testing and evaluation back into the central student branch by each student performing backward integration. The group working on the project has to agree in the LAKS meeting on the change before backward integrating the updates.

The coordinator finally takes care if these changes are also backward integrated into the original project trunk.

Distributed Version Control at the DSS

Step 1: Create a local working copy

First, look for a place where to checkout the working copy. Create a folder, e.g. branchesstudents and use Tortoise to checkout a working copy.

Initial Checkout

Select the student branch and the local folder where to store the working copy to start the checkout.

Branch

After the checkout is finished, you should will have a local working copy of the student branch.

Checkout Complete

Step 2. Create a private branch

Select ‘Branch/tag’ from the context menu of your working copy.

Branch

Use the path of your private project within the repository to create the branch.

Copy (Branch/Tag)

Step 3: Forward integrate changes from the student branch

When the student branch was updated you might be interested in getting these updates into your private branch. To forward integrate these changes into your private branch, select the folder where your local working copy lives, and use the Tortoise context menu to select ‘Merge…’.

Merge Branches

Select the repository folder you want to forward integrate Probably this will be the student branch – the paths in the screenshot below will thus differ from yours. Select the revision from where the forward integration should start. Now select the revision where it should end. Usually, this will be the head revision. After merging submit the changes of your working copy to the repository and you are done.

Merge

To get some more information on Subversion, or to learn how to perform the steps above on command line, refer the Subversion book.

Working on the WebComposition/DGS Documentation

These days I won’t have much time for blogging since I am working on the refresh for the WebComposition/DGS web site. The final site is not online, yet. However, I can give you a first glance how the new site will look like. Stay tuned.

WebComposition Refresh

The documentation will cover the most vital design aspects of the WebComposition/DGS approach as well as all necessary information to write your own extension for the service. I am working hard to make the live demos available as soon as possible as part of the site.