Today I've found another new language (working draft in fact). It's an XML Pipeline Language.

An experience shows a process of language invention is an essential part of computer industry from the very beginning, however...

I must confess I must be too reluctant to any new language: I was happy with C++, but then all these new languages like Delphi, Java, C#, and so many others started to appear. It's correct to say that there is no efficient universal language, however I think it's wrong to say that a domain specific language is required to solve a particular problem in a most efficient way.

And now a question to the point: why do you need a new language for describing operations to be performed on XML documents?

A project I'm currently working on, requires me to manipulate with a big number of documents. This includes accessing these documents with key() function.

I never thought this task poses any problem, until I've discovered that Saxon caches documents loaded using document() function to preserve their identities:

doc("foo.xml") is doc("foo.xml")

Saxon provides a saxon:discard-document() function to release documents from cache. The use case is like this:

You may see, that saxon:discard-document() is bound to a place where document is loaded. In my case this is inefficient, as my code repeatedly accesses documents from different places. To release loaded documents I need to collect them after main processing.

Other issue in Saxon is that, processor may keep document references through xsl:key, thus saxon:discard-document() provides no guaranty of documents to be garbage collected.

To deal with this, I've designed (Saxon specific) api to manage document pools:

The use case is:

Download document-pool.xslt to use this api.

I was already writing about the logical difference between tamplates and functions. This time I've realized another, technical one. It's related to lazy evaluation, permitted by language specification.

I was arguing as follows:

• suppose you define a function returning a sequence;
• this function at final step constructs document using xsl:result-document;
• caller invokes this function and uses only first item of sequence;
• lazy evaluation allows to xslt processor to calculate first item only, thus to avoid creation of output document altogether.

This conclusion looked ridiculous to me, as it means that I cannot reliably expect creation of documents built with xsl:result-document instruction.

To resolve the issue I've checked specification. Someone has already thought of this. This is what specification says:

As you can see, xsl:function is always evaluated in temporary output state, and cannot contain xsl:result-document, in contrast to xsl:template, which may be evaluated in final output state. This difference dictates the role of templates as a "top level functions" and functions as standalone algorithms.

You can find more on subject at "Lazy evaluation and predicted results".

In the era of parallel processing it's so natural to inscribe your favorite programming language in the league of "Multithreading supporter". I've seen such appeals before "Wide Finder in XSLT --> deriving new requirements for efficiency in XSLT processors."

I think both xslt and xquery are well fitted for parrallel processing in terms of type system. This is because of "immutable" nature (until recent additions) of the execution state, which prevents many race conditions. The only missing ingredients are indirect function call, and a couple of core functions to queue parallel tasks.

Suppose there is a type to encapsulate a function call (say function-id), and a function accepting a sequence and a function-id. This function calls function-id for each element of the sequence in a parallel way, and then combines a final result, as if it were implemented serially.

Pretty simple, isn't it?

<!--
  This function runs $id function for each item in a sequence.
    $items - items to process.
    $id - function id.
    Returns a sequece of results of calls to $id function.
-->
<xsl:function name="x:queue-tasks" as="items()*">
  <xsl:param name="items" as="item()*"/>
  <xsl:param name="id" as="x:function-id"/>

  <!-- The pseudo code. -->
  <xsl:sequence select="$items/call $id (.)"/>
</xsl:function>

For the last several weeks I was on my military duty. We were patrolling Israeli border near the Egypt. It was a completely different world, world of guns, Hummers, heat sensors...

There I've met my army friends. It was fun to listen stories they were telling. At some point I've started to realize that I'm growing older. Most of my friends are married and have two or three children.

It seems, this was the genuine world, and my own one is fictitious.

Does WebSphere MQ library for .NET support a connection pool? This is the question, which ask many .NET developers who deal with IBM WebSphere MQ and write multithread applications. The answer to this question unfortunately is NO… The .NET version supports only individual connection types.

I have compared two MQ libraries Java's and one for .NET, and I’ve found that most of the classes have the same declarations except one crucial for me difference. As opposed to .NET, the Java MQ library provides several classes implementing MQ connection pooling. There is nothing similar in .NET library.

There are few common workarounds for this annoying restriction. One of such workarounds (is recommended by IBM in their “MQ using .NET”) is to keep open one MQ connection per thread. Unfortunately such approach is not working for ASP.NET applications (including web services).

The good news is that starting from service pack 5 for MQ 5.3, and of course for MQ 6.xx they are supporting sharing MQ connections in blocked mode:

“The implementation of WebSphere MQ .NET ensures that, for a given connection (MQQueueManager object instance), all access to the target WebSphere MQ queue manager is synchronized. The default behavior is that a thread that wants to issue a call to a queue manager is blocked until all other calls in progress for that connection are complete.”

This allows creating an MQ connection (pay attention that MQQueueManager object is a wrapper for MQ connection) in one thread and exclusive use it in another thread without side-effects caused by multithreading.

Taking in account this feature, I’ve created a simple MQ connection pool. It’s ease in use. The main class MQPoolManager has only two static methods:

public static MQQueueManager Get(string QueueManagerName, string ChannelName, string ConnectionName);

and

public static void Release(ref MQQueueManager queueManager);

The method Get returns MQ queue manager (either existing from pool or newly created one), and Release returns it to the connection pool. Internally the logic of MQPoolManager tracks expired connections and do some finalizations, if need.

So, you may use one MQ connection pool per application domain without additional efforts and big changes in existing applications.

By the way, this approach has allowed us to optimize performance of MQ part considerably in one of ours projects.

Later on...

To clarify using of MQPoolManager I've decided to show here following code snippet:

  MQQueueManager queueManager = 
    MQPoolManager.Get(QueueManagerName, ChannelName, ConnectionName);

  try
  {
     // TODO: some work with MQ here
  }
  finally
  {
    MQPoolManager.Release(ref queueManager);
  }

  // at this point the queueManager is null

Yesterday's idea has inspired me as much as to create a prototype implementation of map and tuple in the xslt 2.0.

Definitely I wished these were a built-in types, and were considered as atomic values for purposes of comparasions and iteration. This way it were possible to create highly efficient grouping per several fields at once.

This pure implementation (xslt-tuple.zip) is rather scetchy, however it allows to feel what can be done with tuples and maps. I guess a good example may say more than many other words, so have a pleasure:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:f="http://www.nesterovsky-bros.com/xslt/functions"
  exclude-result-prefixes="xs f">

  <xsl:include href="tuple.xslt"/>
  <xsl:include href="map.xslt"/>

  <xsl:template match="/">
    <root>
      <xsl:variable name="tuples" as="item()*" select="
        f:tuple
        (
          for $i in 1 to 10 return
            f:tuple(1 to $i)
        )"/>

      <total-items>
        <xsl:sequence select="count($tuples)"/>
      </total-items>
      <tuples-size>
        <xsl:sequence select="f:tuple-size($tuples)"/>
      </tuples-size>
      <sums-per-tuples>
        <xsl:for-each select="1 to f:tuple-size($tuples)">
          <xsl:variable name="index" as="xs:integer" select="position()"/>

          <sum
            index="{$index}"
            value="{sum(f:tuple-items(f:tuple-item($tuples, $index)))}"/>
        </xsl:for-each>
      </sums-per-tuples>

      <xsl:variable name="cities" as="element()*">
        <city name="Jerusalem" country="Israel"/>
        <city name="London" country="Great Britain"/>
        <city name="Paris" country="France"/>
        <city name="New York" country="USA"/>
        <city name="Moscow" country="Russia"/>
        <city name="Tel Aviv" country="Israel"/>
        <city name="St. Petersburg" country="Russia"/>
      </xsl:variable>

      <xsl:variable name="map" as="item()*" select="
        f:map
        (
          for $city in $cities return
            ($city/string(@country), $city)
        )"/>

      <xsl:for-each select="f:map-keys($map)">
        <xsl:variable name="key" as="xs:string" select="."/>

        <country name="{$key}">
          <xsl:sequence select="f:map-value($map, $key)"/>
        </country>
      </xsl:for-each>
    </root>
  </xsl:template>

</xsl:stylesheet>

The type system of xslt 2.0 is not complete (see Sequence of sequences in xslt 2.0). You cannot perform manipulations over items as you could do. The reason is in the luck of set based constructs: xslt 2.0 supports sequences, but not associative maps of items.

If you think that xml can be used as a good approximation of a map, I shan't agree with you. Xml has an application in a very specific cases only. Maps I'm thinking of,  would allow associate items by reference, like sequences do.

This opens a perspective to create a state objects, to manage sequence of sequences, to create cyclic graphs of items, and so on. These maps are richer than what key() function provides right now, and allow to implement for-each-group in xquery.

Such maps can be modeled with several functions, however I would wish they were built in:

f:map($items as item()*) as item()
Returns a map from a sequence $items of pairs (key, value).

f:map-items($map as item()) as item()*
Returns a sequence of pairs (key, value) for a map $map.

f:map-keys($map as item()) as item()*
Returns a sequence of keys contained in a map $map.

f:map-values($map as item()) as item()*
Returns a sequence of values contained in a map $map.

f:map-value($map as item(), $key as item()) as item()*
Returns a sequence of values corresponding to a specified key $key contained a specified map $map.

The other thing I would add is items tuple. It's like a sequence, however a sequence of tuples is never transformed into single sequence, but stays as sequence of tuples.

Fortunately it's possible to implement such extension functions.

xslt 2.0 is a beautiful language and at the same time it allows constructs, which may trouble anyone.

Look at this valid stylesheet:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xsl:template match="/">
    <xsl:variable name="x" as="node()" select="."/>
    <xsl:variable name="x" as="xs:int" select="***"/>

    <xsl:sequence select="$x"/>
  </xsl:template>

</xsl:stylesheet>

Fun, isn't it?

I was thinking earlier about the difference between named tamplates and functions in xslt 2.0 and have not found satisfactory criterion for a decision of what to use in each case. I was not first one who has troubled with this, see stylesheet functions or named templates.

To feel easy I deliberately have decided to use functions whenever possible, avoid named tamplates completely, and use matching templates to apply logic depending on context (something like virtual function). I've forgot about the issue until yesterday. To realize the difference one should stop thinking of it, quite opposite she must start solving practical xslt tasks, and if there is any difference, except syntactic, it will manifest itself somehow.

To make things obvious to those whose programming roots are in a language like C++ I shall compare xsl:function with free standing (or static) C++ function, and named xsl:template with C++ member function. In C++ you can use both free standing and member functions interchangeably, however if there is only one argument (among others) whose state transition this function represents then it's preferrable to define it as a member function. The most important difference between these two type of functions is that a member function has hidden argument "this", and is able to access its private state.

Please, do not try to think I'm going to compare template context item in xslt 2.0 with "this" in C++, quite opposite I consider context item as a part of a state. I'm arguing however, of private state that can be passed through template call chain with tunnel parameters. Think of a call tunneling some state (like options, flags, values), and that state accessed several levels deep in call hierarchy, whenever one needs to. You cannot do it with xsl:function, you cannot pass all private state through the function call, you just do not know of it.

This way my answer to the tacit question is:

•  use xsl:function to perform independent unit of logic;
•  use named xsl:template when a functionality is achieved cooperatively, and when you will possibly need to share the state between different implementation blocks;

After thinking through this, I've noticed that such distinction does not exist in XQuery 1.0. There is no tunneling there.

In the xslt world there is no widely used custom to think of stylesheet members as of public and private in contrast to other programming languages like C++/java/c# where access modifiers are essential. The reason is in complexity of stylesheets: the less size of code - the easier to developer to keep all details in memory. Whenever xslt program grows you should modularize it to keep it manageable.

At the point where modules are introduced one starts thinking of public interface of module and its implementation details. This separation is especially important for the template matching as you won't probably want to match private template just because you've forgotten about some template in implementation of some module.

To make public or private member distinction you can introduce two namespaces in your stylesheet, like:

• xmlns:t="http://www.nesterovsky-bros.com/public" - for public members;

• xmlns:p="http://www.nesterovsky-bros.com/private/expression.xslt" - for private members.

For the private namespace you can use a unique name, e.g. stylesheet name as part of uri.

The following example is based on jxom. This stylesheet builds expression from expression tree. Public part consists only of t:get-expression function, other members are private:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:t="http://www.nesterovsky-bros.com/public"
  xmlns:p="http://www.nesterovsky-bros.com/private/expression.xslt"
  xmlns="http://www.nesterovsky-bros.com/download/jxom.zip"
  xpath-default-namespace="http://www.nesterovsky-bros.com/download/jxom.zip"
  exclude-result-prefixes="xs t p">

  <xsl:output method="text" indent="yes"/>

  <!-- Entry point. -->
  <xsl:template match="/">
    <xsl:variable name="expression" as="element()">
      <lt>
        _{<mul>
            <var name="b"/>
            <var name="b"/>
          </mul>
          <mul>
            <mul>
              <int>4</int>
              <var name="a"/>
            </mul>
            <var name="c"/>
          </mul>}
        <double>0</double>
      </lt>
    </xsl:variable>

    <xsl:value-of select="t:get-expression($expression)" separator=""/>
  </xsl:template>

  <!--
    Gets expression.
      $element - expression element.
      Returns expression tokens.
  -->
  <xsl:function name="t:get-expression" as="item()*">
    <xsl:param name="element" as="element()"/>

    <xsl:apply-templates mode="p:expression" select="$element"/>
  </xsl:function>

  <!--
    Gets binary expression.
      $element - assignment expression.
      $type - expression type.
      Returns expression token sequence.
  -->
  <xsl:function name="p:get-binary-expression" as="item()*">
    <xsl:param name="element" as="element()"/>
    <xsl:param name="type" as="xs:string"/>

    <xsl:sequence select="t:get-expression($element/*[1])"/>
    <xsl:sequence select="' '"/>
    <xsl:sequence select="$type"/>
    <xsl:sequence select="' '"/>
    <xsl:sequence select="t:get-expression($element/*[2])"/>
  </xsl:function>

  <!-- Mode "expression". Empty match. -->
  <xsl:template mode="p:expression" match="@*|node()">
    <xsl:sequence select="error(xs:QName('invalid-expression'), name())"/>
  </xsl:template>

  <!-- Mode "expression". or. -->
  <xsl:template mode="p:expression" match="or">
    <xsl:sequence select="p:get-binary-expression(., '||')"/>
  </xsl:template>

  <!-- Mode "expression". and. -->
  <xsl:template mode="p:expression" match="and">
    <xsl:sequence select="p:get-binary-expression(., '&&')"/>
  </xsl:template>

  <!-- Mode "expression". eq. -->
  <xsl:template mode="p:expression" match="eq">
    <xsl:sequence select="p:get-binary-expression(., '==')"/>
  </xsl:template>

  <!-- Mode "expression". ne. -->
  <xsl:template mode="p:expression" match="ne">
    <xsl:sequence select="p:get-binary-expression(., '!=')"/>
  </xsl:template>

  <!-- Mode "expression". le. -->
  <xsl:template mode="p:expression" match="le">
    <xsl:sequence select="p:get-binary-expression(., '<=')"/>
  </xsl:template>

  <!-- Mode "expression". ge. -->
  <xsl:template mode="p:expression" match="ge">
    <xsl:sequence select="p:get-binary-expression(., '>=')"/>
  </xsl:template>

  <!-- Mode "expression". lt. -->
  <xsl:template mode="p:expression" match="lt">
    <xsl:sequence select="p:get-binary-expression(., '<')"/>
  </xsl:template>

  <!-- Mode "expression". gt. -->
  <xsl:template mode="p:expression" match="gt">
    <xsl:sequence select="p:get-binary-expression(., '>')"/>
  </xsl:template>

  <!-- Mode "expression". add. -->
  <xsl:template mode="p:expression" match="add">
    <xsl:sequence select="p:get-binary-expression(., '+')"/>
  </xsl:template>

  <!-- Mode "expression". sub. -->
  <xsl:template mode="p:expression" match="sub">
    <xsl:sequence select="p:get-binary-expression(., '-')"/>
  </xsl:template>

  <!-- Mode "expression". mul. -->
  <xsl:template mode="p:expression" match="mul">
    <xsl:sequence select="p:get-binary-expression(., '*')"/>
  </xsl:template>

  <!-- Mode "expression". div. -->
  <xsl:template mode="p:expression" match="div">
    <xsl:sequence select="p:get-binary-expression(., '/')"/>
  </xsl:template>

  <!-- Mode "expression". neg. -->
  <xsl:template mode="p:expression" match="neg">
    <xsl:sequence select="'-'"/>
    <xsl:sequence select="t:get-expression(*[1])"/>
  </xsl:template>

  <!-- Mode "expression". not. -->
  <xsl:template mode="p:expression" match="not">
    <xsl:sequence select="'!'"/>
    <xsl:sequence select="t:get-expression(*[1])"/>
  </xsl:template>

  <!-- Mode "expression". parens. -->
  <xsl:template mode="p:expression" match="parens">
    <xsl:sequence select="'('"/>
    <xsl:sequence select="t:get-expression(*[1])"/>
    <xsl:sequence select="')'"/>
  </xsl:template>

  <!-- Mode "expression". var. -->
  <xsl:template mode="p:expression" match="var">
    <xsl:sequence select="@name"/>
  </xsl:template>

  <!-- Mode "expression". int, short, byte, long, float, double. -->
  <xsl:template mode="p:expression"
    match="int | short | byte | long | float | double">
    <xsl:sequence select="."/>
  </xsl:template>

 </xsl:stylesheet>

I often find myself in a position that whenever I'm thinking of something, I can find the idea to be already implemented somewhere.

A good example is xslt/xquery -> java code.

Well, the world is full with smart guys.

Wow, I've found an article Code generation in XSLT 2.0. The article is dated by year 2005.

Well, I was inventing a bicycle. This is a good lesson for me.

I'm going to study very carefully about SQL Code Generation, as this is exacly the same task I'm facing now.

I've updated jxom.zip.

There are minor fixes there. The most important addition is a line breaker. The purpose of the line breaker is to split long lines.

Long lines appear if there are verbose comments, or there is a very long expression, which was not categorized as multiline.

It's not perfect, however looks acceptable.

Now I'm facing a next problem: I need to do a similar job I'm doing to java, however with sql. Moreover, I need to support several dialects of sql. I'm not sure if it's possible (worth) to define single sql-xom.xsd, or should I define sql-db2-v9-xom.xsd, sql-sqlserver-2005-xom.xsd, ...

The bad news are that sql grammar is much more complex than one of java. Probably I'll start from some sql subset. In any case I do not consider generation of sql "directly", as jxom fits remarkably into its role.

Building jxom stylesheets I've learned what is a "good" and "bad" recursion from the saxon's perspective.

I'm using control tokens $t:indent and $t:unindent to control indentation in the sequence of tokens defining java output. To build output lines I need to calculate total indentation for each line. This can be done using cummulative sum, considering $t:indent as +1 and $t:unindent as -1.

This task can be formalized as "calculate cummulative integer sum".

The first approach I've tested is non recursive: "for $i in 1 to count($items) return sum(subsequence($items, 1, $i))".
It is incredibly slow.

The next try was recurrent: calculate and spew results as they are calculated.
This is "crash fast" method. Saxon, indeed, implements this as recursion and arrives to a stack limit early.

The last approach, employes saxon's ability to detect some particular flavour of tail calls. When function contains a tail call, and the output on a tail call code path consists of this tail call only, then saxon transforms such construction into a cycle. Thus I need to accumulate result and pass it down to a tail call chain and output it on the last opportunity only.

The following sample shows this technique:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:t="http://www.nesterovsky-bros.com"
  exclude-result-prefixes="xs t">

  <xsl:output method="xml" indent="yes"/>

  <xsl:template match="/">
    <xsl:variable name="values" as="xs:integer*" select="1 to 10000"/>

    <result>
      <sum>
        <xsl:value-of select="t:cumulative-integer-sum($values)"/>

        <!-- This call crashes with stack overflow. -->
        <!-- <xsl:value-of select="t:bad-cumulative-integer-sum($values)"/> -->

        <!-- To compare speed uncomment following lines. -->
        <!--<xsl:value-of select="sum(t:cumulative-integer-sum($values))"/>-->
        <!--<xsl:value-of select="sum(t:slow-cumulative-integer-sum($values))"/>-->
      </sum>
    </result>
  </xsl:template>

  <!--
    Calculates cumulative sum of integer sequence.
      $items - input integer sequence.
      Returns an integer sequence that is a cumulative sum of original sequence.
  -->
  <xsl:function name="t:cumulative-integer-sum" as="xs:integer*">
    <xsl:param name="items" as="xs:integer*"/>

    <xsl:sequence select="t:cumulative-integer-sum-impl($items, 1, 0, ())"/>
  </xsl:function>

  <!--
    Implementation of the t:cumulative-integer-sum.
      $items - input integer sequence.
      $index - current iteration index.
      $sum - base sum.
      $result - collected result.
      Returns an integer sequence that is a cumulative sum of original sequence.
  -->
  <xsl:function name="t:cumulative-integer-sum-impl" as="xs:integer*">
    <xsl:param name="items" as="xs:integer*"/>
    <xsl:param name="index" as="xs:integer"/>
    <xsl:param name="sum" as="xs:integer"/>
    <xsl:param name="result" as="xs:integer*"/>

    <xsl:variable name="item" as="xs:integer?" select="$items[$index]"/>

    <xsl:choose>
      <xsl:when test="empty($item)">
        <xsl:sequence select="$result"/>
      </xsl:when>
      <xsl:otherwise>
        <xsl:variable name="value" as="xs:integer" select="$item + $sum"/>
        <xsl:variable name="next" as="xs:integer+" select="$result, $value"/>

        <xsl:sequence select="
          t:cumulative-integer-sum-impl($items, $index + 1, $value, $next)"/>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:function>

  <!-- "Bad" implementation of the cumulative-integer-sum. -->
  <xsl:function name="t:bad-cumulative-integer-sum" as="xs:integer*">
    <xsl:param name="items" as="xs:integer*"/>

    <xsl:sequence select="t:bad-cumulative-integer-sum-impl($items, 1, 0)"/>
  </xsl:function>

  <!-- "Bad" implementation of the cumulative-integer-sum. -->
  <xsl:function name="t:bad-cumulative-integer-sum-impl" as="xs:integer*">
    <xsl:param name="items" as="xs:integer*"/>
    <xsl:param name="index" as="xs:integer"/>
    <xsl:param name="sum" as="xs:integer"/>

    <xsl:variable name="item" as="xs:integer?" select="$items[$index]"/>

    <xsl:if test="exists($item)">
      <xsl:variable name="value" as="xs:integer" select="$item + $sum"/>
 
      <xsl:sequence select="$value"/>
      <xsl:sequence select="
        t:bad-cumulative-integer-sum-impl($items, $index + 1, $value)"/>
    </xsl:if>
  </xsl:function>

 <!-- Non recursive implementation of the cumulative-integer-sum. -->
 <xsl:function name="t:slow-cumulative-integer-sum" as="xs:integer*">
   <xsl:param name="items" as="xs:integer*"/>

   <xsl:sequence select="
     for $i in 1 to count($items) return
       sum(subsequence($items, 1, $i))"/>
 </xsl:function>

</xsl:stylesheet>