Many years ago we implemented Akinator like engine purely within SQL Server.

Today we use exactly the same technique to implement vector database.

Please see our GitHub repo: vector-database.

We've found that there is a Saxon HE update that was going to fix problems we mentioned in the previous post, and decided to give it a second chance.

Now Saxon fails with two other errors:

• Bad parent pointer message in Saxon-HE-9.8.0-2
• XTDE1061: There is no current group in Saxon-HE-9.8.0-2

We shall be waiting for the fixes. Mean time we're back to version 9.7.

Angular 2 is already available though there are a lot of code and libraries that are still in Angular 1.x. Here we outline how to write AngularJS 1.x in the modern javascript.

Prerequisites: EcmaScript 2015, javascript decorators, AngularJS 1.x. No knowledge of Angular 2.0 is required.

Please note that decorators we have introduced, while resemble those from Angular 2, do not match them exactly.

A sample uses nodejs, npm and gulp as a building pipeline. In addition we have added Visual Studio Nodejs project, and maven project.

Build pipeline uses Babel with ES 2015 and decorator plugins to transpile sources into javascript that today's browsers do support. Babel can be replaced or augmented with Typescript compiler to support Microsoft's javascript extensions. Sources are combinded and optionally minified into one or more javascript bundles. In addition html template files are transformed into javascript modules that export a content of html body as a string literals. In general all sources are in src folder and the build's output is assembled in the dist folder. Details of build process are in gulpfile.js

So, let's introduce an API we have defined in angular-decorators.js module:

• Class decorators:
  • Component(name, options?) - a decorator to register angular component.
  • Controller(name) - a decorator to register angular controller.
  • Directive(name, options?) - a decorator to register angular directive.
  • Injectable(name) - a decorator to register angular service.
  • Module(name, ...require) - a decorator to declare an angular module;
  • Pipe(name, pure?) - a decorator to register angular filter.

  Component's and Directive's options is the same object passed into Module.component(), Module.directive() calls with difference that no options.bindings, options.scope, options.require is specified. Instead @Attribute(), @Input(), @Output(), @TwoWay(), @Collection(), @Optional() are used to describe options.bindings, and @Host(), Self(), SkipSelf(), @Optional() are used to describe options.require

  Every decorated class can use @Inject() member decorator to inject a service.

• Member decorators:
  • Attribute(name?) - a decorator that binds attribute to the property.
  • BindThis() - a decorator that binds "this" of the function to the class instance.
  • Collection() - a decorator that binds a collection property to an expression in attribute in two directions.
  • Host(name?) - a decorator that binds a property to a host controller of a directive found on the element or its ancestors.
  • HostListener(name?) - a decorator that binds method to a host event.
  • Inject(name?) - an injection member decorator.
  • Input(name?) - a decorator that binds a property to an expression in attribute.
  • Optional() - a decorator that optionally binds a property.
  • Output(name?) - a decorator that provides a way to execute an expression in the context of the parent scope.
  • Self(name?) - a decorator that binds a property to a host controller of a directive found on the element.
  • SkipSelf(name?) - a decorator that binds a property to a host controller of a directive found on the ancestors of the element.
  • TwoWay() - a decorator that binds a property to an expression in attribute in two directions.

  If optional name is omitted in the member decorator then property name is used as a name parameter. @Host(), @Self(), @SkipSelf() accept class decorated with @Component() or @Directive() as a name parameter.

  @Inject() accepts class decorated with @Injectable() or @Pipe() as a name parameter.

• Other:
  • modules(...require) - converts an array of modules, possibly referred by module classes, to an array of module names.

Now we can start with samples. Please note that we used samples scattered here and there on the Anuglar site.

@Component(), @SkipSelf(), @Attribute()

In the Angular's component development guide there is a sample myTabs and myPane components.
Here its rewritten form components/myTabs.js:

import { Component } from "../angular-decorators"; // Import decorators
import template from "../templates/my-tabs.html"; // Import template for my-tabs component
@Component("myTabs", { template, transclude: true }) // Decorate class as a component
export class MyTabs // Controller class for the component
{
  panes = []; // List of active panes
  select(pane) // Selects an active pane
  {
    this.panes.forEach(function(pane) { pane.selected = false; });
    pane.selected = true;
  }
  addPane(pane) // Adds a new pane
  {
    if (this.panes.length === 0)
    {
      this.select(pane);
    }
    this.panes.push(pane);
  }
}

components/myPane.js:

import { Component, Attribute, SkipSelf } "../angular-decorators"; // Import decorators
import { MyTabs } from "./myTabs"; // Import container's directive.
import template from "../templates/my-pane.html"; // Import template.
@Component("myPane", { template, transclude: true }) // Decorate class as a component
export class MyPane // Controller class for the component
{
  @SkipSelf(MyTabs) tabsCtrl; //Inject ancestor MyTabs controller.
  @Attribute() title; // Attribute "@" binding.
  $onInit() // Angular's $onInit life-cycle hook.
  {
    this.tabsCtrl.addPane(this);
    console.log(this);
  };
}    

@Component(), @Input(), @Output()

In the Angular's component development guide there is a sample myTabs component.
Here its rewritten form components/heroDetail.js:

import { Component, Input, Output } from "../angular-decorators";
import template from "../templates/heroDetail.html";
@Component("heroDetail", { template }) // Decorate class as a component
export class HeroDetail // Controller class for the component
{
  @Input() hero; // One way binding "<"
  @Output() onDelete; // Bind expression in the context of the parent scope "&"
  @Output() onUpdate; // Bind expression in the context of the parent scope "&"
  delete()
  {
    this.onDelete({ hero: this.hero });
  };
  update(prop, value)
  {
    this.onUpdate({ hero: this.hero, prop, value });
  };
}

@Directive(), @Inject(), @Input(), @BindThis()

In the Angular's directive development guide there is a sample myCurrentTime directive.
Here its rewritten form directives/myCurrentTime.js:

import { Directive, Inject, Input, BindThis } from "../angular-decorators"; // Import decorators
@Directive("myCurrentTime") // Decorate MyCurrentTime class as a directive
export class MyCurrentTime // Controller class for the directive
{
  @Inject() $interval; // "$interval" service is injected into $interval property
  @Inject() dateFilter; // "date" filter service is injected into dateFilter property
  @Inject() $element; // "$element" instance is injected into $element property.
  @Input() myCurrentTime; // Input one way "<" property.
  timeoutId;
  // updateTime is adapted as following in the constructor: 
  //   this.updateTime = this.updateTime.bind(this);
  @BindThis() updateTime() 
  {
    this.$element.text(this.dateFilter(new Date(), this.myCurrentTime));
  }
  $onInit() // Angular's $onInit life-cycle hook.
  {
    this.timeoutId = this.$interval(this.updateTime, 1000);
  }
  $onDestroy() // Angular's $onDestroys life-cycle hook.
  {
    this.$interval.cancel(this.timeoutId);
  }
  $onChanges(changes) // Angular's $onChanges life-cycle hook.
  {
    this.updateTime();
  }
}

@Directive(), @Inject(), @HostListener(), @BindThis()

In the Angular's directive development guide there is a sample myDraggable directive.
Here its rewritten form. directives/myDraggable.js:

import { Directive, Inject, HostListener, BindThis } from "../angular-decorators"; // Import decorators
@Directive("myDraggable") // Decorate class as a directive
export class MyDraggable // Controller class for the directive
{
  @Inject() $document; // "$document" instance is injected into $document property.
  @Inject() $element;// "$element" instance is injected into $element property.
  startX = 0;
  startY = 0;
  x = 0;
  y = 0;
  // Listen mousedown event over $element.
  @HostListener() mousedown(event)
  {
    // Prevent default dragging of selected content
    event.preventDefault();
    this.startX = event.pageX - this.x;
    this.startY = event.pageY - this.y;
    this.$document.on('mousemove', this.mousemove);
    this.$document.on('mouseup', this.mouseup);
  }
  @BindThis() mousemove(event) // bind mousemove() function to "this" instance.
  {
    this.y = event.pageY - this.startY;
    this.x = event.pageX - this.startX;
    this.$element.css({
      top: this.y + 'px',
      left: this.x + 'px'
    });
  }
  @BindThis() mouseup() // bind mouseup() function to "this" instance.
  {
    this.$document.off('mousemove', this.mousemove);
    this.$document.off('mouseup', this.mouseup);
  }
  $onInit() // Angular's $onInit life-cycle hook.
  {
    this.$element.css(
    {
      position: 'relative',
      border: '1px solid red',
      backgroundColor: 'lightgrey',
      cursor: 'pointer'
    });
  }
}

@Injectable(), @Inject()

In the Angular's providers development guide there is a sample notifier service.
Here its rewritten form. services/notify.js:

import { Inject, Injectable } from "../angular-decorators"; // Import decorators
@Injectable("notifier") // Decorate class as a service
export class NotifierService
{
  @Inject() $window; // Inject "$window" instance into the property
  msgs = [];
  notify(msg)
  {
    this.msgs.push(msg);
    if (this.msgs.length === 3)
    {
      this.$window.alert(this.msgs.join('\n'));
      this.msgs = [];
    }
  }
}

@Pipe()

In the Angular's filters development guide there is a sample reverse custom filter.
Here its rewritten form. filters/reverse.js:

import { Pipe } from "../angular-decorators"; // Import decorators
@Pipe("reverse") // Decorate class as a filter
export class ReverseFilter
{
  transform(input, uppercase) // filter function.
  {
    input = input || '';
    var out = '';
    for(var i = 0; i < input.length; i++)
    {
      out = input.charAt(i) + out;
    }
    // conditional based on optional argument
    if (uppercase)
    {
      out = out.toUpperCase();
    }
    return out;
  }
}

Module(), modules(), angular.bootstrap()

Here are an examples of a class representing angular module, and manual angular bootstrap:

import { angular, modules, Module } from "../angular-decorators"; // Import decorators
import { MyController } from "./controllers/myController"; // Import components.
import { HeroList } from "./components/heroList";
import { HeroDetail } from "./components/heroDetail";
import { EditableField } from "./components/editableField";
import { NotifierService } from "./services/notify";
import { MyTabs } from "./components/myTabs";
import { MyPane } from "./components/myPane";
import { ReverseFilter } from "./filters/reverse";
import { MyCurrentTime } from "./directives/myCurrentTime";
import { MyDraggable } from "./directives/myDraggable";
@Module( // Decorator to register angular module, and refer to other modules or module components.
  "my-app",
  [
    MyController,
    NotifierService,
    HeroList,
    HeroDetail,
    EditableField,
    MyTabs,
    MyPane,
    ReverseFilter,
    MyCurrentTime,
    MyDraggable
  ])
class MyApp { }
// Manual bootstrap, with modules() converting module classes into an array of module names.
angular.bootstrap(document, modules(MyApp));

Please see angular-decorators.js to get detailed help on decorators.

Until now we've been aware of 2 excelent artificial intelligence frameworks written in C#. These are AForge.NET and its successor Accord.NET. The both include a lot of algorithms for solving wide range of tasks.

Yesterday we've discovered that Microsoft has published as an open-source project their Computational Network Toolkit in order to speed up advances in artificial intelligence and made it available to a broader group of developers.

The sources written in C++ which scales good and uses GPUs. This gives a competitive advantage to CNTK, see more details here.

Although the main aim of such development was speech recognition, the CNTK contains a Neural Network framework that may be used for other artificial intelligence tasks.

It's very old theme...

Many years ago we have defined a .NET wrapper around Windows Uniscribe API.

Uniscribe API is used to render bidirectional languages like Hebrew, so it's important mainly here in Israel.

Once in a while we get request from people to give that API, so we published it on GitHub at https://github.com/nesterovsky-bros/BidiVisualConverter.

You're welcome to use it!

A collegue has approached to us with a question on how Akinator engine may work.

To our shame we have never heard about this amazing game before. To fill the gap we have immediately started to play it, and have identified it as a Troubleshooting solver.

It took us a couple of minutes to come up with a brilliant solution: "We just need to google and find the engine in the internet".

Unfortunately, this led to nowhere, as no Akinator itself is open sourced, and no other good quality open source solutions are available.

After another hour we have got two more ideas:

1. The task should fit into SQL;
2. The task is a good candidate for a neural network.

In fact, the first might be required to teach the second, so we have decided to formalize the problem in terms of SQL, while still keeping in mind a neural network.

With this goal we have created a GitHub project. Please see the algorithm and its implementation at github.com/nesterovsky-bros/KB.

It's time to align csharpxom to the latest version of C#. The article New Language Features in C# 6 sums up what's being added.

Sources can be found at nesterovsky-bros/languages-xom, and C# model is at csharp folder.

In general we feel hostile to any new features until they prove they bring an added value. So, here our list of new features from most to least useless:

1. String interpolation

   var s = $"{p.Name} is {p.Age} year{{s}} old";

   This is useless, as it does not account resource localization.

2. Null-conditional operators

   int? first = customers?[0].Orders?.Count();

   They claim to reduce cluttering from null checks, but in our opinion it looks opposite. It's better to get NullReferenceException if arguments are wrong.

3. Exception filters

   private static bool Log(Exception e) { /* log it */ ; return false; }
…
try { … } catch (Exception e) when (Log(e)) {}

   "It is also a common and accepted form of “abuse” to use exception filters for side effects; e.g. logging."

   Design a feature for abuse just does not tastes good.

4. Expression-bodied function and property members.

   public Point Move(int dx, int dy) => new Point(x + dx, y + dy);
public string Name => First + " " + Last;

   Not sure it's that usefull.

After ECMAScript Xml Object Model we aligned JXOM to support Java 8. This includes support of:

• Lambda Expressions - http://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html
• Method References - http://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html
  
• Default Methods - http://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html
  

As with ECMAScript, all sources are available at https://github.com/nesterovsky-bros/languages-xom

Much time has passed since we fixed or extended Languages Xml Object Model. But now we needed to manipulate with and generate javascript programs.

Though xslt today is not a language of choice but rather niche language, it still fits very well to tasks of code generation and transformation.

So, we're pleased to announce ECMAScript Xml Object Model, which includes:

• xml schema ecmasript.xsd, which is based on ECMAScript 6^th Edition / Draft February 20, 2015;
• a set of xslt to generate the text code, and perform some basic transformations over the tree;
• a set of smoke tests to verify the generation.

All sources are available at github: https://github.com/nesterovsky-bros/languages-xom

A year ago we had published out attempt at creating WeakTable in .NET. This is not too often used API but at rare case it's just irreplaceable.

But yesterday a user @Paya has finally found our class to be useful, performed the code review, and identified some bugs. See discussion at StackOverflow.

Bug fixes are available at: https://github.com/nesterovsky-bros/WeakTable.

Dasha Nesterovsky

While trying to generalize our practices from KendoUI related projects we've participated so far, we updated control.js - a small javascript additions to KendoUI.

At present we have defined:

1. An extended model. See KendoUI extended model.

2. A lightweight user control - a widget to bind a template and a model, and to facilitate declarative instantiation. See KendoUI User control.

3. A reworked version of nesterovskyBros.defineControl() function.

var widgetType = scope.defineControl(
{
   name: widget-name-string,
   model: widget-model-type,
   template: optional-content-template,
   windowOptions: optional-window-options
},
base);

When optional-content-template is not specified then template is calculated as following:

var template = options.temlate || proto.template || model.temlate;

if (template === undefined)
{
  template = scope.template(options.name.toLowerCase() + "-template");
}

When windowOptions is specified then widgetType.dialog(options) function is defined. It's used to open dialog based on the specified user control. windowOptions is passed to kendo.ui.Window constructor. windowOptions.closeOnEscape indicates whether to close opened dialog on escape.

widgetType.dialog() returns a kendo.ui.Window instance with content based on the user control. Window instance contains functions:

• result() - a $.Deffered for the dialog result, and
• model() - referring to the user control model.

 The model instance has functions:

•  dialog() referring to the dialog, and
• result() referring to the dialog result.

widget.dialog() allows all css units in windowOptions.width and windowOptions.height parameters.

base - is optional user control base. It defaults to nesterovskyBros.ui.UserControl.

4. Adjusted splitter. See Adjust KendoUI Splitter.

5. Auto resize support.

Layout is often depends on available area. One example is Splitter widget that recalculates its panes when window or container Splitter is resized. There are other cases when you would like to adjust layout when a container's area is changed like: adjust grid, tab, editor or user's control contents.

KendoUI does not provide a solution for this problem, so we have defined our own.

• A widget can be marked with class="auto-resize" marker;
• A widget may define a widgetType.autoResize(element) function that adapts widget to a new size.
• A code can call nesterovskyBros.resize(element) function at trigger resizing of the subtree.

To support existing controls we have defined autoResize() function for Grid, Splitter, TabStrip, and Editor widgets.

To see how auto resizing works, it's best to look into index.html, products.tmpl.html, and into the implementation controls.js.

Please note that we consider controls.js as an addition to KendoUI library. If in the future the library will integrate or implement similar features we will be happy to start using their API.

See also: Compile KendoUI templates.

This time we update csharpxom to adjust it to C# 4.5. Additions are async modifier and await operator.

They are used to simplify asynchronous programming.

The following example from the msdn:

private async Task<byte[]> GetURLContentsAsync(string url)
{
  var content = new MemoryStream();
  var request = (HttpWebRequest)WebRequest.Create(url);

  using(var response = await request.GetResponseAsync())
  using(var responseStream = response.GetResponseStream())
  {
    await responseStream.CopyToAsync(content);
  }

  return content.ToArray();
}

looks like this in csharpxom:

<method name="GetURLContentsAsync" access="private" async="true">
  <returns>
    <type name="Task" namespace="System.Threading.Tasks">
      <type-arguments>
        <type name="byte" rank="1"/>
      </type-arguments>
    </type>
  </returns>
  <parameters>
    <parameter name="url">
      <type name="string"/>
    </parameter>
  </parameters>
  <block>
    <var name="content">
      <initialize>
        <new-object>
          <type name="MemoryStream" namespace="System.IO"/>
        </new-object>
      </initialize>
    </var>
    <var name="request">
      <initialize>
        <cast>
          <invoke>
            <static-method-ref name="Create">
              <type name="WebRequest" namespace="System.Net"/>
            </static-method-ref>
            <arguments>
              <var-ref name="url"/>
            </arguments>
          </invoke>
          <type name="HttpWebRequest" namespace="System.Net"/>
        </cast>
      </initialize>
    </var>

    <using>
      <resource>
        <var name="response">
          <initialize>
            <await>
              <invoke>
                <method-ref name="GetResponseAsync">
                  <var-ref name="request"/>
                </method-ref>
              </invoke>
            </await>
          </initialize>
        </var>
      </resource>
      <using>
        <resource>
          <var name="responseStream">
            <initialize>
              <invoke>
                <method-ref name="GetResponseStream">
                  <var-ref name="response"/>
                </method-ref>
              </invoke>
            </initialize>
          </var>
        </resource>
        <expression>
          <await>
            <invoke>
              <method-ref name="CopyToAsync">
                <var-ref name="responseStream"/>
              </method-ref>
              <arguments>
                <var-ref name="content"/>
              </arguments>
            </invoke>
          </await>
        </expression>
      </using>
    </using>

    <return>
      <invoke>
        <method-ref name="ToArray">
          <var-ref name="content"/>
        </method-ref>
      </invoke>
    </return>
  </block>
</method>

Some time ago our younger brother Aleksander had started studying of cinematography.

Few days ago he started his own "multimedia" blog (you'll better understand me when you'll see it), where you can see his portfolio. Aleksander's latest work was made with cooperation with Ilan Lahov, see "Bar mitzvah". This work demonstrates Aleksander's progress in this field.

Our congratulations to Aleksander!

Please welcome a new human being Masha Vladimirovna Nesterovsky!

Recently one of users of java yield return annotation has kindly informed us about some problem that happened in his environment (see Java's @Yield return annotation update).

Incidentally we have never noticed the problem earlier. Along with this issue we have found that eclipse compiler has changed in the Indigo in a way that we had to recompile the source. Well, that's a price you have to pay when you access internal API.

Updated sources can be found at Yield.zip, and compiled jars at Yield.jar (pre-Indigo), and Yield.3.7.jar (Indigo and probably higher).

See also:

Yield return feature in java
Why @Yield iterator should be Closeable
What you can do with jxom.

We did not update languages-xom already for many monthes but now we have found a severe bug in the jxom's algorithm for eliminating unreachable code. The marked line were considered as unreachable:

check:
  if (condition)
  {
    break check;
  }
  else
  {
    return;
  }

  // due to bug the following was considered unreachable
  expression;

Bug is fixed.

Current update contains other cosmetic fixes.

Please download xslt sources from languages-xom.zip.

Summary

Languages XOM is a set of xml schemas and xslt stylesheets that allows:

• to define programs in xml form;
• to perform transformations over code in xml form;
• to generate sources.

Languages XOM includes:

• jxom - Java Xml Object model;
• csharpxom - C# Xml Object Model;
• cobolxom - COBOL Xml Object Model;
• sqlxom - SQL Xml Object Model (including several sql dialects);
• aspx - ASP.NET Object Model;

A proprietary part of languages XOM also includes XML Object Model for a language named Cool:GEN. In fact the original purpose for this API was a generation of java/C#/COBOL from Cool:GEN. For more details about Cool:GEN conversion please see here.

Ladies and gentlemen!

We are proud and would like to announce few works of our younger brother Aleksander, who studies cinematography now.

We have updated @Yield annotation processor to support better debug info.

Annotation processor can be downloaded from Yield.zip or Yield.jar.

We also decided to consider jxom's state machine refactoring as obsolete as @Yield annotation allows to achieve the same effect but with more clear code.

JXOM can be downloaded from languages-xom.zip

See also:

Yield return feature in java
Why @Yield iterator should be Closeable
What you can do with jxom.

We're happy to announce that we have implemented @Yield annotation both in javac and in eclipse compilers.

This way you get built-in IDE support for the feature!

To download yield annotation processor please use the following link: Yield.zip

It contains both yield annotation processor, and a test project.

If you do not want to compile the sources, you can download Yield.jar

We would like to reiterate on how @Yield annotation works:

1. A developer defines a method that returns either Iterator<T> or Iterable<T> instance and marks it with @Yield annotation.
2. A developer implements iteration logic following the pattern:
   
   • declare a variable to accumulate results:
       ArrayList<T> items = new ArrayList<T>();
   • use the following statement to add item to result:
       items.add(...);
   • use
       return items;
     or
       return items.iterator();
to return result;
   • mark method's params, if any, as final.
3. A devoloper ensures that yield annotation processor is available during compilation (see details below).
4. YieldProcessor rewrites method into a state machine at compilation time.

The following is an example of such a method:

@Yield
public static Iterable<Integer> generate(final int from, final int to)
{
  ArrayList<Integer> items = new ArrayList<Integer>();

  for(int i = from; i < to; ++i)
  {
    items.add(i);
  }

  return items;
}

The use is like this:

for(int value: generate(7, 20))
{
  System.out.println("generator: " + value);
}

Notice that method's implementation still will be correct in absence of YieldProcessor.

Other important feature is that the state machine returned after the yield processor is closeable.

This means that if you're breaking the iteration before the end is reached you can release resources acquired during the iteration.

Consider the example where break exits iteration:

@Yield
public static Iterable<String> resourceIteration()
{
  ArrayList<String> items = new ArrayList<String>();

  acquire();

  try
  {
    for(int i = 0; i < 100; ++i)
    {
      items.add(String.valueOf(i));
    }
  }
  finally
  {
    release();
  }

  return items;
}

and the use

int i = 0;
Iterable<String> iterator = resourceIteration();

try
{
  for(String item: iterator)
  {
    System.out.println("item " + i + ":" + item);

    if (i++ > 30)
    {
      break;
    }
  }
}
finally
{
  close(iterator);
}

...

private static <T> void close(T value)
  throws IOException
{
  if (value instanceof Closeable)
  {
    Closeable closeable = (Closeable)value;

    closeable.close();
  }
}

Close will execute all required finally blocks. This way resources will be released.

To configure yield processor a developer needs to refer Yield.jar in build path, as it contains @Yield annotation. For javac it's enough, as compiler will find annotation processor automatically.

Eclipse users need to open project properties and:

• go to the "Java Compiler"/"Annotation Processing"
• mark "Enable project specific settings"
• select "Java Compiler"/"Annotation Processing"/"Factory Path"
• mark "Enable project specific settings"
• add Yield.jar to the list of "plug-ins and JARs that contain annotation processors".

At the end we want to point that @Yield annotation is a syntactic suggar, but it's important the way the foreach statement is important, as it helps to write concise and an error free code.

See also
  Yield feature in java implemented!
  Yield feature in java

For some reason we never knew about instance initializer in java; on the other hand static initializer is well known.

class A
{
  int x;
  static int y;

  // This is an instance initializer.
  {
    x = 1;
  }

  // This is a static initializer.
  static
  {
    y = 2;
  }
}

Worse, we have missed it in the java grammar when we were building jxom. This way jxom was missing the feature.

Today we fix the miss and introduce a schema element:

<class-initializer static="boolean">
  <block>
    ...
  </block>
</class-initializer>

It superseeds:

<static>
  <block>
    ...
  </block>
</static>

 that supported static initializers alone.

Please update languages-xom xslt stylesheets.

P.S. Out of curiosity, did you ever see any use of instance initializers?

We could not stand the temptation to implement the @Yield annotation that we described earlier.

Idea is rather clear but people are saying that it's not an easy task to update the sources.

They were right!

Implementation has its price, as we were forced to access JDK's classes of javac compiler. As result, at present, we don't support other compilers such as EclipseCompiler. We shall look later what can be done in this area.

At present, annotation processor works perfectly when you run javac either from the command line, from ant, or from other build tool.

Here is an example of how method is refactored:

@Yield
public static Iterable<Long> fibonachi()
{
  ArrayList<Long> items = new ArrayList<Long>();

  long Ti = 0;
  long Ti1 = 1;

  while(true)
  {
    items.add(Ti);

    long value = Ti + Ti1;

    Ti = Ti1;
    Ti1 = value;
  }
}

And that's how we transform it:

@Yield()
public static Iterable<Long> fibonachi() {
  assert (java.util.ArrayList<Long>)(ArrayList<Long>)null == null : null;

  class $state$ implements java.lang.Iterable<Long>, java.util.Iterator<Long>, java.io.Closeable {

    public java.util.Iterator<Long> iterator() {
      if ($state$id == 0) {
        $state$id = 1;
        return this;
      } else return new $state$();
    }

    public boolean hasNext() {
      if (!$state$nextDefined) {
        $state$hasNext = $state$next();
        $state$nextDefined = true;
      }

      return $state$hasNext;
    }

    public Long next() {
      if (!hasNext()) throw new java.util.NoSuchElementException();

      $state$nextDefined = false;

      return $state$next;
    }

    public void remove() {
      throw new java.lang.UnsupportedOperationException();
    }

    public void close() {
      $state$id = 5;
    }

    private boolean $state$next() {
      while (true) switch ($state$id) {
      case 0:
        $state$id = 1;
      case 1:
        Ti = 0;
        Ti1 = 1;
      case 2:
        if (!true) {
          $state$id = 4;
          break;
        }

        $state$next = Ti;
        $state$id = 3;

        return true;
      case 3:
        value = Ti + Ti1;
        Ti = Ti1;
        Ti1 = value;
        $state$id = 2;

        break;
      case 4:
      case 5:
      default:
        $state$id = 5;

        return false;
      }
    }

    private long Ti;
    private long Ti1;
    private long value;
    private int $state$id;
    private boolean $state$hasNext;
    private boolean $state$nextDefined;
    private Long $state$next;
  }

  return new $state$();
}

Formatting is automatic, sorry, but anyway it's for diagnostics only. You will never see this code.

It's iteresting to say that this implementation is very precisely mimics xslt state machine implementation we have done back in 2008.

You can download YieldProcessor here. We hope that someone will find our solution very interesting.

You might be interested in the following article that was written in form of a little guide. It can educate about new ways to learn SQL and hopefully may help someone to get a job. See "How to get MS SQL certification" that was written by Michele Rouse.

Historically jxom was developed first, and as such exhibited some imperfectness in its xml schema. csharpxom has taken into an account jxom's problems.

Unfortunately we could not easily fix jxom as a great amount of code already uses it. In this refactoring we tried to be conservative, and have changed only "type" and "import" xml schema elements in java.xsd.

Consider type reference and package import constructs in the old schema:

<!-- import java.util.ArrayList; -->
<import name="java.util.ArrayList"/>

<!-- java.util.ArrayList<java.math.BigDecimal> -->
<type package="java.util">
  <part name="ArrayList">
    <argument>
      <type name="BigDecimal" package="java.math">
    </argument>
  </part>
</type>

<!-- my.Parent.Nested -->
<type package="my">
  <part name="Parent"/>
  <part name="Nested"/>
<type>

Here we can observe that:

• type is referred by a qualified name in import element;
• type has two forms: simple (see BigDecimal), and other for nested or generic type (see ArrayList).

We have made it more consistent in the updated jxom:

<!-- import java.util.ArrayList; -->
<import>
  <type name="ArrayList" package="java.util"/>
</import>

<!-- java.util.ArrayList<java.math.BigDecimal> -->
<type name="ArrayList" package="java.util">
  <argument>
    <type name="BigDecimal" package="java.math">
  </argument>
</type>

<!-- my.Parent.Nested -->
<type name="Nested">
  <type name="Parent" package="my"/>
<type>

We hope that you will not be impacted very much by this fix.

Please refresh Languages XOM from languages-xom.zip.

P.S. we have also included xml schema and xslt api to generate ASPX (see Xslt serializer for ASPX output). We, in fact, in our projects, generate aspx documents with embedded csharpxom, and then pass it through two stage transformation.

In the previous post we have announced an API to parse a COBOL source into the cobolxom.

We exploited the incremental parser to build a grammar xml tree and then were planning to create an xslt transformation to generate cobolxom.

Now, we would like to declare that such xslt is ready.

At present all standard COBOL constructs are supported, but more tests are required. Preprocessor support is still in the todo list.

You may peek into an examples of COBOL:

Cobol grammar:

And cobolxom:

While we were building a grammar to cobolxom stylesheet we asked ourselves whether the COBOL parsing could be done entirely in xslt. The answer is yes, so who knows it might be that we shall turn this task into pure xslt one. :-)

Once ago we have created an incremental parser, and now when we have decided to load COBOL sources directly into cobolxom (XML Object Model for a COBOL) the parser did the job perfectly.

The good point about incremental parser is that it easily handles COBOL's grammar.

The whole process looks like this:

1. incremental parser having a COBOL grammar builds a grammar tree;
2. we stream this tree into xml;
3. xslt to transform xml from previous step into cobolxom (TODO).

This is an example of a COBOL:

IDENTIFICATION DIVISION.
PROGRAM-ID. FACTORIAL RECURSIVE.

DATA DIVISION.
WORKING-STORAGE SECTION.
01 NUMB PIC 9(4) VALUE IS 5.
01 FACT PIC 9(8) VALUE IS 0.

LOCAL-STORAGE SECTION.
01 NUM PIC 9(4).

PROCEDURE DIVISION.
  MOVE 'X' TO XXX
  MOVE NUMB TO NUM

  IF NUMB = 0 THEN
    MOVE 1 TO FACT
  ELSE
    SUBTRACT 1 FROM NUMB
    CALL 'FACTORIAL'
    MULTIPLY NUM BY FACT
  END-IF

  DISPLAY NUM '! = ' FACT

  GOBACK.
END PROGRAM FACTORIAL.

And a grammar tree:

<Program>
  <Name data="FACTORIAL"/>
  <Recursive/>
  <DataDivision>
    <WorkingStorageSection>
      <Data>
        <Level data="01"/>
        <Name data="NUMB"/>
        <Picture data="9(4)"/>
        <Value>
          <Numeric data="5"/>
        </Value>
      </Data>
      <Data>
        <Level data="01"/>
        <Name data="FACT"/>
        <Picture data="9(8)"/>
        <Value>
          <Numeric data="0"/>
        </Value>
      </Data>
    </WorkingStorageSection>
    <LocalStorageSection>
      <Data>
        <Level data="01"/>
        <Name data="NUM"/>
        <Picture data="9(4)"/>
      </Data>
    </LocalStorageSection>
  </DataDivision>
  <ProcedureDivision>
    <Sentence>
      <MoveStatement>
        <From>
          <String data="'X'"/>
        </From>
        <To>
          <Identifier>
            <DataName data="XXX"/>
          </Identifier>
        </To>
      </MoveStatement>
      <MoveStatement>
        <From>
          <Identifier>
            <DataName data="NUMB"/>
          </Identifier>
        </From>
        <To>
          <Identifier>
            <DataName data="NUM"/>
          </Identifier>
        </To>
      </MoveStatement>
      <IfStatement>
        <Condition>
          <Relation>
            <Identifier>
              <DataName data="NUMB"/>
            </Identifier>
            <Equal/>
            <Numeric data="0"/>
          </Relation>
        </Condition>
        <Then>
          <MoveStatement>
            <From>
              <Numeric data="1"/>
            </From>
            <To>
              <Identifier>
                <DataName data="FACT"/>
              </Identifier>
            </To>
          </MoveStatement>
        </Then>
        <Else>
          <SubtractStatement>
            <Value>
              <Numeric data="1"/>
            </Value>
            <From>
              <Identifier>
                <DataName data="NUMB"/>
              </Identifier>
            </From>
          </SubtractStatement>
          <CallStatement>
            <Name>
              <String data="'FACTORIAL'"/>
            </Name>
          </CallStatement>
          <MultiplyStatement>
            <Value>
              <Identifier>
                <DataName data="NUM"/>
              </Identifier>
            </Value>
            <By>
              <Identifier>
                <DataName data="FACT"/>
              </Identifier>
            </By>
          </MultiplyStatement>
        </Else>
      </IfStatement>
      <DisplayStatement>
        <Values>
          <Identifier>
            <DataName data="NUM"/>
          </Identifier>
          <String data="'! = '"/>
          <Identifier>
            <DataName data="FACT"/>
          </Identifier>
        </Values>
      </DisplayStatement>
      <GobackStatement/>
    </Sentence>
  </ProcedureDivision>
  <EndName data="FACTORIAL"/>
</Program>

The last step is to transform tree into cobolxom is in the TODO list.

We have commited COBOL grammar in the same place at SourceForge as it was with XQuery grammar. Solution is now under the VS 2010.

We have updated C# XOM (csharpxom) to support C# 4.0 (in fact there are very few changes).

From the grammar perspective this includes:

• Dynamic types;
• Named and optional arguments;
• Covariance and contravariance of generic parameters for interfaces and delegates.

Dynamic type, C#:

dynamic dyn = 1;

C# XOM:

<var name="dyn">
  <type name="dynamic"/>
  <initialize>
    <int value="1"/>
  </initialize>
</var>

Named and Optional Arguments, C#:

int Increment(int value, int increment = 1)
{
  return value + increment;
}

void Test()
{
  // Regular call.
  Increment(7, 1);

  // Call with named parameter.
  Increment(value: 7, increment: 1);
 
  // Call with default.
  Increment(7);
}

C# XOM:

<method name="Increment">
  <returns>
    <type name="int"/>
  </returns>
  <parameters>
    <parameter name="value">
      <type name="int"/>
    </parameter>
    <parameter name="increment">
      <type name="int"/>
      <initialize>
        <int value="1"/>
      </initialize>
    </parameter>
  </parameters>
  <block>
    <return>
      <add>
        <var-ref name="value"/>
        <var-ref name="increment"/>
      </add>
    </return>
  </block>
</method>

<method name="Test">
  <block>
    <expression>
      <comment>Regular call.</comment>
      <invoke>
        <method-ref name="Increment"/>
        <arguments>
          <int value="7"/>
          <int value="1"/>
        </arguments>
      </invoke>
    </expression>

    <expression>
      <comment>Call with named parameter.</comment>
      <invoke>
        <method-ref name="Increment"/>
        <arguments>
          <argument name="value">
            <int value="7"/>
          </argument>
          <argument name="increment">
            <int value="1"/>
          </argument>
        </arguments>
      </invoke>
    </expression>

    <expression>
      <comment>Call with default.</comment>
      <invoke>
        <method-ref name="Increment"/>
        <arguments>
          <int value="7"/>
        </arguments>
      </invoke>
    </expression>
  </block>
</method>

Covariance and contravariance, C#:

public interface Variance<in T, out P, Q>
{
  P X(T t);
}

C# XOM:

<interface access="public" name="Variance">
  <type-parameters>
    <type-parameter name="T" variance="in"/>
    <type-parameter name="P" variance="out"/>
    <type-parameter name="Q"/>
  </type-parameters>
  <method name="X">
    <returns>
      <type name="P"/>
    </returns>
    <parameters>
      <parameter name="t">
        <type name="T"/>
      </parameter>
    </parameters>
  </method>
</interface>

Other cosmetic fixes were also introduced into Java XOM (jxom), COBOL XOM (cobolxom), and into sql XOM (sqlxom).

The new version is found at languages-xom.zip.

See also: What's New in Visual C# 2010

Recently we were raising a question about serialization of ASPX output in xslt.

The question went like this:

What's the recommended way of ASPX page generation?
E.g.:

------------------------
 <%@ Page AutoEventWireup="true"
   CodeBehind="CurMainMenuP.aspx.cs"
   EnableSessionState="True"
   Inherits="Currency.CurMainMenuP"
   Language="C#"
   MaintainScrollPositionOnPostback="True"
   MasterPageFile="Screen.Master" %>

<asp:Content ID="Content1" runat="server" ContentPlaceHolderID="Title">CUR_MAIN_MENU_P</asp:Content>

<asp:Content ID="Content2" runat="server" ContentPlaceHolderID="Content">
  <span id="id1222146581" runat="server"
    class="inputField system UpperCase" enableviewstate="false">
    <%# Dialog.Global.TranCode %>
  </span>
  ...
------------------------

Notice aspx page directives, data binding expessions, and prefixed tag names without namespace declarations.

There was a whole range of expected answers. We, however, looked whether somebody have already dealed with the task and has a ready solution at hands.

In general it seems that xslt community is very angry about ASPX: both format and technology. Well, put this aside.

The task of producing ASPX, which is almost xml, is not solvable when you're staying with pure xml serializer. Xslt's xsl:character-map does not work at all. In fact it looks as a childish attempt to address the problem, as it does not support character escapes but only grabs characters and substitutes them with strings.

We have decided to create ASPX serializer API producing required output text. This way you use <xsl:output method="text"/> to generate ASPX pages.

With this goal in mind we have defined a little xml schema to describe ASPX irregularities in xml form. These are:

• <xs:element name="declared-prefix"> - to describe known prefixes, which should not be declared;
• <xs:element name="directive"> - to describe directives like <%@ Page %>;
• <xs:element name="content"> - a transparent content wrapper;
• <xs:element name="entity"> - to issue xml entity;
• <xs:element name="expression"> - to describe aspx expression like <%# Eval("A") %>;
• <xs:element name="attribute"> - to describe an attribute of the parent element.

This approach greately simplified for us an ASPX generation process.

The API includes:

• aspx.xsd - an xml schema for the ASPX elements;
• aspx-serializer.xslt - a serializer API;
• aspx-test.xslt - a test stylesheet (any xml input is used).

We have implemented report parser in C#. Bacause things are spinned around C#, a schema definition is changed.

We have started from classes defining a report definition tree, annotated these classes for xml serialization, and, finally, produced xml schema for such tree. So, at present, it is not an xml schema with annotations but a separate xml schema.

In addition we have defined APIs:

• to enumerate report data (having report definition and report data one can get IEnumerable<ViewValue> to iterate report data in structured form);
• to read report through XmlReader, which allows, for example, to have report as input for an xslt tranformation.
• to write report directly into XmlWriter.

An example of report definition as C# code is: MyReport.cs. The very same report definition but serialized into xml is my-report.xml. A generated xml schema for a report definition is: schema0.xsd.

The good point about this solution is that it's already flexible enough to describe every report layout we have at hands, and it's extendable. Our measurments show that report parsing is extremely fast and have very small memory footprint due to forward only nature of report definitions.

From the design point of view report definition is a view of original text data with view info attached.

At present we have defined following views:

• Element - a named view to generate output from a content view;
•  Content - a view to aggregate other views together;
• Choice - a view to produce output from one of content views;
• Sequence - a view to sequence input view by key expressions, and to attach an index to each sequence item;
• Iterator - a view to generate output from input view while some condition is true, and to attach an iteration index to each part of output view;
• Page - a view to remove page headers and footers in the input view, and to attach an index to each page;
• Compute - a named view to produce result of evaluation of expression as output view;
•  Data - a named view to produce output value from some bounds of input view, and optionally to convert, validate and format the value.

To specify details of definitions there are:

• expressions to deal with integers: Add, Div, Integer, MatchProperty, Max, Min, Mod, Mul, Neg, Null, Sub, VariableRef, ViewProperty, Case;
• conditions to deal with booleans: And, EQ, GE, GT, IsMatch, LE, LT, NE, Not, Or.

At present there is no a specification of a report definitions. Probably, it's the most complex part to create such a spec for a user without deep knowledge. At present, our idea is that one should use xml schema (we should polish generated schema) for the report definition and schema aware editor to build report definitions. That's very robust approach working perfectly with languages xom.

C# sources can be found at: ReportLayout.zip including report definition classes and a sample report.

We're facing a task of parsing reports produced from legacy applications and converting them into a structured form, e.g. into xml. These xml files can be processed further with up to date tools to produce good looking reports.

Reports at hands are of very different structure and of size: from a couple of KB to a several GB. The good part is that they mostly have a tabular form, so it's easy to think of specific parsers in case of each report type.

Our goal is to create an environment where a less qualified person(s) could create and manage such parsers, and only rarely to engage someone who will handle less untrivial cases.

Our analysis has shown that it's possible to write such parser in almost any language: xslt, C#, java.

Our approach was to create an xml schema annotations that from one side define a data structure, and from the other map report layout. Then we're able to create an xslt that will generate either xslt, C#, or java parser according to the schema definitions. Because of languages xom, providing XML Object Model and serialization stylesheets for C# and java, it does not really matter what we shall generate xslt or C#/java, as code will look the same.

The approach we're going to use to describe reports is not as powerfull as conventional parsers. Its virtue, however, is simplicity of specification.

Consider a report sample (a data to extract is in bold):

1 TITLE ...                    PAGE:            1
 BUSINESS DATE: 09/30/09   ... RUN DATE: 02/23/10
 CYCLE : ITD      RUN: 001 ... RUN TIME: 09:22:39

        CM         BUS   ...
  CO    NBR  FRM   FUNC  ...
 ----- ----- ----- -----  
 XXX   065   065   CLR   ...
 YYY   ...
...
1 TITLE ...                    PAGE:            2
 BUSINESS DATE: 09/30/09   ... RUN DATE: 02/23/10
 CYCLE : ITD      RUN: 001 ... RUN TIME: 09:22:39

        CM         BUS   ...
  CO    NBR  FRM   FUNC  ...
 ----- ----- ----- -----  
 AAA   NNN   MMM   PPP   ...
 BBB   ...
...

* * * * *  E N D   O F   R E P O R T  * * * * *

We're approaching to the report through a sequence of views (filters) of this report. Each veiw localizes some report data either for the subsequent filterring or for the extraction of final data.

Looking into the example one can build following views of the report:

1. View of data before the "E N D   O F   R E P O R T" line.
2. View of remaining data without page headers and footers.
3. Views of table rows.
4. Views of cells.

A sequence of filters allows us to build a pipeline of transformations of original text. This also allows us to generate a clean xslt, C# or java code to parse the data.

At first, our favorite language for such parser was xslt. Unfortunatelly, we're dealing with Saxon xslt implementation, which is not very strong in streaming processing. Without a couple of extension functions to prevent caching, it tends to cache whole input in the memory, which is not acceptable.

At present we have decided to start from C# code, which is pure C# naturally.

Code still is in the development but at present we would like to share the xml schema annotations describing report layout: report-mapping.xsd, and a sample of report description: test.xsd.

A few little changes in streaming and in name normalization algorithms in jxom and in csharpxom and the generation speed almost doubled (especially for big files).

We suspect, however, that our xslt code is tuned for saxon engine.

It would be nice to know if anybody used languages XOM with other engines. Is anyone using it at all (well, at least there are downloads)?

Languages XOM (jxom, csharpxom, cobolxom, sqlxom) can be loaded from: languages-xom.zip

When time has come to process big xml log files we've decided to implement streamable tree in saxon the very same way it was implemented in .net eight years ago (see How would we approach to streaming facility in xslt).

It's interesting enough that the implementation is similar to one of composable tree. There a node never stores a reference to a parent, while in the streamed tree no references to children are stored. This way only a limited subview of tree is available at any time. Implementation does not support preceding and preceding-sibling axes. Also, one cannot navigate to a node that is out of scope.

Implementation is external (there are no changes to saxon itself). To use it one needs to create an instance of DocumentInfo, which pulls data from XMLStreamReader, and to pass it as an input to a transformation:

Controller controller = (Controller)transformer;
XMLInputFactory factory = XMLInputFactory.newInstance();
StreamSource inputSource = new StreamSource(new File(input));
XMLStreamReader reader = factory.createXMLStreamReader(inputSource);
StaxBridge bridge = new StaxBridge();

bridge.setPipelineConfiguration(
  controller.makePipelineConfiguration());
bridge.setXMLStreamReader(reader);
inputSource = new DocumentImpl(bridge);

transformer.transform(inputSource, new StreamResult(output));

This helped us to format an xml log file of arbitrary size. An xslt like this can do the work:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns="http://www.w3.org/1999/xhtml"
  exclude-result-prefixes="xs">

  <xsl:template match="/log">
    <html>
      <head>
        <title>Log</title>
      </head>
      <body>
        <xsl:apply-templates/>
      </body>
    </html>
  </xsl:template>

  <xsl:template match="message">
   ...
  </xsl:template>

  <xsl:template match="message[@error]">
    ...
  </xsl:template>

  ...

</xsl:stylesheet>

Implementation can be found at: streamedtree.zip

Earlier, in the entry "Inline functions in xslt 2.1" we've described an implementation of xml tree model that may share subtrees among different trees.

This way, in a code:

<xsl:variable name="elements" as="element()*" select="..."/>

<xsl:variable name="result" as="element()">
  <result>
    <xsl:sequence select="$elements"/>
  </result>
</xsl:variable>

the implementation shares internal representation among $elements and subtree of $result. From the perspective of xslt it looks as completely different subtrees with different node identities, which is in the accordance with its view of the world.

After a short study we've decided to create a research implementation of this tree model in saxon. It's took only a couple of days to introduce a minimal changes to engine, to refactor linked tree into a new composable tree, and to perform some tests.

In many cases saxon has benefited immediately from this new tree model, in some other cases more tunings are required.

Our tests've showed that this new tree performed better than linked tree, but a little bit worser than tiny tree. On the other hand, it's obvious that conventional code patterns avoid subtree copying, assuming it's expensive operation, thus one should rethink some code practices to benefit from composable tree.

Implementation can be downloaded at: saxon.composabletree.zip

We've added a new language to the set of Xml Object Model schemas and stylesheets.

The newcomer is COBOL! No jokes. It's not a whim, really. Believe it or not but COBOL is still alive and we need to generate it (mainly different sorts of proxies).

We've used VS COBOL II grammar Version 1.0.3 as a reference. Implemented grammar is complete but without preprocessor statements. On the other hand it defines COPY and EXEC SQL constructs.

Definitely, it'll take a time for the xml schema and xslt implementation to become mature.

Now language XOM is:

• jxom - for java;
• csharpxom - for C#;
• cobolxom - for COBOL.

Sources can be found at languages-xom.

At present C# serializer knows how to print comments and do some formatting (we had to create micro xml serializer within xslt to serialize xml comments). C#'s formatting is not as advanced as java's one, but it should not be such in the first place, as C# text tends to be more neat due to properties and events. Compare:

Java: instance.getItems().get(10).setValue(value);

vs

C#: instance.Items[10].Value = value;

TODO: implement API existing in jxom and missing in C# xom. This includes:

• name normalization - rewriting tree to make names unique (duplicate names are often appear during generation from code templates);
• namespaces normalization - rewriting tree to elevate type namespaces (during generation, types are usually fully qualified);
• unreachable code detection - optional feature (in java it's required, as unreachable code is an error, while in C# it's only a warning);
• compile time expression evaluation - optional feature used in code optimization and in reachability checks;
• state machine refactoring - not sure, as C# has yield statement that does the similar thing.

Update can be found at: jxom/C# xom.

June, 24 update: name and namespace normalizations are implemented.

Writing a language serializer is an as easy task, as riding a bicycle. Once you learned it, you won't apply a mental force anymore to create a new one.

This still requires essential mechanical efforts to write and test things.

Well, this is the first draft of the C# xslt serializer. Archive contains both C# xom and jxom.

Note: no comments are still supported; nothing is done to format code except line wrapping.

Well, it's jxom no more but also csharpxom!

A project concerns demanded us to create a C# 3.0 xml schema.

Shortly we expect to create an xslt serializing an xml document in this schema into a text. Thankfully to the original design we can reuse java streamer almost without changes.

A fact: C# schema more than twice bigger than the java's.

Recently, we have started looking into a task of creating an interactive parser. A generic one.

Yes, we know there are plenty of them all around, however the goals we have defined made us to construct the new implementation.

The goals:

• Parser must be incremental.
  You should direct what to parse, and when to stop.
  This virtually demands rather "pull" than conventional "push" implementation.
• Parser must be able to synchronize a tree with text.
  Whenever the underlying text is changed, a limited part of a tree should to be updated.
• Parser should be able to recover from errors, and continue parsing.
• Parser should be manageable.
  This is a goal of every program, really.
• Parser must be fast.
• A low memory footprint is desired.

What's implemented (VS2008, C#) and put at SourceForge, is called an Incremental Parser.

These are parser's insights:

• Bookmarks are objects to track text points. We use a binary tree (see Bare binary tree algorithms) to adjust positions of bookmarks when text is changed.
• Ranges define parsed tree elements. Each range is defined by two bookmarks, and a grammar annotation.
• There are grammar primitives, which can be composed into a grammar graph.
• A grammar graph along with ranges form a state machine.
• Grammar chains are cached, turning parsing into a series of probes of literal tokens and transitions between grammar chains. This caching is done on demand, which results in warming-up effect.
• Parser itself includes a random access tokenizer, and a queue of ranges pending to be parsed.
• Parsing is conducted as a cycle of pulling and parsing of pending ranges.
• Whenever text is changed a closest range is queued for the reparsing.
• A balance between amount of parsing and memory consumption can be achieved through a detalization of grammar annotation for a range. An active text fragment can be fully annotated, while for other text parts a coarse range can be stored.

We have defined xpath like grammar to test our ideas. See printed parsed trees to get understanding of what information can be seen from ranges.

We have created Java Xml Object Model purely for purposes of our project. In fact jxom at present has siblings: xml models for sql dialects. There are also different APIs like name normalizations, refactorings, compile time evaluation.

It turns out that jxom is also good enough for other developers.

The drawback of jxom, however, is rather complex xml schema. It takes time to understand it. To simplify things we have created (and planning to create more) a couple of examples allowing to feel how jxom xml looks like.

The latest version can be loaded from jxom.zip

We would be pleased to see more comments on the subject.

Although in last our projects we're using more Java and XSLT, we always compare Java and .NET features. It's not a secret that in most applications we may find cache solutions used to improve performance. Unlike .NET providing a robust cache solution Java doesn't provide anything standard. Of course Java's adept may find a lot of caching frameworks or just to say: "use HashMap (ArrayList etc.) instead", but this is not the same.

Think about options for Java:
1. Caching frameworks (caching systems). Yes, they do their work. Do it perfectly. Some of them are brought to the state of the art, but there are drawbacks. The crucial one is that for simple data caching one should use a whole framework. This option requires too many efforts to solve a simple problem.

2. Collection classes (HashMap, ArrayList etc.) for caching data. This is very straightforward solution, and very productive. Everyone knows these classes, nothing to configure. One should declare an instance of such class, take care of data access synchronization and everything starts working immediately. An admirable caching solution but for "toy applications", since it solves one problem and introduces another one. If an application works for hours and there are a lot of data to cache, the amount of data grows only and never reduces, so this is the reason why such caching is very quickly surrounded with all sort of rules that somehow reduce its size at run-time. The solution very quickly lost its shine and become not portable, but it's still applicable for some applications.

3. Using Java reference objects for caching data. The most appropriate for cache solution is a java.util.WeekHashMap class. WeakHashMap works exactly like a hash table but uses weak references internally. In practice, entries in the WeakHashMap are reclaimed at any time if they are not refered outside of map. This caching strategy depends on GC's whims and is not entirely reliable, may increase a number of cache misses.

We've decided to create our simple cache with sliding expiration of data.

One may create many cache instances but there is only one global service that tracks expired objects among these instances:

private Cache<String, Object> cache = new Cache<String, Object>();

There is a constructor that specifies an expiration interval in milliseconds for all cached objects:

private Cache<String, Object> cache = new Cache<String, Object>(15 * 60 * 1000)

Access is similar to HashMap:

instance = cache.get("key"); and cache.put("key", instance);

That's all one should know to start use it. Click here to download the Java source of this class. Feel free to use it in your applications.

We are designing a rather complex xslt 2.0 application, dealing with semistructured data. We must tolerate with errors during processing, as there are cases where an input is not perfectly valid (or the program is not designed or ready to get such an input).

The most typical error is unsatisfied expectation of tree structure like:
  <xsl:variable name="element" as="element()" select="some-element"/>

Obviously, dynamic error occurs if a specified element is not present. To concentrate on primary logic, and to avoid a burden of illegal (unexpected) case recovery we have created a try/catch API. The goal of such API is:

• to be able to continue processing in case of error;
• report as much as possible useful information related to an error.

Alternatives:

Do not think this is our arrogance, which has turned us to create a custom API. No, we were looking for alternatives! Please see [xsl] saxon:try() discussion:

• saxon:try() function - is a kind of pseudo function, which explicitly relies on lazy evaluation of its arguments, and ... it's not available in SaxonB;
• ex:error-safe  extension instruction - is far from perfect in its implementation quality, and provides no error location.

We have no other way except to design this feature by ourselves. In our defence one can say that we are using innovatory approach that encapsulates details of the implementation behind template and calls handlers indirectly.

Use:

Try/catch API is designed as a template <xsl:template name="t:try-block"/> calling a "try" handler, and, if required, a "catch" hanler using <xsl:apply-templates mode="t:call"/> instruction. Caller passes any information to these handlers by the means of tunnel parameters.

Handlers must be in a "t:call" mode. The "catch" handler may recieve following error info parameters:

<xsl:param name="error" as="xs:QName"/>
<xsl:param name="error-description" as="xs:string"/>
<xsl:param name="error-location" as="item()*"/>

where $error-location is a sequence of pairs (location as xs:string, context as item())*.

A sample:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:t="http://www.nesterovsky-bros.com/xslt/public/"
  exclude-result-prefixes="xs t">

<xsl:include href="try-block.xslt"/>

<xsl:template match="/">
  <result>
    <xsl:for-each select="1 to 10">
      <xsl:call-template name="t:try-block">
        <xsl:with-param name="value" tunnel="yes" select=". - 5"/>
        <xsl:with-param name="try" as="element()">
          <try/>
        </xsl:with-param>
        <xsl:with-param name="catch" as="element()">
          <t:error-handler/>
        </xsl:with-param>
      </xsl:call-template>
    </xsl:for-each>
  </result>
</xsl:template>

<xsl:template mode="t:call" match="try">
  <xsl:param name="value" tunnel="yes" as="xs:decimal"/>

  <value>
    <xsl:sequence select="1 div $value"/>
  </value>
</xsl:template>

</xsl:stylesheet>


The sample prints values according to the formula "1/(i - 5)", where "i" is a variable varying from 1 to 10. Clearly, division by zero occurs when "i" is equal to 5.

Please notice how to access try/catch API through <xsl:include href="try-block.xslt"/>. The main logic is executed in <xsl:template mode="t:call" match="try"/>, which recieves parameters using tunneling. A default error handler <t:error-handler/> is used to report errors.

Error report:

Error: FOAR0001
Description:
Decimal divide by zero

Location:
1. systemID: "file:///D:/style/try-block-test.xslt", line: 34
2. template mode="t:call" match="element(try, xs:anyType)"
  systemID: "file:///D:/style/try-block-test.xslt", line: 30
  context node:
    /*[1][local-name() = 'try']
3. template mode="t:call"
  match="element({http://www.nesterovsky-bros.com/xslt/private/try-block}try, xs:anyType)"
  systemID: "file:///D:/style/try-block.xslt", line: 53
  context node:
    /*[1][local-name() = 'try']
4. systemID: "file:///D:/style/try-block.xslt", line: 40
5. call-template name="t:try-block"
  systemID: "file:///D:/style/try-block-test.xslt", line: 17
6. for-each
  systemID: "file:///D:/style/try-block-test.xslt", line: 16
  context item: 5
7. template mode="saxon:_defaultMode" match="document-node()"
  systemID: "file:///D:/style/try-block-test.xslt", line: 14
  context node:
    /

Implementation details:

You were not expecting this API to be pure xslt, weren't you?

Well, you're right, there is an extension function. Its pseudo code is like this:

function tryBlock(tryItems, catchItems)
{
  try
  {
    execute xsl:apply-templates for tryItems.
  }
  catch
  {
    execute xsl:apply-templates for catchItems.
  }
}

The last thing. Please get the implementation saxon.extensions.zip. There you will find sources of the try/catch, and tuples/maps API.

Recently I've proposed to add two new atomic types tuple and map to the xpath/xslt/xquery type system (see "Tuples an maps"). Later I've implemented tuple and map pure xslt approximation. Now I want to present java implementation for Saxon.

I've created TupleValue and MapValue atomic types, and Collections class exposing extension functions api. It's easy to use this api. I'll repeat an example that I was showing earlier:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:f="http://www.nesterovsky-bros.com/xslt/functions/public"
  xmlns:p="http://www.nesterovsky-bros.com/xslt/functions/private"
  xmlns:c="java:com.nesterovskyBros.saxon.Functions"
  exclude-result-prefixes="xs f p c">

<xsl:template match="/">
   <root>
     <xsl:variable name="tuples" as="item()*" select="
       for $i in 1 to 20
         return c:tuple(1 to $i)"/>

    <total-items>
      <xsl:sequence select="
        sum
        (
          for $tuple in $tuples return
            count(c:tuple-items($tuple))
        )"/>
    </total-items>
    <tuples-size>
      <xsl:sequence select="count($tuples)"/>
    </tuples-size>
    <sums-per-tuples>
      <xsl:for-each select="$tuples">
        <xsl:variable name="index" as="xs:integer" select="position()"/>

        <sum index="{$index}" value="{sum(c:tuple-items(.))}"/>
      </xsl:for-each>
    </sums-per-tuples>

    <xsl:variable name="cities" as="element()*">
      <city name="Jerusalem" country="Israel"/>
      <city name="London" country="Great Britain"/>
      <city name="Paris" country="France"/>
      <city name="New York" country="USA"/>
      <city name="Moscow" country="Russia"/>
      <city name="Tel Aviv" country="Israel"/>
      <city name="St. Petersburg" country="Russia"/>
     </xsl:variable>

    <xsl:variable name="map" as="item()" select="
      c:map
      (
        for $city in $cities return
        (
          $city/string(@country),
          $city
        )
      )"/>

    <xsl:for-each select="c:map-keys($map)">
      <xsl:variable name="key" as="xs:string" select="."/>

      <country name="{$key}">
        <xsl:sequence select="c:map-value($map, $key)"/>
     </country>
    </xsl:for-each>
  </root>
</xsl:template>

</xsl:stylesheet>

Download java source.

P.S. I would wish this api be integrated into Saxon, as at present java extension functions are called through reflection.

Download sample code.

In our recent .NET 2.0 GUI project our client ingenuously asked us to implement undo and redo facility. Nothing unusual nowadays, however it's still not the easiest thing in the world to implement.

Naturally you want to have this feature for a free. You do not want to invest too much time to support it. We had no much time to implement this "sugar" also. I know, I know, this is important for a user, however when you're facing a big project with a lot of logic to be implemented in short time you're starting to think it would be nice to have undo and redo logic that works independently (at least almost independently) on business logic.

Thus, what's that place where we could plug this service? - Exactly! - It's data binding layer.

When you're binding your data to controls the "Type Descriptor Architecture" is used to retrieve and update the data. Fortunately this architecture is allowing us to create a data wrapper (ICustomTypeDescriptor). Such wrapper should track property modifications of the data object thus providing undo and redo service. In short that's all, other are technical details.

Let's look at how undo and redo service goes into the action. Instead of:

you have to write:

There should also be a class to collect and track actions. User should create an instance of this class to implement the simplest form of code with undo and redo support:

Now turn our attention to the implementation of the undo and redo mechanism. There are two types in the core: UndoRedoManager and IAction. The first one is to track actions, the later one is to define undo and redo actions. UndoRedoManager performs either "Do/Redo", or "Undo" operations over IAction instances. We have provided two useful implementations of the IAction interface: UndoRedoTypeDescriptor - wrapper around an object that tracks property changes, and UndoRedoList - wrapper around the IList that tracks collection modifications. Users may create their implementations of the IAction to handle other undo and redo activities.

We have created a sample application to show undo and redo in action. You can download it from here.