Friday, November 25, 2011

Siddhi: A Second Look at Complex Event Processing Architectures

The paper titled "Siddhi: A Second Look at Complex Event Processing Architectures" has been presented in Proceedings of the Gateway Computing Environments Workshop (GCE) at Supercomputing 2011 in Seattle, WA, USA. Siddhi was a final year project of '07 batch of University of Moratuwa by a group comprising of Suho, Isuru, Subash, and me. Dr. Srinath Perera, and Ms. Vishaka Nanayakkara supervised the project.

The abstract of the paper is as follows -

Today there are so much data being available from sources like sensors (RFIDs, Near Field Communication), web activities, transactions, social networks, etc. Making sense of this avalanche of data requires efficient and fast processing. Processing of high volume of events to derive higher-level information is a vital part of taking critical decisions, and Complex Event Processing (CEP) has become one of the most rapidly emerging fields in data processing. e-Science use-cases, business applications, financial trading applications, operational analytics applications and business activity monitoring applications are some use-cases that directly use CEP. This paper discusses different design decisions associated with CEP Engines, and proposes some approaches to improve CEP performance by using more stream processing style pipelines. Furthermore, the paper will discuss Siddhi, a CEP Engine that implements those suggestions. We present a performance study that exhibits that the resulting CEP Engine Siddhi has significantly improved performance.  Primary contributions of this paper are performing a critical analysis of the CEP Engine design and identifying suggestions for improvements, implementing those improvements through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence.

Goals of this paper are to critically evaluate decisions taken at CEP design and to present Siddhi CEP Engine that incorporates several improvements uncovered at the discussions. To that end, next section will present a survey of CEP Engine design, and Section 3 will describe Siddhi. Thereafter Section 4 will present a performance comparison of Siddhi and Esper, where latter is an open source, established CEP Engine. The primary contributions of this paper are critical analysis of the CEP Engine design and identifying area for improvements, implementing those improvements, and demonstrating the soundness of those suggestions through empirical evidence. The paper describes how those improvements are achieved through a new CEP called Siddhi.

If you like to contribute to the project, join us

Thursday, October 20, 2011

Gentoo Maven Integration - Progress Report

Here's my progress report for the first phase of Gentoo Maven Integration project for 
finishing the work under Google Summer of Code and starting move in to 
a voluntary position. 

The goal of this project was to build maven and it's huge number of 
dependencies from source, and then facilitate the packagers for 
packaging maven-based Java packages. There are two eclasses which will 
facilitate bootstrapping maven along with building maven-based 
packages, and packaging Maven plugins. These eclasses address some 
fundamental issues of incompatibilities between Gentoo build system 
and Maven build system. 

There were two main goals for the project. One is building 
maven-from-source. It is now completed and has been thoroughly tested. 
There are around 40+ ebuilds that are direct dependencies of Maven 
which were packaged/bumped during the project period. General users of 
maven can have the full benefit from this package now. Please file 
bugs at if you find any. 

The second phase was a lengthy process and the scope wasn't fit for 
one and half months time. But with mentors' blessing, I've made a 
quite a big progress and was able to emerge a minimal package built 
via native Maven. 

Let me describe the surface details of the second phase. The idea was 
to facilitate the packagers to package maven-based packages. This has 
been a long-time blocker for Gentoo-Java (which extends to more than 
3-4 years). For this phase, we needed several requirements including 
dependency management issues and rewriting of pom.xml to match 
Gentoo's needs. One requirement in it was the need to have a mechanism 
to use the installed system jars instead of downloading the jars from 
maven repos. One another is that pom needs to respect the Gentoo SLOT 
system. Further, configuration details needed to be added to tell the 
JDK and JRE versions needed for building (ie need to add config bits 
to maven-compiler-plugin section in the pom). And, then it needs 
several maven plugins to build packages. There were enormous amount of 
plugins available that most of them need special attention separately. 
For the second phase, the hard part is over. And, as I said, I was 
able to emerge a minimal maven-based implementation. Maven isn't much 
cooperative when it comes to dependency management, but our solution 
worked well. 

Along with that, the first iteration of work is complete. I'm hoping 
to be the maintainer for Maven under Gentoo Java herd for the 
foreseeable future. And, I'm eagerly waiting to wear the Gentoo 
Developer hat one day. I'm interested in knowing the generic plan for 
recruiting developers who come via Summer of Code as well. 

There are few things to be done to bring the use of the Maven 
integration to it's full potential. These are more like plug-ins to 
the core base, and beautifying the process. I need to make new plans 
for these with help from Java herd. 

 * There's only five maven plugins has been packaged. Have a fresh 
look at maven-surefire-plugin. Needs to add all other plugins. 

 * Currently, when MAVEN_PARENTPOM_UNIQUE_ID is set to rewrite 
<parent> node of the pom, it rewrites all the poms in the project 
including sub-modules. The most probable usecase is that rewrite the 
parent element of the top-level/aggregator pom. The configuration bits 
needed are already there (-w option), but the implementation needs to 
be done. 

 * Merge the separate ebuilds of  maven-2.2.1 maven-2.2.1 release in 
to one. There are around 20+ ebuilds dedicated for this. These ebuilds 
probably won't be needed separately so it's ok to merge these 
together. Need to evaluate possibility of issues of having all these 

Here are some references if you are interested in getting deeper in 
Maven in Gentoo. Feel free to contact me if you like to extend your 
helping hand for the project. I'm at kasunbg +spamfree at
 * The wiki 1 - Developer and User guide for Maven in Gentoo -
 * The wiki 2 - Manpage for java-maven-2 eclass -
 * Repository 1 - gsoc-maven-overlay -
 * Repository 2 - Branch for Javatoolkit -
       * TracBrowser view -

Sunday, October 9, 2011

Siddhi CEP has won a bronze award at NBQSA 2011

Siddhi Complex Event Processing Engine has won the bronze award in Tertiary category at the recently concluded National Best Quality Software Awards 2011 held in Hotel Galadari, Colombo, Sri Lanka. The winners are now published at NBQSA official website. Further, Siddhi has been nominated to APICTA (Asia Pacific ICT Awards) 2011 which will be held during November 8 – 11, 2011 in Thailand.

Siddhi was our final year project at Dept. of Computer Science & Engineering, Faculty of Engineering, University of Moratuwa. It was a four member project with other members being Suho, Isuru, and Subash. We were overwhelmed with joy on the day we heard that we have secured a place at NBQSA. They weren't specific about the exact award we got. So, we have been waiting with nervously, and full of excitement for the moment for winning the award. It's really a pleasure to know that a year old hard work has gained benefits.

Along with us, another two other projects from our department has won a merit, and a special recognition award. Further, another two projects from Dept. of Electronic & Telecommunication of University of Moratuwa also able to secure a merit and a special recognition award. In total, it's five awards.

Special thanks should go to our supervisors, Dr. Srinath Perera, and Ms. Vishaka Nanayakkara who have given us enormous support. Our project coordinator Dr. Shantha Fernando, Dr. Sanjiva Weerawarana who has provided us guidance when in need should also be remembered.

The Siddhi Team (from left-to-right Suho, me, and Subash; Isuru missed it!)

Award winners at NBQSA 2011

TagSoup - Parse even the worst HTML as XML

 If you are in ever in need of parsing XML to retrieve some contents from a HTML file, you know how irritating it is. Generally, HTML is supposed to contain a well-formed XML. But it's quite often far from it most of the time. HTML files contain mismatched tags, missing end-tags etc. So, if you parse these HTML files with a popular Java XML Parser like Xerces, it will throw an exception, and will stop parsing. Here, I have used SAX interface with Java JAXP; the outcome is same for DOM.

[java] SaxParseException: The indexing file contains incorrect xml syntax.
[java] org.xml.sax.SAXParseException: The element type "div" must be terminated by the matching end-tag "</div>".
[java]     at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
[java]     at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
[java]     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
[java]     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
[java]     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
[java]     at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
[java]     at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
[java]     at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
[java]     at com.nexwave.nquindexer.SaxDocFileParser.parseDocument(

Why does this happen? Because the provided html file contained the following malformed xml.

This is <B>bold, <div>a div with bold text </b> plain div </div>normal text

Unfortunately, the html files in the web contain lot of syntax issues like this. TagSoup to the rescue. 


TagSoup is a SAX-compliant parser written in Java (C++ port is now available) that parses malformed html like the above according to a specially designed schema for html. And, TagSoup is a Free and Open Source software released under Apache License 2.0. (aah, I saw that little smile in your face ;-) ) Read more about it at tagsoup home.

How to use it?

TagSoup can be invoked directly via command-line. See for more info. This blog post if primarily intended on showing you how to use it in Java. 
TagSoup provides SAX interface, so, if you are already familiar with SAX (Simple API for XML), and JAXP (Java API for XML Processing), then you are almost there. If not, read these up-to-the-point articles from IBM developerWorks; JAXP, SAX, Vendor independence in SAX.

If your project already have a SAX implementation designed to work with other parsers like Xerces, then you can just use TagSoup by changing two system properties, "org.xml.sax.driver", and "javax.xml.parsers.SAXParserFactory".
 java -Dorg.xml.sax.driver=org.ccil.cowan.tagsoup.Parser -Djavax.xml.parsers.SAXParserFactory=org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl  -cp myjar-1.0.jar:tagsoup-1.2.1.jar org.Main
//Runs the main class org.Main in myjar.jar using the tagsoup as the sax parser.

I use this in my Apache ANT script for the DocBook WebHelp Search Indexer . For that, I've added following content to my build.xml. Have a look at it here from line 91.

<java classname="org.Main" fork="true">
    <sysproperty key="org.xml.sax.driver" value="org.ccil.cowan.tagsoup.Parser"/>
    <sysproperty key="javax.xml.parsers.SAXParserFactory" value="org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl"/>
        <path refid="classpath"/>
        <pathelement location="myjar-1.0.jar"/>
        <pathelement location="tagsoup-1.2.1.jar"/>

As you can see, we can achieve vendor independence easily here. So, in case you want to use Xerces as your sax parser, then just replace "org.ccil.cowan.tagsoup.Parser" with "org.apache.xerces.parsers.SAXParser", and "org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl" with "org.apache.xerces.jaxp.SAXParserFactoryImpl".

Sunday, March 13, 2011

How to run Java applets in Google Chrome under Ubuntu

I found that Google Chrome doesn't auto-detect Java applets in Linux. Following commands will link the Java applet plugin to Chrome. Enter these in Terminal. Make sure you put the correct path for the JRE in /usr/lib/jvm/<jdk1.6.0_22>/jre/lib/i386/

sudo mkdir /opt/google/chrome/plugins
cd /opt/google/chrome/plugins
Now, link the plugin  libnpjp2 to /opt/google/chrome/plugins directory by running the following. Double check whether the JRE path is correct.
sudo ln -s /usr/lib/jvm/jdk1.6.0_22/jre/lib/i386/

Like to know whether this solution works in other Linux distros as well. Let us know in the comments.

Monday, February 21, 2011

ගූගල් සමර් ඔෆ් කෝඩ් 2011 සදහා ශ්‍රී ලංකාවේ ප්‍රවර්ධන සැසි සහ අත් පත්‍රිකා

ඔන්න හා හා පුරා කියල සිංහලෙනුත් බ්ලොග් පෝස්ට් එකක් දාන්නයි මේ ලැහැස්තිය :)
මුලින්ම දාන්න යන්නේ නම් ගූගල් සමර් ඔෆ් කෝඩ් (GSoC) 2011 වෙනුවෙන් හදපු සිංහල අත් පත්‍රිකාව ගැනයි. මේ දවස් වල GSoC උණුසුම හුඟක් අයට දැනිල ඉවරනේ... දැන් අපේ කට්ටිය බරටම FOSS මෘදුකාංග තෝරාගෙන ඒවාට දායක වෙන්න (contribute) පටන් අරන්. අපේ කැම්පස් එක පහු ගිය අවුරුදු වල GSoC වලින් අංක එකේ හිටපු හින්ද [1] [2] ලොකු උනන්දුවක් තියනවා මේ වැඩසටහන ගැන. ඒ ගැන නම් හරිම සතුටුයි, විශේෂයන්ම මටත් ඒ සදහා දායක වෙන්න ලැබීම ගැන. 2010 අවුරුද්දේ ශ්‍රී ලංකාව දහ වෙනි තැන ලබාගත්තා... GSoC කියන්නේ මොකක්ද කියල දැන ගන්න කැමති නම් මෙන්න මේක කියවන්න.
හරි ඉතින් තමන්ට විතරක් කරගෙන හරි යන්නේ නැහැනේ, අනිත් අයටත් උදව් කරන්න එපැයි. :) මේ අවුරුද්දේ එක නිසා මමයි, මගේ බැච් එකේ GSoC කරපු කීප දෙනෙකුයි, සිනියර් බැච් එකේ ප්‍රදීබන් අයියයි එකතු වෙලා අපිට හැකි පමණින් වැඩ කොටසක් කළා. ලොකුවටම කරපු දේ තමයි GSoC ගැන දැනුවත් කිරීමේ සැසි දෙක. ඒවා ගැන දැනගන්න මෙන්න මේක කියවන්න; මේ තියෙන්නේ "FOSS User" සගරාවේ එකේ මෙම සැසි දෙක ආවරණය කරමින් පළවුණු ලිපිය. :) 
ඒ දෙකෙන් පේරාදෙණිය විශ්වවිද්‍යාලයේ තිබුන සැසිය මට විශේෂයි. ඒ වෙන මොකක් හින්ද වත් නෙවෙයි එකට මම ලොකු සහයෝගයක් ලබාදුන්නා සංවිධාන කටයුතු වලට. :) පේරාදෙණිය පැත්තෙන් රසික, මේ සැසිය පවත්වන්න සෑහෙන්න මහන්සි ගත්තා. සම්පත් දායකයින් විදියට සහභාගී වූ ප්‍රදීබන්, නිර්මාල්, තිලංක, කසුන් ලක්ප්රිය අයව අනිවාර්යෙන්ම මතක් කරන්න ඕනේ. අපි ටික තමා මේ බර ඔක්කොම ඇද්දේ! ඒ වගේම IESL හිදී තිබුණු සැසිය සදහා දැන් නම් කිව්ව අය ඇරෙන්න උමාශාන්ති, කෙහෙළිය ඇතුළු සුහෝ යන අය විශාල වැඩ කොටසක් කළා.
අත් පත්‍රිකාවක් කිව්වට මොකද අන්තිමට පුණ්‍ය අනුමෝදනාවක් වගේ උනා නේද? ;)) තරහ වෙන්න එපා, මෙන්න තියනවා හදපු සිංහල අත් පත්‍රිකාව. සෝදුපත් බලන්න උදව් උනා කෙහෙළිය, අකිල, නිර්මාල්. මෙන්න ලින්ක් එක.
උමාශාන්ති දෙමල අත් පත්රිකාවකුත් හැදුවා, ඒක මෙන්න මෙතන ඇති.