TagSoup - Parse even the worst HTML as XML

 If you are in ever in need of parsing XML to retrieve some contents from a HTML file, you know how irritating it is. Generally, HTML is supposed to contain a well-formed XML. But it's quite often far from it most of the time. HTML files contain mismatched tags, missing end-tags etc. So, if you parse these HTML files with a popular Java XML Parser like Xerces, it will throw an exception, and will stop parsing. Here, I have used SAX interface with Java JAXP; the outcome is same for DOM.

[java] SaxParseException: The indexing file contains incorrect xml syntax.
[java] org.xml.sax.SAXParseException: The element type "div" must be terminated by the matching end-tag "</div>".[java]     at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
[java]     at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
[java]     at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
---

[java]     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
[java]     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

---
[java]     at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
[java]     at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
[java]     at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
[java]     at com.nexwave.nquindexer.SaxDocFileParser.parseDocument(SaxDocFileParser.java:96)

Why does this happen? Because the provided html file contained the following malformed xml.

This is <B>bold, <div>a div with bold text </b> plain div </div>normal text

Unfortunately, the html files in the web contain lot of syntax issues like this. TagSoup to the rescue. 

TagSoup

TagSoup is a SAX-compliant parser written in Java (C++ port is now available) that parses malformed html like the above according to a specially designed schema for html. And, TagSoup is a Free and Open Source software released under Apache License 2.0. (aah, I saw that little smile in your face ;-) ) Read more about it at tagsoup home.

How to use it?

TagSoup can be invoked directly via command-line. See http://home.ccil.org/~cowan/tagsoup/#program for more info. This blog post if primarily intended on showing you how to use it in Java. 

TagSoup provides SAX interface, so, if you are already familiar with SAX (Simple API for XML), and JAXP (Java API for XML Processing), then you are almost there. If not, read these up-to-the-point articles from IBM developerWorks; JAXP, SAX, Vendor independence in SAX.

If your project already have a SAX implementation designed to work with other parsers like Xerces, then you can just use TagSoup by changing two system properties, "org.xml.sax.driver", and "javax.xml.parsers.SAXParserFactory".

 

 java -Dorg.xml.sax.driver=org.ccil.cowan.tagsoup.Parser -Djavax.xml.parsers.SAXParserFactory=org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl  -cp myjar-1.0.jar:tagsoup-1.2.1.jar org.Main

//Runs the main class org.Main in myjar.jar using the tagsoup as the sax parser.

I use this in my Apache ANT script for the DocBook WebHelp Search Indexer . For that, I've added following content to my build.xml. Have a look at it here from line 91.

<java classname="org.Main" fork="true">

    <sysproperty key="org.xml.sax.driver" value="org.ccil.cowan.tagsoup.Parser"/>
    <sysproperty key="javax.xml.parsers.SAXParserFactory" value="org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl"/>

    <classpath>
        <path refid="classpath"/>
        <pathelement location="myjar-1.0.jar"/>

        <pathelement location="tagsoup-1.2.1.jar"/>
    </classpath>
</java>

As you can see, we can achieve vendor independence easily here. So, in case you want to use Xerces as your sax parser, then just replace "org.ccil.cowan.tagsoup.Parser" with "org.apache.xerces.parsers.SAXParser", and "org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl" with "org.apache.xerces.jaxp.SAXParserFactoryImpl".

How to run Java applets in Google Chrome under Ubuntu

I found that Google Chrome doesn't auto-detect Java applets in Linux. Following commands will link the Java applet plugin to Chrome. Enter these in Terminal. Make sure you put the correct path for the JRE in /usr/lib/jvm/<jdk1.6.0_22>/jre/lib/i386/libnpjp2.so.

sudo mkdir /opt/google/chrome/plugins
cd /opt/google/chrome/plugins

Now, link the plugin  libnpjp2 to /opt/google/chrome/plugins directory by running the following. Double check whether the JRE path is correct.

sudo ln -s /usr/lib/jvm/jdk1.6.0_22/jre/lib/i386/libnpjp2.so

Like to know whether this solution works in other Linux distros as well. Let us know in the comments.

Remote Debugging Java packages in WSO2 Carbon using IntelliJ IDEA

Here I'm going to tell you an easy way of debugging java packages deployed under any WSO2 product. So, you don't need to go wondering and go through thousands of lines looking what the bug was in your package.
What you need:

• IntelliJ IDEA

•  Download the latest version from http://www.jetbrains.com/idea/download/index.html#linux
•  Installation instructions can be found here.
• The WSO2 Product you are deploying the package in. All WSO2 Products are listed here.

Thats it. Now, open your Java package/bundle from IDEA.
Then, go to Run -> Edit Configurations
Add a remote debug configuration by clicking on '+' sign (at top left corner) and then remote.
It comes with a default set of settings. Keep them intact. get to know the port number(generally it's 5005)

Now, start the WSO2 Product you are using with the parameter of debug followed by the port number.

sh wso2server.sh debug 5005

It waits and listens on port 5005 until remote debugging client is started. Start the client by, (in IDEA)

Run -> Debug

All done. Now you can do normal debugging by putting break-points in your .java files.

That's how I got it done. Does this help you to succeed in remote debugging?

Send Ajax Requests from JavaScript easily with Prototype framework

Did you have a need of setting up an JSP Java variable in a JSP file inside a JavaScript code. Generally, this is not possible as JavaScript works on client-side and JSP Java compilation happens server-side. i.e. When a user request a JSP file, first, the Java codes in it is compiled and generates pure HTML (and JavaScript) code. So, if you want to set a Java variable or any other thing involving Java in the client-side, you have to send a request to the server saying what you have to do.

As, you know when submitting a form with POST or GET, it sends a POST or GET request with the fields in that form. But, it loads another page which takes time and may not be the case you need.

With AJAX you can send a request to server very easily.Here, I'm telling you to send a AJAX request with the use of prototype framework. Prototype offers a Object Oriented with extensive Ajax support which aims to ease development of dynamic web applications. The framework offers various features and enhancements for JavaScript. Here, we are looking sending a AJAX request. You can learn more about this at http://www.prototypejs.org/learn/introduction-to-ajax

• First download the prototype.js file from http://prototypejs.org/download
• Then copy it to a suitable location and add this code in your HTML: (Make sure to set the correct path in src)

 <script type="text/javascript" src="/path/to/prototype.js"></script>  

 Then you can send a AJAX request by simply creating a new Ajax object and calling it's request method.

 new Ajax.Request('/some_url', { method:'post' }); 

      Here is an example code:

      new Ajax.Request('pathToJSPFile.jsp',
            {

                 method:'post',
                 parameters:{para1:"para1Text", para2:"para2Text"},
                 onSuccess: function(transport) {
                       var response = transport.responseText;  
                       alert("success "+response);
                  },
                  onFailure: function(transport) {
                       alert("fail");   
                  }
        });
  Just put above code in a method and call it when necessary. I think the code is self explainable.

Install IntelliJ IDEA on Linux

Here's a simple way to install IntelliJ IDEA on Ubuntu. I've done this on all Ubuntu Karmic Koala, Lucid Lynx and Maverick Meerkat. This worked just fine on Gentoo Linux too.

1). Install Java (Use Sun JDK. OpenJDK is NOT recommended as the JDK.)
sudo apt-get install sun-java5-jdk
It seems sun-java5-jdk is no longer available now. Refer Oracle Java SE Downloads for Linux and it's installation instructions. Keep the path in your memory; you'll need it when setting up $JDK_HOME path variable.

Then, to make sure that it is installed properly, type
java -version
if the version shows, then it is installed successfully.

2) Now, install IDEA
Download IntelliJ IDEA from the below link
 http://www.jetbrains.com/idea/download/index.html#linux
 Unpack the downloaded archive (ideaIU-9.0.3.tar.gz) by,
     tar xfz ideaIU-9.0.3.tar.gz
(match the file name to your version!)

 Move the extracted archive to /usr/lib
     sudo mv idea-IU-95.429 /usr/lib

3) Configure PATH variables for JDK_HOME and IDEA
     sudo gedit ~/.bashrc
Add following for JDK_HOME to the file, make sure to change the path to reflect your java installation directory.

export JDK_HOME=/usr/lib/jvm/java-6-sun

Append the line for PATH variable with this, /usr/lib/idea-IU-95.429/bin . Finally, it should look like this.

export PATH=$PATH:/usr/lib/idea-IU-95.429/bin
Now, run `source ~/.bashrc` to take effect the changes.

4) Run IDEA  by entering following in the terminal
   idea.sh (or sh idea.sh)

NOTE: If you get permission denied error, then you will have to set the correct permissions to idea.sh file by `sudo chmod 755 /usr/lib/idea-IU-95.429/bin/idea.sh` 

For easier execution, create a "Custom Application Launcher" at top panel. Access it via; right-click the top Panel -> "Add  to Panel" -> double click "Custom Application Launcher". Browse and give an icon if you like. Then, use a command similar to,
sh /usr/share/idea-IU-95.66/bin/idea.sh

Custom Application Launcher Properties

If you have any problem in setting up this, drop a comment below. Well, even if you got right, just tell me so. Love to hear that too! :). Good Luck!