Officially a DocBook Mentor for Google Summer of Code 2012

I have got the opportunity to be a mentor for DocBook organization under Google Summer of Code 2012. I was a former participant of Google Summer of Code 2010, and 2011. In 2010 I worked with DocBook as a student with my mentor David Cramer where I developed the DocBook WebHelp output format. I always fascinated contributing to an Open-Source project, so it was a big achievement for me. I kept contributing to it which made me a part of the community, and a Committer for the project.

As many of you probably know, Google Summer of Code is an annual program administered by Google, in which Google sponsor students to complete a free and open-source software coding project. This is eighth iteration of GSoC. Most of the giants in Open-Source has been part of it including GNU, Gentoo, Apache, KDE, Ubuntu, Mozilla with student participants from all over the world.

DocBook has secured five slots this time. We had enough resources to mentor five slots, on top of that there were five good proposals to mentor for. Fortunately, there wasn't any issue with de-duplication or from anything for that matter, so all five were selected into DocBook. This time, as usual, Richard, and Stefan Seefeld are the org admins. Stefan Seefeld, David, Jirka, Steve, and me are primary mentors for the five students while Stefan Hinz, and Peter Desjardins will be co-mentors for the two selected webhelp proposals.
Yes, two proposals were for WebHelp. That made this year's program more exciting. The two students were, Visitha, from University of Moratuwa, Sri Lanka (my Alma Mater), and Arun, from PES Institute of Technology Bangalore, India. There were overlap between them, so we had to sorted those out first before selecting both.

I found Visitha during the Google Summer of Code meetup at University of Moratwa we, past Sri Lankan gsocers, organized. Buddhi, who also attended that meetup has selected into DocBook. It's so glad to hear that people appreciating the effort we put there, and that it actually meant a lot for most.

I must mention Gihan Chanuka here. He contacted me on last January where he wanted to contribute to DocBook outside of GSoC. He has been asking various questions he had while trying out DocBook. Though he wasn't able to do that, he was a good candidate for GSoC. I am delighted that he also got selected into flying colors. Of course there is a long way to go from selecting to completion, but when it comes to GSoC, selection is a special milestone to pass.

Hoping to have a great time being a mentor this year!

DocBook WebHelp UI features by screenshots!

I wrote this blog post to show some of the UI features provided by DocBook WebHelp. So, following is a set of screenshots I took which which is how webhelp will look like in the next release. The next release would probably be versioned as 1.77.x. These are taken from revision r9240 on 2012-03-16. You may download the latest snapshot from DocBook snapshots site. You may download the latest release here.

A chunked output page docbook webhelp. It includes a header in top with a company logo, Contents, and Search panes at left side; the real contents in the right larger section; And some minor but useful features at top-right corner. The browser is Chromium/Google Chrome 16. Click on the screenshot for a larger view.

You may hide/unhide the left sidebar by clicking on  SIDEBAR at top-right corner in the page. Isn't that a easier way to read the contents? or do you prefer the default way? For larger monitors the effect is minimal!

A zoomed view the Contents pane in left. This shows the Table of Contents of the documentation, broken down by chapters/sections. You can configure the number of tree levels if you think it's necessary.

Plain search view. You can turn on/off the Search Highlighter provided. Try it yourself!

A typical search results section.  Read here to understand how the search result summaries appeared! And, we provide search scoring/rating mechanism for better searching.

A search with several words. WebHelp displays the results for the whole query, as well as for individual words which is nicely sectioned as shown above.

How to Use docbkx-tools as Maven DocBook plugin

When it comes to documentation of open source projects, DocBook is the favorite choice among many developers. As you all know, DocBook provides a comprehensive XML schema to document your project, which you can transform to any format such as html, pdf, webhelp etc. Here, I'm going to show you how to integrate DocBook in to your Maven project via docbkx-tools. This docbkx-maven-plugin makes docbook integration to Apache Maven much easier.

Reference DocBook XSLs from your customization layer

Probably you are not planning to use DocBook standard stylesheets alone, but have a little more customization to it. For that, you have to create a new customization layer, and import the standard docbook.xsl stylesheet. 

It's a little tricky how it's done in this plugin. You should give a symbolic reference rather than an actual file location when importing the stylesheets. You should use <xsl:import href="urn:docbkx:stylesheet/docbook.xsl" /> to give the symbolic reference.

What "urn:docbkx:stylesheet" refers will depend on what customization (html, xhtml, etc.) you want to use. If you want to use xhtml/chunk.xsl customization, first specify it in your XSL by     <xsl:import href="urn:docbkx:stylesheet/chunk.xsl"/> and then, in your POM, specify the <xhtmlCustomization>. Further details are explained below.

Integrating dockbkx-tools to Maven projects

OK, now let's see how to integrate this nice Docbook plugin to your maven project.

• First, you have to add the docbook plugin to your parent POM file pom.xml. The docbook xml documentation should be on path trunk/src/docbkx.

    <build>
        <plugins>
            <plugin>
                <groupId>com.agilejava.docbkx</groupId>
                <artifactId>docbkx-maven-plugin</artifactId>
                <version>2.0.11</version>
                <dependencies>
                    <dependency>
                        <groupId>org.docbook</groupId>
                        <artifactId>docbook-xml</artifactId>
                        <version>4.4</version>
                        <scope>runtime</scope>
                    </dependency>
                </dependencies>
                <configuration>
                    .......
                </configuration>
                <executions>
                    ......
                </executions>
            </plugin>
        </plugins>
    </build>

As you can see, the plugin depends on the docbook-xml artifact. After adding this structure to your POM, you just have to customize it according to your needs.

• Then, you have to set the configurations in <configuration> tag. In that, you should menion the xsl customization file, if you have a customized docbook XSLs. Else, you can just use the generic XSLs provided by DocBook. At this point, the docbkx's version is 2.0.11 and it uses DocBook-XSL-NS 1.75.2  version. Mind that it's the namespace version, so make sure you make your customized version with NS too. Else, you may encounter errors which could be hard to track. For the customizations, for html, use <htmlCustomization> tag to mention the customizations. For PDF, it's <foCustomization>. and <xhtmlCustomization> for, obviously, xhtml. A sample is below. You may put configurations under <executions> too, if you wish.
• The new customization to the DocBook-XSL family, DocBook WebHelp, is not yet included on this version, but the people are working on it to make it available as soon as possible. I'm eagerly waiting for it, mainly because I contributed to it a lot; plus it's a nice customization which you can actually put in the web for your audience to see.

        <configuration>
            <foCustomization>src/docbkx/xsl/fo.xsl</foCustomization>
            <xhtmlCustomization>src/docbkx/xsl/xhtml.xsl</xhtmlCustomization>
        </configuration>                

• Now it's to set the 'executions' you need to run. Executions specify what goals you run, copy CSS/JavaScript files you might have etc. A sample execution code is given below.

        <executions>

            <execution>
                <id>userguide</id>
                <phase>pre-site</phase>
                <goals>
                    <goal>generate-xhtml</goal>
                </goals>
                <configuration>
                    <includes>userguide.xml</includes>
                    <targetDirectory>target/site/userguide</targetDirectory>
                    <chunkedOutput>true</chunkedOutput>
                    <xhtmlCustomization>src/docbkx/xsl/userguide.xsl</xhtmlCustomization>
                    <postProcess>
                        <copy todir="target/site/userguide/">
                            <fileset dir="src/docbkx/template">
                            </fileset>
                        </copy>
                    </postProcess>
                </configuration>
            </execution>
        </executions>  

What happens there? First, it specifies the phase 'pre-site', which tells to Maven that, this execution should be run first when invoking 'mvn site' command. 

• <goal> - is set to generate-xhtml which will generate a xhtml output. Other options are generate-html and generate-pdf.
• <includes> - This tag specifies which document should be processed. The path was trunk/src/docbkx/userguide.xml.
• <configuration> - Then it comes to the configuration part. Most of it are self understandable. As you can see I've set the XSL customization I wish to use there. You can put it anywhere, both worked fine for me. 
• <chunkedOutput> will generate set of html files rather than one BIG html file which are broken down based on the tags <chapter>, <section> etc. in your docbook xml file.
• I copied all the CSS/JavaScript/Images to target folder by using the <postProcess> tag. It's easier. The source files were at src/docbkx/template.

• You may write another <execution> to generate the PDF output. The whole code I used is given below.

        <build>
            <plugins>
                <plugin>
                    <groupId>com.agilejava.docbkx</groupId>
                    <artifactId>docbkx-maven-plugin</artifactId>
                    <version>2.0.11</version>
                    <dependencies>
                        <dependency>
                            <groupId>org.docbook</groupId>
                            <artifactId>docbook-xml</artifactId>
                            <version>4.4</version>
                            <scope>runtime</scope>
                        </dependency>
                    </dependencies>
                    <configuration>
                        <foCustomization>src/docbkx/xsl/fo.xsl</foCustomization>
                    </configuration>
                    <executions>
                        <execution>
                            <id>userguide</id>
                            <phase>pre-site</phase>
                            <goals>
                                <goal>generate-xhtml</goal>
                            </goals>
                            <configuration>
                                <includes>userguide.xml</includes>
                                <targetDirectory>target/site/userguide</targetDirectory>
                                <chunkedOutput>true</chunkedOutput>
                                <xhtmlCustomization>src/docbkx/xsl/userguide.xsl</xhtmlCustomization>
                                <postProcess>
                                    <copy todir="target/site/userguide/">
                                        <fileset dir="src/docbkx/template">
                                        </fileset>
                                    </copy>
                                </postProcess>
                            </configuration>
                        </execution>


                        <execution>
                            <id>userguide-pdf</id>
                            <goals>
                                <goal>generate-pdf</goal>
                            </goals>
                            <phase>compile</phase>
                            <configuration>
                                <includes>userguide.xml</includes>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>

YBQQQSEANSZ5

DocBook WebHelp Project

DocBook WebHelp was the project I worked on for the Google Summer of Code 2010 program. Pencil down date for it was on 16th August, 2010, which means the Coding officially finished on that day. So, I with my mentor David Cramer finished all the requirements planned, and wrote all the documentation needed. Results was announced today, 21st August by Google Open Source Program team; I successfully finished the project :)

The demo of the output produced by WebHelp XSL customization is available on following link. The demo shown is the documentation of DocBook WebHelp.
http://docbook.sourceforge.net/release/xsl/current/webhelp/docs/ch01.html
http://www.thingbag.net/docbook/gsoc2010/doc/content/ch01.html

The latest output in the snapshots have lot more features and looks quite beautiful compared to the released version. Do have a look -
http://snapshots.docbook.org/xsl/webhelp/docs/index.html
http://vulture.gentoo.org/~kasun/docbook/docbook-webhelp-snapshot-current/content/ch01.html

WebHelp Output

WebHelp Search tab

You can download DocBook installation from,
http://sourceforge.net/projects/docbook/

The Webhelp customization is available under, docbook-xsl-ns-1.76.1/webhelp. Following is some brief details about the DocBook WebHelp customization.

A common requirement for technical publications groups is to produce a Web-based help format that includes a table of contents pane, a search feature, and an index similar to what you get from the Microsoft HTML Help (.chm) format or Eclipse help. If the content is help for a Web application that is not exposed to the Internet or requires that the user be logged in, then it is impossible to use services like Google to add search.

DocBook WebHelp provides a browser-independent, platform-independent documentation “Web Help” output format for DocBook files. WebHelp provides a sophisticated but inexpensive web publishing option for DocBook.

Features

• Full text search.
• Stemming support for English, French, and German. Stemming support can be added for other languages by implementing a stemmer.

• Support for Chinese, Japanese, and Korean using code from the Lucene search engine.
• Search highlighting shows where the searched for term appears in the results. Use the H button to toggle the highlighting on and off.
• Search results can include brief descriptions of the target.

• Table of Contents (TOC) pane with collapsible toc tree.

• Autosynchronization of content pane and TOC.
• TOC and search pane implemented without the use of a frameset.
• An Ant build.xml file to generate output. You can use this build file by importing it into your own or use it as a model for integrating this output format into your own build system.

So, what do you think of the output? Are you interested to give it a try?

Follow up of my posts related to DocBook WebHelp are here
Follow up of my posts related to DocBook are here

PS: For discussions, please subscribe to my comment feed such that you won't miss my replies. Alternatively contact us via docbook-apps list - http://www.oasis-open.org/mlmanage/index.php

Proposal "Web Help Output for DocBook" for GSoC 2010

UPDATE: DocBook WebHelp XSL Customization is now integrated DocBook XSL release starting from version 1.76.0. The release is available for download at http://docbook.sourceforge.net/snapshots/. Release notes are at DocBook WebHelp Project (22/10/2010)

UPDATE: Google Summer of Code 2010 program finished on 20th August. See DocBook WebHelp Project for the end notes, features and to view the demo of the beta release. 

The modified schedule can be found in WebHelpGsoc2010. Though the schedule is not necessary at this moment it may give an idea of the development process went on which might help for a new developer coming in to WebHelp.

- 23/08/2010

Google Summer of Code 2010 - Project Proposal

Project	WebHelp Output for DocBook
Student Name	Kasun Gajasinghe
Email	******
IM	kasun (irc://irc.freenode.net)
Time zone
Mentors	David Cramer, Jirka Kosek

DocBook is a set of standards and tools for technical documentation. A vital requirement for technical publications is to produce a Web-based help format that is synchronized with the content. So the documentation is up-to-date making site maintenance easier. This will contain client-side searching with support for stemming, table of contents, Index and a HTML export ability. The main idea is to generate a Web Help Output from the DocBook content XML files using an Ant build.

Background
About me
---
Participating in GSoC
I am passionate about Open Source World and love contributing to free software. I hope that Google Summer of Code will be a great opportunity for me to become part of another open source community, contribute for the development of the project, make new friends, and develop new skills. I believe that GSoC will be a excellent starting point for this.
Why DocBook
DocBook is a leading format for documentation and is especially popular with Open Source projects. So, I am particularly interested in DocBook and hope to become a permanent member of DocBook project.
I have planned to devote 35-40 hours per week for this project.

Experience
----
 
Explorations
I have researched with suggestions from my co-mentors on ways to implementing client-side searching and came up with with following options.

Use Lucene QueryParser
Use the Java Indexer of the htmlsearch demo plugin as a base and add needed features

As Lucene works in Server-side, we have to compile it into JavaScript to make it work in client-side. For that Jirka suggested the use of GWT. But unfortunately Lucene isn't ported to GWT yet. I've looked at it and found that Luke, the Lucene Index toolbox has ported to GWT. Then, I went to Lucene IRC channel to get further details (#lucene @ freenode). Their I found that it is not possible to use Lucene only in client-side. They said that having queryparser in JavaScript can not be done and Luke uses a Java back-end in server-side for searching. So have to give up this option.
Java indexer is a good starting point. It does basic indexing and stores it in js files with keys (words) and their relevant file names. Then, it does basic searching based on given key words. This could be used as the base and improve the code and add new features. I have downloaded the source code and studied it. The proposed enhancements are listed under Proposal section.
For Table of Content tree generation, Considered,

Frameset approach with the tree included in a separate file.
Generate complete toc for every generated files and make it appear to be a pane

I chose the second method. As with that, "Deep linking" happens automatically and will be functional to some good extent under a no javascript environment. And this is the method mentor recommended. Further, researching will happen in the following days.

Proposal
DocBook is a set of standards and tools for technical documentation. It was initially and is primarily intended for technical documentation, but has been extended for use in other domains. The current DocBook schema is available in several languages including RELAX NG and DTD and is maintained by the DocBook Technical Committee of OASIS. The DocBook Open Repository is a project hosted on SourceForge that maintains a set of XSL stylesheets for converting a DocBook instance into a variety of output formats, including various html formats, pdf (via XSL-FO), man pages. The currently supported html output formats include monolithic html, chunked html, Microsoft HTML Help (.chm), Eclipse documentation plugins, and Java Help.

This proposal is to add a browser-independent, platform-independent documentation “Web Help” output format using a combination of HTML, CSS, and JavaScript with a search index created at build time by an indexer application written in Java.

Search is done in client-side. For that, I plan to use the “htmlsearch1.04” demo plugin from DITA Open Toolkit as a base and enhance it with the needed features. As DocBook is included as one of their supported products, it will be compatible for this project. Further, it's license allows the use of it in commercial applications as well.

The proposed design for searching is, first generate JavaScript files which contains all glossary terms of the html files with matched file locations (i.e. as key, value pairs). Then, for a given query, keywords are extracted and then by the use of generated glossary/index, the output will be displayed.
The enhancements currently planned are,

• Support for stemming and lemmatization for a given query
• Search with Boolean operators (AND, OR)
• Meta-data such as 'Prev' and 'Next' in the content page will be ignored when indexing.
• Improve support for Asian Languages (Japanese and other Asian languages, meta tag content is used.)
• As searching in client-side may slow-down the application, necessary optimization will be adopted.

I plan to use YUI library for the TOC tree generation. I will abandon the frameset-based approach and instead use a CSS-based mechanism in which the TOC is generated in every page and CSS is used to properly format it for viewing. With this approach, synchronization with the content file happens automatically. Further with this mechanism deep-linking happens automatically.
UI design will be developed using CSS and other technologies and will be little similar to Eclipse Help.
The Planned development schedule is given below.

Development Schedule
I am already proficient in Java, JavaScript, XML and CSS, but will start studying XSL in the bonding period and continue to learn it while doing the programming.

Community Bonding Period: April 26 - May 24	Get to know the mentor and the community Study the required API and features for WebHelp Preparing the development environment  Look for a good searching approach. Start designing a good model
Interim Period: May 24 - July 12	Dividing the development process into stages with the help of the mentor Developing the TOC tree using a CSS-based mechanism (YUI) Implementing the synchronization with the content Adding an index with the help of DocBook schema Designing client-side search mechanism with all the things such as stemming and lemmatization into consideration and start coding.  Designing a better user interface.
July 12 - July 16	Submitting mid-term evaluations and continue with the development
Interim Period: July 16 - August 9	Completing TOC with synchronization Continue developing the search mechanism Testing the synchronization and searching Developing the User Interface
August 9 - August 16	Refine the code and testing the code and doing necessary improvements.
August 20	Final evaluation deadline
August 30	Submitting required code to Google

References and Resources

[1] My Blog: Kasun's Tech Thoughts http://kasunbg.blogspot.com
[2] Twitter: http://twitter.com/kasunbg
[3] My Google Code Hosting Profile
(Projects hosted: documentation-aggregation-application, KFinder:A file searcher, cse-checkers(Java), cse-l3-2009-070137m:A Firefox extension)
[4] DocBook 5.0: The Definitive Guide
[5] DocBook XSL: The Complete Guide
[6] dita-users · DITA users yahoo group
[7] YUI Library
[8] Documentation Aggregation Application
[9] Delicious Extension for Google Chrome
[10] University of Moratuwa, Sri Lanka
[11] Deparment of Computer Science & Engineering, Faculty of Engineering