Setup WSO2 Deployment Synchronizer in WSO2 Application Server

This post describes how to setup Deployment Synchronizer for a WSO2 Application Server cluster. The process would be the quite the same for WSO2 Enterprise Service Bus (ESB) as well. Here, we are going to have two AS cluster nodes (master, and one slave) with a shared registry (WSO2 Governance Registry).

Deployment Synchronizer (DepSync) is a recent addition to WSO2 products. It provides the ability to sync the product cluster. We use it to keep the axis2 repository of our products (by default, it's product-home/repository/deployment/server/) in sync between all the cluster nodes. Since axis2 repository includes the user-uploaded service archives, modules, service metadata etc, it's important that all the cluster nodes have the same content. DepSync takes care of this.

Pre-requisites:
WSO2 Application Server 4.1.2+, and Goverance Registry.

• First, follow the instructions in Application Server documentation on Clustering Configuration to setup a basic cluster environment.  Usually, you will need a shared registry for your production deployment. But for testing purposes of WSO2 Application Server, you may ignore this step. This is needed for DepSync in WSO2 ESB though.

• Enable the "State Replication Configuration" for both nodes as shown here. That is  done by setting the enable attribute value of clustering element to true in axis2.xml file. axis.xml file can be found at AS-HOME/repository/conf/axis2.xml (Starting from AS 4.5.0 ( 4.1.2+) the axis2.xml is located at AS-HOME/repository/conf/axis2/axis2.xml)

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">

• Now, you have a Master AS node, a Slave AS node, and a shared registry. The Master node will act as a Registry Read-Write (RW) Node, and the slave node will act as a Registry Read-only node.

There are two mechanisms for deployment synchronizing. One is a SVN-based mechanism, and the other is Registry-based mechanism.

SVN-based Deployment Synchronizer

SVN-based deployment synchronizer uses a Subversion repository to sync the contents in the sync directory (the axis2 repo directory, AppService-Home/repository/deployment/server,  by default). This includes several steps compared to registry-based depsync, but is efficient and high in performance.
What happens here is that when there are changes to the contents of axis2 repo directory, the Read-Writw nodes commits those to the Subversion repository. Then, this node sends a cluster message to all other (slave) nodes specifying the repo is updated. When these nodes receive the message, they updates there axis2 repo directory with what's in the svn repository.

• Download and install SVNKit implementation as described in in this JIRA ESBJAVA-950.  Download the properly bundled svnkit bundle or just get the one available in the JIRA (svnClientBundle-1.0.0.jar). Make sure you have read the svnkit license. We do not ship svnkit by default because of licensing incompatibilities on re-distributions. This is an optional step, but it's highly recommended that you do this. If svnkit is not there, DepSync can use the SVN installed on your machine. But this default subversion implementation is not suitable for production. You need to copy this to repository/components/dropins/.

• Now, you need to have a svn repository. You may use a existing svn repository, or create one locally. If you are in Linux, you can simply create a repository by following the command. 

svnadmin create /home/kasun/mylocalsvnrepo

For further information setting up svn repository, refer this link.

• Now, you may use file:// protocol rather than setting up http:// url for the repo (by using a webserver like Apache) if the repo will be in local system. For example, you can do a manual checkout of the repository by using the command `svn checkout file:///home/kasun/mylocalsvnrepo/`

• Next, we have to set the DepSync config in carbon.xml file located in AppService-Home/repository/conf/carbon.xml. Add the following xml content to carbon.xml of AppServer Master node. 

<DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>true</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType> 
        <SvnUrl>file:///home/kasun/mylocalsvnrepo/</SvnUrl>         
        <!-- <SvnUrl>http://svnrepo.example.com/repos/</SvnUrl> -->            
        <SvnUser>kasun</SvnUser>
        <SvnPassword>password</SvnPassword>
        <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
    </DeploymentSynchronizer>

I setup a local svn repo, and didn't set any user-passwords. So, SvnUser and SvnPassword is not effective in this scenario.

Here's an explanation on the options.

• Auto Commit - Allows to perform commit operations automatically.
• Auto Checkout - When this option is selected, an additional option named "Use Eventing" will be enabled.
• Use Eventing - Allows to trigger checkout actions on registry events. This is not needed most of the time.
• Synchronization Period - Synchronizer will get initialized and start running periodically as specified in this parameter. Default is 10 seconds.

• Now, add the following content to the AppServer slave node(s).

<DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>false</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType>
        <SvnUrl>file:///home/kasun/mylocalsvnrepo/</SvnUrl>
        <!-- <SvnUrl>http://svnrepo.local.com/repos/</SvnUrl> -->                
        <SvnUser>kasun</SvnUser>
        <SvnPassword>password</SvnPassword>
        <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
    </DeploymentSynchronizer>

If you look closely, the only two differences between these two configs is that in slave nodes, we set AutoCommit to false, and AutoCheckout to true.

• Now, deployment synchronizer is setup. Start greg, and the two AS nodes You'll see that when you do an operation like uploading a service to master node, the service archive get synced to slave nodes.

Registry-based DeploymentSynchronizer

Setting up a registry-based depsync is quite easy compared to svn-based depsync. There are two ways to enable this.Home/repository/conf

1. Via carbon.xml configuration file
2. Via DeploymentSynchronizer UI

Registry-based depsync via carbon.xml

• If you have added any config at svn-based depsync, uncomment/remove those.
• Add the following configuration to the AppServer Master node's carbon.xml file located in AppService-Home/repository/conf/carbon.xml.

<DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>true</AutoCommit>
        <AutoCheckout>false</AutoCheckout>
    </DeploymentSynchronizer>

• For slave nodes, add the configuration above but with AutoCommit set to to false, and AutoCheckout set to true.

Registry-based depsync via UI

Start the master and slave instances of AS, and go to the management console from the browser.  If you configured the nodes for master and slave with port Offsets 1 and 2 respectively, the possible link would be https://localhost:9444/carbon/deployment-sync/index.jsp.

• Log in to the master node, and go to Configure -> Deployment Synchronizer via left pane. 

• Check "Auto Commit", and click on enable.

• Now go the management console of the Slave node. (https://localhost:9445/carbon/deployment-sync/index.jsp)
• There, check "Auto Checkout", and click on enable.

There's no need of doing any changes to greg instance.

Now, deployment synchronizer is setup. Start greg, and two AS nodes and see how it goes. You'll see that when you do an operation like uploading a service to master node, the service archive get synced to slave nodes.