Jabber Server Farming How-To

Ryan Eatmon

2002-06-06

Revision History
Revision 0.12002-06-06Revised by: rwe
First draft.

Table of Contents
1. About this How-To
1.1. Purpose / Scope
1.2. About Jabber
1.3. Feedback
1.4. Copyrights and Trademarks
1.5. Acknowledgments and Thanks
2. Introduction to Jabberd and Farming
2.1. Jabberd - The Jabber.org Jabber Server
2.2. Farming
3. Implementing the Farm
3.1. Client-to-Server Farming w/ Round Robin DNS
3.2. Connection Redirection
3.3. Router Layer
3.4. Moving JSM State data into a DB
3.5. Multiple JSMs
3.6. JSM Communications
List of Figures
3-1. c2s Farming Diagram

Chapter 1. About this How-To

1.1. Purpose / Scope

This document was started on June 06, 2002 by Ryan Eatmon to explain how to set up a Jabber Server Farm using the Jabber.org Jabber Server (http://jabberd.jabberstudio.org).

This document is broken into two main sections. What is, and what will be. As of the writing of this document the Jabber Server does not have all of the pieces that are required to do full farming. There are steps that can be taken to get partial farming, and those will be covered. There are also steps that we are going to take, and those are covered to so that others can contribute and comment.


Chapter 2. Introduction to Jabberd and Farming


Chapter 3. Implementing the Farm

3.1. Client-to-Server Farming w/ Round Robin DNS


3.1.2. The Reality

This is doable right now. The 1.5 version of jabberd has sought to break all of the pieces out into seperate processes. This was not done for farming specifically, but we will not complain since the jadc2s component can handle upwards of 10k users. (The 1.4.x series c2s could only handle ~1024).

Currently there is a forked version of jadc2s that works with the 1.4.2 server. It is located in the jabberd14 CVS repository on JabberStudio. The following example is running with two jadc2s boxes, and one central jabberd box. To set this up do the following:

  1. Get all of the source code for jabberd14. Build the server, configure/make.

  2. Get the source for xdb_sql from IDEALX , build it, and setup the jabber.xml to use it.

    Note

    This is a very important step. xdb_file, the default xdb that comes with jabberd, is limited to open file descriptors too. You can play the same shell games that we are going to play with jadc2s later, but if you want a server that can handle millions or users, then you need something other than xdb_file. Enter xdb_sql, which only uses one file descriptor to connect to the mysql server.

    For more information on how to configure xdb_sql, please see the README that is distributed in the release.

  3. Build jadc2s and distribute the binaries to the boxes where they will run.

  4. Setup the main jabberd to accept the jadc2s component connections. In the jabber.xml config file, add the following XML:

    
<service id="jadc2s-1">
        <accept>
            <ip/>
            <port>5111</port>
            <secret>secret</secret>
        </accept>
    </service>
    
    <service id="jadc2s-2">
        <accept>
            <ip/>
            <port>5112</port>
            <secret>secret</secret>
        </accept>
    </service>
                            

    Now you can run the main jabberd and get it listening for the jadc2s to connect to it.

  5. Configure the jadc2s.xml on each box to connect to the SM, where to listen, etc...

    
<!--  session manager configuration                                    -->
    <sm>
        <!--  host and port of the SM                                      -->
        <host>localhost</host>
        <port>5111</port>
        <!--  shared secret, for authenticating us                         -->
        <secret>secret</secret>
    
        <!--  our ID, for authenticating us to the sm                      -->
        <id>jadc2s</id>
    
        <!--  how many times to try to connect to the sm (default: 5)      -->
        <retries>5</retries>
    </sm>
    
    <!--  local network settings                                           -->
    <local>
        <!--  who we identify ourselves as. This should correspond to the  -->
        <!--  ID (host) that the session manager thinks it is.             -->
        <id>localhost</id>
    
        <!--  IP address to bind to (default: 0.0.0.0)                     -->
        <!-- <ip>0.0.0.0</ip> -->
            
        <!--  port to bind to (default: 5222)                              -->
        <port>5222</port>
    
        <!--  SSL configuration                                            -->
        <!--  Specify the port to listen on, and the key file to use for   -->
        <!--  the certificate.                                             -->
        <!--  <port/>     (default: 5223)                                  -->
        <!--  <pemfile/>  (default: ./server.pem)                          -->
        <!--
        <ssl>
            <port>5223</port>
            <pemfile>./server.pem</pemfile>
        </ssl>
        -->
    </local>
                            

    For more information on how to configure jadc2s, please see the README in the jadc2s source directory.

  6. Open a shell where you can change file system parameters (root usually) and execute the following command:

    This bumps the upper bound on the number of allowed file descriptors that can be open at one time.

  7. Set the limit for the shell you are in to use more than the default 1024 file descriptors.

  8. Tell jadc2s how many file descriptors it is allowed to use:

    
<!--  maximum number of file descriptors. Should be a prime        -->
    <!--  number. At least four will be used by jadc2s itself,         -->
    <!--  usually around six or seven (default: 1023)                  -->
    <!--  For a list of prime numbers:                                 -->
    <!--    http://www.utm.edu/research/primes/lists/small/10000.txt   -->
    <max_fds>10007</max_fds>
                            

    It is important that the number you pick is a prime number. To make it easy to find the prime you want you can visit this page http://www.utm.edu/research/primes/lists/small/10000.txt.

  9. All that's left is to run the server, and the jadc2s processes. Everything should work fine. If it doesn't, then PLEASE tell me at so that I can fix this document.