Copyright © TIBCO Software Inc. All Rights Reserved
Copyright © TIBCO Software Inc. All Rights Reserved


Chapter 14 Multicast Deployment and Troubleshooting : Walking Through a Multicast Deployment

Walking Through a Multicast Deployment
This section describes the steps needed to set up a simple example TIBCO Enterprise Message Service multicast deployment:
This example assumes that multicast connectivity exists and available bandwidth on the network is known. While not every aspect of a multicast deployment is covered in this example, it does illustrate the general thought process applied to multicast deployment.
Step 1: Design the Multicast Network Architecture
The location of the EMS server and clients are very important to a multicast deployment. You must ensure that multicast packets can get to all network nodes intended to receive multicast data, and you must account for all bandwidth across the network and network segments that the multicast data traverses. While TIBCO Enterprise Message Service detects and reports general connectivity problems, it is generally much easier to determine if there is connectivity before testing an EMS deployment. Your network administrator should be able to help with this.
For this example, let us assume that we are multicasting two streams of data: a fast data feed to some high performance processes on a 1 Gb network, and on a separate 100 Mb network a slower stream to a number of desktop applications. This leads us to the architecture shown in Figure 22:
Figure 22 Sample Multicast Deployment Architecture
Note that two separate channels using different interfaces are to be configured at the server, allowing the server to simultaneously multicast on a high speed Gigabit network and a slower 100Mb network.
Step 2: Install and Set Up EMS
Installation is straightforward, as described in the TIBCO Enterprise Message Service Installation. The only requirement above a regular EMS installation is that the multicast daemon must be running on any machine that receives multicast data.
On Windows systems, you can register the multicast daemon as a service using the emsntsrg utility. See emsntsrg on page 107 for more information.
Setup the EMS Server
Before sending multicast data, first the EMS server needs to be configured. Configuring the EMS server requires you to change some global settings in the tibemsd.conf file, and to configure multicast channels in the channels.conf file. After channels are configured, you enable topics for multicast by setting their channel properties in the topics.conf file.
Enable the Server for Multicast
To begin, some general settings must be configured in the EMS server's main configuration file, tibemsd.conf:
While enabling this trace is not required, it is very useful during the initial deployment, providing multicast-related warnings and errors.
Under heavy load, it is possible for publishers to feed data into the server faster than the server can multicast the data out. Enabling flow control causes the server to push back on the publishers, slowing them down if the server falls behind. This is not required, but highly suggested because it gives the server some room to minimize loss if this happens.
You should have added the following lines to the tibemsd.conf:
    multicast=enabled
    console_trace= DEFAULT,+MULTICAST
    flow_control=enabled
 
You may also want to add MULTICAST to the server's startup_abort_list, if multicast is required in your architecture.
Configure Multicast Channels
The next step configures the multicast channels. In this example there are two multicast channels, mcast-1Gb and mcast-100Mb. The section Sample channels.conf Settings below shows specific settings for these steps:
1.
Create the channels.conf file.
This file is described in channels.conf on page 232.
2.
Create two channels in the channels.conf file, [mcast-1Gb] and [mcast-100Mb].
3.
4.
For this example, the server is on a multi-homed machine so we must explicitly specify interfaces for each channel. If an interface is not specified, the EMS server uses the default interface. Note that this is also true for the multicast daemon. Use the -ifc command line parameter when running multicast daemons on multi-homed machines, described in Command Line Options.
5.
Set a maxrate for each channel.
The maxrate parameter restricts the rate at which the server sends messages over the channel. See Estimating the Maxrate below for a discussion of how the maxrate was determined.
Sample channels.conf Settings
When you have completed your channel configuration, the channels.conf file should contain the following lines:
[mcast-1Gb]
    address=239.1.1.1:10
    interface=10.99.99.99
    maxrate=112MB
 
[mcast-100Mb]
    address=239.1.1.2:10
    interface=10.99.99.100
    maxrate=8MB
Estimating the Maxrate
In this example, we have set the maxrate properties using arbitrary network usage numbers to arrive at an estimate of network capacity. The process used to estimate the maxrate can be described as follows:
First find your average network usage, not including expected multicast data. This assumes metric data rate measurement.
From here, calculate the available bytes per second for your network:
These initial rates are for testing purposes, and these will be modified later to maximize performance. Remember the cardinal rule with multicast performance is that sometimes you have to slow down to speed up. A rate that is too high will induce loss, which in turn causes messages to be resent, slowing the actual rate to something far below what your network is capable of.
This example uses only one channel per network. If your architecture has multiple multicast groups (channels with different address properties), remember to include all channels on the network in your maximum bandwidth calculations. This may require some balancing of data rates across channels.
Configure Multicast Topics
After the channels are defined, you must set the channel properties for topics so the server will send messages using multicast to multicast-enabled consumers subscribed to the topics. The channel property is set in the topics.conf configuration file.
In this example, we use two topics, feed-1Gb and feed-100Mb. These topic names are arbitrary; the key is assigning the correct channels to the topics.
Sample topics.conf
feed-1Gb channel=mcast-1Gb
feed-100Mb channel=mcast-100Mb
EMS Client Setup
There are two main requirements for EMS clients to receive multicast data:
The client must use an acknowledgement mode of NO_ACKNOWLEDGE when subscribing to the multicast topic. See Creating a Multicast Consumer for more information.
TIBCO Software also highly suggests that applications take advantage of the multicast exception listener to be notified of multicast related events, errors, and warnings. This is accomplished in two simple steps, illustrated in java code below:
1.
class MulticastExceptionHandler implements com.tibco.tibjms.TibjmsMulticastExceptionListener
{
    public void onMulticastException(Connection connection,
                                     Session session,
                                     MessageConsumer consumer,
                                     JMSException e)
        {
           System.out.println(e.getMessage());
        }
}
2.
com.tibco.tibjms.Tibjms.setMulticastExceptionListener(new MulticastExceptionHandler());
To set up a multicast exception listener using the C API, see the TIBCO Enterprise Message Service C & COBOL API Reference.
To set up a multicast exception listener using the .NET API, see the TIBCO Enterprise Message Service .NET API Reference, available through the HTML documentation interface.
Step 3: Determine Network and Application Capabilities
It is valuable to know what EMS data rates the network can accommodate. If your application can handle data at least as fast as your network can, you will encounter the unusual situation where the network is your throughput bottleneck, which is ideal—as long as those data rates meet your requirements.
Determine Network Capabilities
Now that the server is enabled, you can test and fine tune the maxrate specified for the channels. This section describes one method for testing your settings.
This example assumes that the messages multicast on the network are small, on average 100 bytes per message.
These steps describe how to test the network bandwidth settings:
1.
Start the EMS server using the -trace FLOW option, as described in Starting the EMS Server.
2.
Using the -trace option is not required, but may assist in detecting any problems. See Starting the Multicast Daemon for more information.
3.
On each node receiving multicast data, open a command line window and navigate to the TIBCO_HOME/ems/6.3/samples/java folder:
4.
Launch the tibjmsPerfSlave sample program included with EMS:
> java tibjmsPerfSlave -server serverURL
It is very important to run the jmsPerfSlave application on every node that will receive multicast data. EMS Multicast must be tuned to perform at the level of the slowest receiver, or congestion and loss can occur.
5.
> java tibjmsPerfMaster -topic feed-1Gb -channel mcast-1Gb -ackmode NO -time 30 -size 100
or
> java tibjmsPerfMaster -topic feed-100Mb -channel mcast-100Mb -ackmode NO -time 30 -size 100
These performance applications should be run on each node the publisher will run.
6.
Review the server and multicast daemon output for any warnings or errors. If you see any trace messages indicating loss, or if drastic rate fluctuations occur, this usually means you may be exceeding the maximum rate selected.
For example, a multicast error might look like:
channel='mcast-1Gb', Loss Detected, status=IO failed
On the server it is typical to see the following:
2008-11-13 17:11:57.300 Multicast channel 'mcast-100Mb' has exceeded its allotted bandwidth
If flow control is and FLOW tracing are enabled, you should see the following as well:
2008-11-13 17:11:57.781 Flow control engaged on topic 'feed-100Mb'
....
When flow control is enabled, this simply means that the server is pushing back on the publisher to slow down to the rate defined by the multicast channel.
When the trace messages indicate that multicast channels have exceeded their bandwidth, this indicates that the channel maxrate is too low—your publisher is publishing faster than the channel’s maxrate allows. On the other hand, when the maxrate is too high, you will see errors indicating that loss is detected.
Depending on what the trace messages show, try adjusting the maximum rate of the channel (the maxrate property) up or down, and repeat this test.
Evaluate Multicast Receiver Applications
One key to a successful multicast deployment is ensuring that the EMS server does not overrun your applications with data. This frequently this means setting the delivery rates (the channel's maxrate property) to a rate below what your network and EMS alone can handle.
The channel’s maximum delivery rate, or maxrate, should not exceed the rate at which the slowest message consumer can consume incoming messages.
Determining the maximum message rate that your slowest application can handle reduces the time spent during trial and error testing. If your applications can process data faster than the network can deliver it, you will have already determined the maximum rate from determining your network capabilities.
Largely, determining the maximum speed at which the slowest application can process incoming data is a trial and error process. It is often useful to programmatically determine an application's maximum rate of consumption. The multicast daemon buffers messages for slower applications, but this increases the latency of data and memory usage of the multicast daemon, and is not considered a sustainable condition.
If a multicast-enabled consumer is expected to fall behind at times and can sustain loss, you can account for this using the maxbytes and maxmsgs properties for topics. See Destination Properties for details about these properties.
Tune Channel Parameters
Once you have determined network capabilities and multicast receiver rates, you can experiment with increasing (or sometimes decreasing) channel maxrate properties to achieve maximum throughput. Finding the maximum multicast rate your environment can handle often requires more experimentation than anything else. Always remember that once the network has been saturated, throughput will drastically drop.
Tuning the Operating System
Unfortunately, operating systems are not normally tuned for high performance with raw sockets. There are a number of performance changes you can make; typically, these changes involve socket buffering and can yield significant increases in throughput.
For example, on Linux one can modify window sizes in the /etc/sysctl.conf file:
net.core.wmem_max=1073741824
net.core.rmem_max=1073741824
net.core.wmem_default=1073741824
net.core.rmem_default=1073741824
However, operating system tuning for multicast falls outside of the scope of this document. The TIBCO Professional Services Group can provide assistance with advanced tuning specific for TIBCO Enterprise Message Service, and there are many resources on the internet for general tuning of operating systems concerning network performance.
Development and Production Environments
Configuring multicast is specific to a particular network, and your configuration must account for traffic patterns and characteristics of nodes that are unique to your network. Consequently, the tuning parameters applied to a development environment may not be optimum in a production environment, and the reverse is also true. When migrating from one environment to another, it is important to remember that although the application and EMS architecture pattern may be identical, the network and application capabilities will need to be reevaluated through the repetition of the steps described in this section. Topic and channel definition names should remain the same, but rate, interface, and timeout parameters for multicast must be reevaluated.
The channel properties that should be reevaluated upon deployment include:

Copyright © TIBCO Software Inc. All Rights Reserved
Copyright © TIBCO Software Inc. All Rights Reserved