PRODUCTS AND SERVICES INDUSTRIES SUPPORT PARTNERS COMMUNITIES ABOUT
  Coherence 3.4 User Guide
  How to Manage Coherence Using JMX
Added by Rob Misek, last edited by Everett Williams on Aug 21, 2008  (view change)

Labels

 

Coherence includes facilities for managing and monitoring Coherence resources by using the Java Management Extensions (JMX) API. JMX is a Java standard for managing and monitoring Java applications and services. It defines a management architecture, design patterns, APIs, and services for building general solutions to manage Java-enabled resources. This section assumes familiarity with JMX terminology. If you are new to JMX, you should start is with this article.

To manage Coherence using JMX:

JMX support

Coherence Enterprise Edition and higher support clustered JMX, allowing access to JMX statistics for the entire cluster from any member. Coherence Standard Edition provides only local JMX information.

Adding JMX libraries to the Coherence classpath

To manage a Coherence cluster using JMX, ensure that you have the necessary JMX 1.0 or later classes (javax.management.*) in the classpath of at least one Coherence cluster node, known as an MBeanServer host. The cluster nodes that are not MBeanServer hosts will be managed by the MBeanServer host(s) by using the Coherence Invocation service.

All compliant Java SE 5.0 JREs and Java EE application servers supply a JMX 1.0 or later implementation; therefore, if the MBeanServer host node is running within a Java SE 5.0 JVM or Java EE application server, no additional actions are necessary. For standalone applications running within a pre-Java SE 5.0 JVM, you can download the necessary JMX libraries here and add them to the classpath.

Configuring the Coherence Management Framework

In most cases, you can enable JMX management simply by setting the tangosol.coherence.management Java system property on all Coherence cluster nodes that are acting as MBeanServer hosts:

-Dtangosol.coherence.management=all

and the tangosol.coherence.management.remote Java system property on all cluster nodes:

-Dtangosol.coherence.management.remote=true

The use of dedicated JMX cluster members is a common pattern. This approach avoids loading JMX software into every single cluster member, while still providing fault-tolerance should a single JMX member run into issues.

In general, the Coherence Management Framework is configured by the management-configuration operational configuration element in the Coherence Operational Configuration deployment descriptor (tangosol-coherence.xml). The following sub-elements control the behavior of the Management Framework:

Element Description
domain-name Specifies the name of the JMX domain used to register MBeans exposed by the Coherence Management Framework.
managed-nodes Specifies whether a cluster node's JVM has an in-process MBeanServer and if so, whether the node allows management of other nodes' managed objects. Valid values are none, local-only, remote-only and all. For example, if a node has an in-process MBeanServer and you would like this node to manage other nodes' MBeans, then set this attribute to all.
allow-remote-management Specifies whether this cluster node will register its MBeans in a remote MBeanServer(s).
read-only Specifies whether the MBeans exposed by this cluster node allow operations that modify run-time attributes.

For additional information on each of these attributes, please see the Operational Configuration Elements.

Accessing Coherence MBeans

After configuring the Coherence Management Framework and launching one or more Coherence cluster nodes (at least one being an MBeanServer host) you can view and manipulate the Coherence MBeans registered by all cluster nodes using standard JMX API calls. See the Javadoc for the com.tangosol.net.management.Registry class for details on the various MBean types registered by Coherence clustered services.

Coherence ships with two examples that demonstrate accessing Coherence MBeans by using JMX. The first uses the HttpAdapter, shipped as part of the JMX reference implementation (jmxtools.jar). To run the example on a pre-Java SE 5.0 JVM, start the Coherence command line application using the following command on Windows (note that it is broken up into multiple lines here only for formatting purposes; this is a single command entered on one line):


java -cp jmxri.jar;jmxtools.jar;coherence.jar 
     -Dtangosol.coherence.management=all
     -Dtangosol.coherence.management.remote=true 
      com.tangosol.net.CacheFactory

On Unix:


java -cp jmxri.jar:jmxtools.jar:coherence.jar
     -Dtangosol.coherence.management=all
     -Dtangosol.coherence.management.remote=true 
      com.tangosol.net.CacheFactory

Once the Coherence command line application has started, type jmx 8082 and press Enter. This starts the HttpAdapter on http://localhost:8082 in the cluster node's JVM and makes the cluster node an MBeanServer host. You can now use the HttpAdapter web application to view and manipulate Coherence MBeans registered by all cluster nodes:

Alternatively, you can run this example with the Sun Java SE 5.0 JVM and use the JConsole utility included with the Sun Java SE 5.0 JDK to view and manipulate Coherence MBeans. To do so, start the Coherence command line application using the following command (note that it is broken up into multiple lines here only for formatting purposes; this is a single command entered on one line):


java -Dcom.sun.management.jmxremote
     -Dtangosol.coherence.management=all
     -Dtangosol.coherence.management.remote=true 
     -jar coherence.jar

Once the Coherence command line application has started, launch the JConsole utility (located in the bin directory of the Sun Java SE 5.0 JDK distribution) and open a new connection to the JVM running the Coherence command line application:

!

The second example is a JSP page (JmxCacheExplorer.jsp) that displays basic information on each running Coherence cache using JMX API calls. You can find this example in the examples/jsp/explore directory under the root of your Coherence installation.

Additional JMX examples may be found on the Coherence Forums.

Using Coherence MBeanConnector to Access MBeans

Coherence ships with a program to launch a cluster node as a dedicated MBeanServer host. This program provides access to Coherence MBeans by using the JMX Remote API using RMI or the HTTP server provided by Sun's JMX RI. The RMI and HTTP ports are user-configurable, allowing for access through a firewall. The server is started using the following command (note that it is broken up into multiple lines here only for formatting purposes; this is a single command entered on one line):


java -Dtangosol.coherence.management=all
     -cp coherence.jar com.tangosol.net.management.MBeanConnector [-http -rmi]

To allow access by using JMX RMI, include the -rmi flag. To allow access by using HTTP and a web browser, include the -http flag. Both flags may be included; however at least one must present for the node to start.

The following table describes optional properties that can be used for JMX RMI configuration:

Property Description
tangosol.coherence.management.remote.host The host that the JMX server will bind to. Default is localhost. (NOTE: on Redhat Linux this may have to be changed to the host name or IP address)
tangosol.coherence.management.remote.registryport The port used for the JMX RMI registry. Default is 9000.
tangosol.coherence.management.remote.connectionport The port used for the JMX RMI connection. Default is 3000.

The following table describes optional properties that can be used for HTTP configuration. (NOTE: This flag requires Sun's JMX RI in the classpath):

Property Description
tangosol.coherence.management.remote.httpport The port used for the HTTP connection. Default is 8888.

To connect by using JConsole with default settings use the following command:


jconsole service:jmx:rmi://localhost:3000/jndi/rmi://localhost:9000/server

To connect by using HTTP with default settings use the following URL:


http://localhost:8888

Configuring Management Refresh Methodology

The current release of Coherence offers several ways to reduce the latency of management information. Refresh policy was introduced in Coherence 3.3 to allow for optimization of the retrieval of information from remotely managed nodes. Two new settings were added to help integrators and administrators configure the refresh policy.

The tangosol.coherence.management.refresh.expiry property specifies the minimum time interval between the remote retrieval of management information from remote nodes.


-Dtangosol.coherence.management.refresh.expiry

The value of this element must be in the following format:

[\d]+[[.][\d]+]?[MS|ms|S|s|M|m|H|h|D|d]?

where the first non-digits (from left to right) indicate the unit of time duration:

  • MS or ms (milliseconds)
  • S or s (seconds)
  • M or m (minutes)
  • H or h (hours)
  • D or d (days)

If the value does not contain a unit, a unit of milliseconds is assumed.

The tangosol.coherence.management.refresh.policy property defines the refresh policy for the MBean.


-Dtangosol.coherence.management.refresh.policy   

Valid values for this property are:

Setting Description
refresh-expired (default) This setting has the same functionality as in pre-3.4 Coherence releases. Each MBean will be refreshed from the remote node when it is accessed and the expiry delay has passed from the last refresh. This setting is best used when MBeans are accessed in a random pattern.
refresh-ahead MBeans are refreshed before they are requested based on prior usage patterns after the expiry delay has passed. This setting can reduce latency of the management information with a minor increase in network consumption. This setting is best when MBeans are accessed in a repetitive/programmatic pattern.
refresh-behind Each MBean will be refreshed after the data is accessed. This method ensures optimal response time. However, the information returned will be offset by the last refresh time.

Custom MBean Configuration

Coherence 3.4 can be configured to load platform and standard MBeans on connection to the cluster. This allows administrators and support personnel to update and view system and application information from all nodes in a cluster from a single location. This feature also eliminates the need for JMX programs to connect to multiple sources to gather information.

How to add a standard MBean to Coherence:

  1. Create a standard MBean.
  2. Add a standard MBean Class or jar to the Coherence classpath (including central management node).
  3. Create a custom MBean configuration file (custom-mbean.xml).
  4. Modify node startup scripts to reference custom-mbean.xml.

How to add a the results of a JMX Query to Coherence:

  1. Create a Custom MBean XML file
  2. Configure node startup script to include JMX MBean Server
  3. Configure node startup script to reference custom-mbean.xml

The following figure illustrates an example of a JMX Query.

JMX Reporter

Coherence 3.4 provides a JMX reporting capability (the Reporter). The Reporter provides out-of-the-box reports that help administrators and developers manage capacity and trouble shoot problems.

Plan for archiving and removing
Due to the volume of the information created by the Reporter, you must have a plan for archiving and/or removing the results BEFORE starting the Reporter.

Basic Configuration

Enabling the Reporter with basic content requires setting the system properties:

On the "management" node:


-Dtangosol.coherence.management.report.autostart=true
-Dtangosol.coherence.management=all
-Dcom.sun.management.jmxremote

On the "managed" node:


-Dtangosol.coherence.management.remote=true

Basic configuration will create a single Reporter node that will log the JMX statistics for all nodes in the cluster. The log files will be placed in the working directory of the application.

Administration

The JMX Reporter is managed through an MBean under the Coherence Domain. The Reporter MBean provides information related to the status and performance of the Reporter. The MBean also provides the capability to start and stop the service as well as run a report on demand.

For a full description of the Reporter Attribute see the Reporter section of the java doc

Data Analysis

Seven files are created each hour by the Reporter. Each file is prefixed with the date and hour the report was executed in a YYYYMMDDHH format. This allows for easy location and purging of unwanted information. The files generated are:

YYYYMMDDHH-memory-status.txt Contains memory and garbage collection information about each node.
YYYYMMDDHH-network-health.txt Contains the publisher success rates and receiver success rates for the entire grid
YYYYMMDDHH-network-health-detail.txt Contains the publisher success rates and receiver success rates for each node
YYYYMMDDHH-node.txt Contains the list of nodes that were members of the grid
YYYYMMDDHH-service.txt Contains Request and Task information for each service.
YYYYMMDDHH-proxy.txt Contains utilization information about each proxy node in the grid
YYYYMMDDHH-cache-usage.txt Contains cache utilization(put, get,etc) statistic for each cache

Click here for a complete description of the data contained in each file.

Advanced Configuration

Custom Reports

  1. Create the custom report configuration file.
  2. Update report batch to execute the report.
  3. Run on demand.

Distributed Configuration

A distributed configuration is only recommended in situations where grid stability is an issue. In this configuration, the distributed reporters will run independently, and the execution times will not align. Therefore, grid level analysis is extremely difficult but node level analysis during periods when nodes may be leaving or joining the grid will still be available.

When running in distributed mode, each node logs local JMX statistics while allowing for centralized management of the Reporters. To enable this configuration set the following system properties

On the "managing" node:


-Dtangosol.coherence.management.report.autostart=false
-Dtangosol.coherence.management.report.distributed=true
-Dtangosol.coherence.management=all
-Dcom.sun.management.jmxremote

On the "managed" node:


-Dtangosol.coherence.management.report.autostart=true
-Dtangosol.coherence.management.report.distributed=true
-Dtangosol.coherence.management=local-only
-Dtangosol.coherence.management.remote=true

Unknown macro: {rate-table}