In this how-to we will cover how to monitor your application server via JMX with op5 Monitor with the plugin check_jmx4perl and the agent jolokia. We will also have a glance at monitoring application  servers trough a JMX-proxy for agentless monitoring where agents are not possible to use.

Prerequisites

This guide covers the installation on op5 Appliance System 3.5.2 and op5 Monitor 5.4.3, but should also apply on CentOS 5. In this example Jboss6 is used as the server of choice.  Make sure that you have perl, gcc and make installed, and the necessary ports opened in your firewall. Basic UNIX/Linux knowledge is needed.

Additional Information

JMX4Perl: http://labs.consol.de/lang/de/jmx4perl/

Agent information: http://www.jolokia.org/reference/html/agents.html

Proxy information: http://www.jolokia.org/reference/html/proxy.html

Security configuration: http://www.jolokia.org/reference/html/agents.html#agent-war-security

Disclaimer

This how-to  is not officialy supported by op5. It is just a glimpse on how to preform monitoring of application servers with op5 Monitor. And I am not a sysadmin working with application servers all day long, so there may be some glitches here and there. This is not a security document, so it is your responsibility that your system is secure. Please refer to the developers manual for more information about securing your installation.

Installation

Note1: If you run a distributed and/or Load Balanced setup, you will need to install the plugin on all systems that are running jmx-checks.

Note2: When using a agentless approach these steps only applies for the server that is acting as proxy.

• Download

# wget http://search.cpan.org/CPAN/authors/id/R/RO/ROLAND/jmx4perl-0.95.tar.gz

• Extract the file:

# tar xvzf jmx4perl*.tar.gz
# cd jmx4perl-<version>

• Configuration

# perl Build.PL

This command will most likely complain about the following dependencies: CBuilder, ParseXS and Module-Build.

Install them via rpm as described below. We have collected these rpm’s on our download site for your convenience.

• Download and unpack:

# wget http://download.op5.com/tmp/jmx4perl_dependencies.tar.gz
# tar xvzf jmx4perl_dependencies.tar.gz

• Installation

# rpm -Uvh perl-ExtUtils-ParseXS*.rpm
# rpm -Uvh perl-ExtUtils-CBuilder*.rpm
# rpm -Uvh perl-Module-Build*.rpm

Now it’s time to choose which components to install from the jmx4perl-package.

The required components are jmx4perl, and check_jmx4perl, although we recommend to install “j4psh” which is a JMX-shell that you can connect to JBoss and browse around the MBeans.

• Re-run Build.pl:

# perl Build.PL
Install 'jmx4perl' ? (y/n) [y ]y
Install 'check_jmx4perl' ? (y/n) [y ]y
Install 'cacti_jmx4perl' ? (y/n) [y ]n
Install 'j4psh' ? (y/n) [y ]y
Use Term::ReadLine::Gnu ? (y/n) [n ]n
Install 'jolokia' ? (y/n) [y ]n

When this is done there may still be some missing dependencies and the Build-script will try to install these by itself.

Run the following commands:

# ./Build installdeps
# ./Build
# ./Build test
# ./Build install

If all goes well you should have JMX4Perl installed on your system.

• Deployment of webapp

Next we will need to deploy the webapp “jolokia” in our Application server. In this example I used Jboss6.

# ssh root@jboss-server
# wget http://labs.consol.de/maven/repository/org/jolokia/jolokia-war/0.91/jolokia-war-0.91.war
# mv jolokia-war-0.91.war jolokia.war
# cp jolokia.war /home/jboss/jboss-6.0.0.Final/server/default/deploy

Now the webapp is deployed, and JBoss will extract it by it self.

To test if the webapp works, browse to: http://<server>:<port>/jolokia/version

or run the following command from your op5 Monitor server:

#  jmx4perl http://<server>:<port>/jolokia

Now jmx4perl should return some information about JBoss and jolokia.

You can also connect with j4psh to the server and browse around the MBeans:

# j4psh http://<server>:<port>/jolokia

• Copying files

You need to copy the installed check_jmx4perl and it’s config-files to a directory that op5 monitor can browse:

# mkdir /opt/plugins/custom/jmx4perl
# cp /usr/bin/check_jmx4perl /opt/plugins/custom/jmx4perl/

• Test check_jmx4perl

Now we can try to do a check in command-line using this example:

# cd /opt/plugins/custom/jmx4perl/
# ./check_jmx4perl -u http://<server>:<port>/jolokia --alias MEMORY_HEAP_USED --base MEMORY_HEAP_MAX --warning 80 --critical 90

That should give a output similar to this:

OK - [MEMORY_HEAP_USED] : In range 23.82% (123159632 / 517013504) | [MEMORY_HEAP_USED]=123159632;413610803.2;465312153.6;0;517013504

Config files

First copy all the default configs to your created “jmx4perl” folder in /opt/plugins/custom/jmx4perl/

# cd /root/jmx4perl-0.95
jmx4perl-0.95/ # cp -R config/ /opt/plugins/custom/jmx4perl/

• One config to rule them all

In this example we will create one config that includes all the others to make changes more simple to maintain and to shorten the check commands.

# cd /opt/plugins/custom/jmx4perl/

Create this config called jmx4perl.cfg:

# Default definitions
include memory.cfg
#include tomcat.cfg #Is included in jboss.cfg
include jboss.cfg
include threads.cfg
# ====================================
# Check definitions
<Check j4p_memory_heap>
Use memory_heap
Critical 90
Warning 80
</Check>
<Check j4p_thread_count>
Use thread_count
Critical 1000
Warning 800
</Check>
# Check for uptime, used as kind of 'ping' for
# service dependencies
<Check j4p_uptime>
MBean java.lang:type=Runtime
Attribute Uptime
Warning 120:
Critical 60:
</Check>
# A multi check combining two checks
<MultiCheck j4p_jvm>
Check j4p_memory_heap
Check j4p_thread_count
</Check>

These files contains the variables set in “check_command_args” for the check commands that we will create.

# cd /opt/plugins/custom/jmx4perl/config/
# ls
common.cfg  jboss.cfg jetty.cfg  jmx4perl.cfg memory.cfg  threads.cfg  tomcat.cfg

I will go trough: memory.cfg, thread.cfg and some possibilities to customize your jboss.cfg.

Please have a look in the config files for further explanation of the options.

• memory.cfg

• threads.cfg

Check commands

• Using agents

First we create check commands to be used with agents. You can read the pros and cons for the different usages of jolokia at:

http://www.jolokia.org/reference/html/architecture.html#agent-mode and http://www.jolokia.org/reference/html/architecture.html#proxy-mode

When using config-files we minimize the number of check commands to a bare minimum, and we will only create two of them.

One for static checks and one for an incremental check that shows how much a value has changed in a given time period.

• Go to: Configure -> Commands

Create a new check command with the following values:

• Click “Apply Changes”

And the same procedure for the incremental check.

And save the configuration.

First we add host to op5 Monitor with the ip-address of our application server.

• Go to: Configure -> New Host

Fill in the configuration information: host name, alias, address etc.

• Click “Scan host for services” -> Click “Continue to step 3″

• Next, click “Services for <hostname>”

Select the check command that you just created, and fill in the following arguments:

• Go to the host -> Services -> Add new service

The syntax of check_command_args are:

<port>!<command>!<warning>!<critical>

• Click “Apply Changes”

And Save applied changes

Repeat these steps for the incremental check

The syntax of check_command_args are:

<port>!<command>!<delta>!<critical>

This command checks how much memory usage has changed the last 5 minutes with a warning threashold of 25%.

Proxy mode

When using a agentless approach we need a app-server with the jolokia webapp installed, and configured to accept connections via RMI.

This was default on my installation of Jboss6 and i won’t expand it any further.

RMI test:

# cd /opt/plugins/custom/jmx4perl

# ./check_jmx4perl -u http://<proxy-server>:<port>/jolokia --target service:jmx:rmi:///jndi/rmi://<app-server>:port/jmxrmi
--config config/jmx4perl.cfg --check memory_heap --warning 80 --critical 90

If this works you should get something like this in return:

OK - Heap-Memory: 25.79% used (127.18 MB / 493.06 MB) | Heap=133359304B;413610803.2;465312153.6;0;517013504

Now we will create a check command using config-files as explained earlier.

• Check command:

First we add host to op5 Monitor with the ip-address of our application server.

• Go to: Configure -> New Host

Fill in the configuration information: host name, alias, address etc. Click “Scan host for services” -> Click “Continue to step 3″

• Next, click “Services for <hostname>”

Select the check command that you just created, and fill in the following params:

Argument explanation:

First we define which host that runs the JMX-proxy, port, rmiport on the actual server we want to monitor, what checks we want to run on this server, and finally the thresholds.

These arguments is also listed in the config-files for jmx4perl just as the previous example with agents.

What you want to monitor in your environment is your call, this is just a glance at the possibilities using jmx4perl.

Jboss Labs (Optional)

This part shows some custom options that are not covered with the default configuration and is entirely optional. I just want to show examples for custom checks that you can create and include in your configs according to your needs. I found these values using the “j4psh” shell and browsing around the MBeans and editing jboss.cfg according to the MBean names and creating check commands for these:

• jboss.cfg

<Check jboss_cpool_available>
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = AvailableConnectionCount
Name = Avalible Connections
</Check>

<Check jboss_cpool_used_max>
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = MaxConnectionsInUseCount
Name = Max. connections in use
</Check>

<Check jboss_cpool_used>
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = InUseConnectionCount
Name = Connections in use
</Check>

<Check jboss_cpool_creation_rate>
Use = count_per_minute("connections")
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = ConnectionCreatedCount
Name = Connection creation rate
</Check>

You can of course create additional config files with your specific needs and include them in “jmx4perl.cfg”

Author: Fredrik Mikker / op5 Professional Services

Rev. 1.0