If implemented correcly a centralized logserver with monitoring capabilities can really influence and support how you work, what you spend your time on and how you make decisions. Below are some main goals are listed which are important to keep in mind in order to prioritize wisely what you spend your time on when implementing a centralized logserver.

Stop guessing!

We all need to make well informed desicions, and sometimes even investigate things. Understanding in what order things happen, how one system differs from another and how your systems behaved previously, before changes, becomes crucial in todays complex it-systems and ever changing it-infrastructure. With a centralized logserver in place you have access to a data-mining tool providing you with a full time-stamped view of everything you've decided to log. You can easily filter out what you need to know, monitor the log filters, schedule a report for it, or just follow the trend of hits for your filter.

Work proactively!

Secure your business!

• Make use of the application logs of your main production systems to improve performance.
• Guarantuee your customers their transactions are logged and recorded.
• Minimize downtime in case of network intrusion.
• Verify that you don't have unauthorized access to corporate information.
• Make sure to monitor your logs to recieve notifications and alarms and to be responsive

The work of actually tuning paramaters, having proper logging in applications, exterminating compromized systems and scheduling access-reports can be delegated to "system owners" in the organisation.

When planning how to perform logging, filtering, archiving and perhaps filter monitoring for your entire network, start by putting together two things:

• A summary of your own organizations and your customers general demands on transaction logging, system logging, archiving/rotation, access-logging, reporting and monitoring.
• A checklist table where rows are each production system, and columns are "tasks", or type of logging/reporting/monitoring to add (see table below).

Split the work that lies ahead into three to ten stages and populate each stage with one to three tasks (types of logging to add). Plan for a test- and adjustment-period of a least a week between the end of one stage and the start of the next. The test- and adjustment-periods are needed to be able to remedy errors in your IT-environment that has been discovered in each stage, and to confirm that filters and report-types are correctly configured/adjusted. Plan for follow-up meetings with department managers at the end of the stages.

Don't log everything!

Please observe that you should exclude at least two things in your checklist-matrix:

• Logging from system where you already have specialized tools for analazys and archiving (like checkpoint secure client log viewer and webserver-statistics tools like AWstats). There is no point in centralizing logs where specialized tools are already in use. It will make firewall log filtering and web statistics generation less easy, while adding unneccesary load on the LogServer.
• Excessive/unneccesary logging (like debug-logging from development systems). Don't centralize and record info that has no value for others. Developers are likely to enable debug-logging and verbose logging whereever available. These logs will just produce unneccesary load without adding value for others.

Below is a list of tasks (logserver configuration and types of logging to set up), listed in order of importance. How important it is to add logging of dirrent types is of course specific for each organisation, but the list below can be a good starting-point.

In short you start by making sure your system logs are all available in a central location and finish up by adding scheduled reports and monitoring of log filters matching on "bad signs" (early warnings) in your logs.

This how-to will show you step by step as to how you will get started with the op5 Cloud Monitor service.

op5 Cloud Monitor is a service for those who need to monitor their external services from the users' point of view.

First of all you need to register a user account at City Cloud. You have two sites to chose between

• www.citycloud.se for swedish users
• www.citycloud.eu for international users

In this how-to we are using the international site (www.citycloud.eu).

To register you account at City Cloud

1. Go to www.citycloud.eu.
2. Click Create account.
3. Fill in the form.
4. Read the terms and accept them by checking the checkbox next to the link to the terms.
5. Click Register account

You will now get an email with all information about your account that you need to be able to login to your account at City Cloud.

Now when your account is ready you may login to the admin pages using the credentials that was sent to you when you created your account.

To create your new machine

1. Go to admin.citycloud.se.
2. Type in your username, password and click Login

To create your op5 Monitor machine

1. Login to the admin pages.
2. Click App center.
3. Click the plus sign to the right of the op5 server.
4. Give the virtual machine a name.
5. Chose what hardware template to use (we recomend the Standard template).
6. Click Create.

It will now take about 10 to 20 minutes for the machine to be created.

To be able to reach your op5 Cloud Monitor server you need the ip address. The ip address can be found in the information about the machine you created earlier.

To reach your op5 Cloud Monitor server.

1. Login to the admin pages.
2. Click VM
3. Click the i icon on the op5 Cloud Monitor server in the list.
4. In the top frame of the window shown you will see the ip address to your op5 Cloud Monitor server.
5. Use that ip address and point your browser to your op5 Cloud Monitor server

Now you are at the portal page of your op5 Cloud Monitor server.

The first thing you need to do is to change the root password of your op5 Cloud Monitor server.

The default root password is:

InCl0ud

To change the root password

1. Go to the portal page of your op5 Cloud Monitor server.
2. Click Configure system.
3. Type in the root password and click Login.
4. Click Change password in the menu to the right.
5. Type in the current root password, the new root password and repeat the new root password.
6. Click Save.
7. Click Logout at the top to the right on the page.

Note: Make sure you store the new root password in a safe place.

To start use your op5 Cloud Monitor server you need to login. When you are in you need to change the password for the monitor user.

The default login to the op5 Cloud Monitor server user interface is:

User: monitor
Password: monitor

To login to your op5 Cloud Monitor server

1. Go to the portal page op5 Cloud Monitor server.
2. Click Monitor.
3. Type in the username and password.
4. Click Login

Sinice your op5 Cloud Monitor server is using a public ip address it is very important to change the password of the monitor user.

Changing the monitor user password

1. Login to the op5 Cloud Monitor user interface.
2. Click Configure in the main menu to the left.
3. Click Access rights.
4. Click monitor user.
5. Type in the new password and repeat it.
6. Click Apply and then click the Save icon at the top menu to save the configuration.

Note: Make sure you store the new root password in a safe place.

Now when you are ready to start using the newly created op5 Cloud Monitor server you might need some help. The following resources will be a good help for you.

• op5 Monitor user manual
• op5 Appliance system manual

Revision 1
2010-06-22

• op5 Monitor is installed
• you have SNMP access from the op5 Monitor server to the switch(es) you would like to monitor

1. Open up the op5 Monitor configuration tool and click Commands.
2. Add a new command with the following data:

   command_name  check_portstatus
   command_line  $USER1$/check_portstatus -H $HOSTADDRESS$
                 -C $ARG1$ -v $ARG2$ -w $ARG3$ -c $ARG4$

3. Click Apply

1. Open up the op5 Monitor configuration tool.
2. Pick up the host you like to add the service to and click Services for this host...
3. Add a new service like this

   Service description  Port status
   check_command        check_portstatus
   check_command_args   snmpcommunity!1!10!5

4. Click Apply
5. Click Save

On the Service information page you can find the status output for this service. It looks like this:

OK: 49 of 52 ports available, 1 down, click here to view the report.

From here you can view a report showing all ports on the host. In the report you can see the port

• description
• state
• speed
• idle time

To view the report click on the click here to view the report link.

• op5 Monitor and/or op5 Statistics installed correctly
• Sensatronics EM1 set up in accordance with the op5 Environment module manual or the accompanying Sensatronics Model EM1 Environmental Monitor Quick Start Guide.
• Http network access from the op5 server to the environmental monitor.

• when a certain service will be checked
• when notifications will be sent out
• to which person to send notifications based on what time the alarm triggered
• when dependencies are valid.

Before you start configuring your own time periods, there are a couple of things o consider.

Checks made manually are not restricted by time periods. Only the scheduled checks are restricted to the time period configured.

When you’re adding dates, weekdays, days of the months and calendar dates to your time period, it is important to remember that some directives over rides the other depending on how they were specified. The order of precedence for different types of directives (in descending order) is as follows:

• Calendar date (2008-01-01)
• Specific month date (January 1st)
• Generic month date (Day 15)
• Offset weekday of specific month (2nd Tuesday in December)
• Offset weekday (3rd Monday)
• Normal weekday (Tuesday)

Now let us take a look at a couple of examples to show how the timeperiods can be used.

Example 1

Let’s say we wanted to configure a time period to include non-work-hours and every holiday as well. This is one way to achieve this.

1. Make a new time period
2. Add every holiday as an exception.
3. Make yet another time period.
4. Copy the settings from 24x7, and then exclude the work hours and holidays.

Now you have a time period that defines every non working hour, including holidays.

Now you might want to have a certain contact, other than the contacts used for working hours, during the timeperiod you just created. To do this you only have to follow the steps below:

1. Change the time period for the contact you want to be notified when alarms trigger.
2. Select the time period for host_notification_period and/or service_notification_period.

Example 2

Another scenario might be when you want to exclude a period of time when no checks are being made. Say you have a backup job running at 3:00 every night. The system might be hogged and you don’t want any alarms being triggered at this time.

To exclude the time between 3:00 and 4:00 every day from being part of any scheduled check.

1. Create a new time period with the settings from 24x7.
2. Exclude 03:00-04:00, every day, month and year.
3. Edit the host or service you want this time period to be used with and use it with check_period.

Usually it is recommended to monitor 24x7. If you choose to do so instead of disabling any checks during this time, you could specify that no notification could be sent out during this time. This is achieved by adding a time period to the notification_period directive of either a host or service.

This was just a few simple examples of how time periods can be used. It can also be used to decide when escalations can be made or when dependencies are considered valid. A few other scenarios wheere time periods are really useful might be specifying vacation days and creating more advanced on-call contact schedules etc.

As you probably already know, VLAN (Virtual LAN) is used to segment networks. There are several reasons why you would want to configure VLAN on your op5 server.

Let us say you have a very large amount of monitored hosts and services. All the traffic destined for the server will then probably be routed at the distribution layer. This could be avoided with VLAN configured on the server. Since the server would be on the same network as your hosts, the traffic will never be sent to your router/firewall. The traffic would only be handled in the access layer.

Before you start configuring your VLAN, there are a few things to take under consideration. For starters, where should the traffic destined for the internet go?

Either you could remove the IP details from the physical interface and use one of the VLAN for the default route. Note that you might have to add the option “BOOTPROTO=none” to the physical interface for this to work. Or you could just tag untagged traffic at the switch with the desired VLAN ID and leave the configuration for the physical interface. Most often referred to as native VLAN in terms of switch configuration.

It is recommended that you have physical access to the server since you could easily lose contact with the server if something isn’t correctly configured.

1. Use your favorite SSH client to connect to your op5 server.
2. Create a new file in /etc/sysconfig/network-scripts/ and name it "ifcfg-eth0.X". Where "X" is your VLAN ID. The valid VLAN ID range is between 1 and 4096. VLAN 1 is untagged traffic.
3. Add the following to that file and replace the options to match your network configuration.

   DEVICE=eth0.x
   BOOTPROTO=static
   BROADCAST=1.2.3.4
   IPADDR=1.2.3.4
   NETMASK=255.255.255.0
   NETWORK=1.2.3.0
   TYPE=Ethernet
   ONBOOT=yes
   VLAN=yes

   Note:Only add the “GATEWAY” statement for the interface used for outbound traffic. There can only be one default route.
4. Save the new configuration file and restart the network service. Note that you will lose connection to the server and will have to reconnect.

   # /etc/init.d/network restart

If your organization uses a Citrix server it is usually one of the more critical pieces of your infrastructure because of the significant user impact when it is down or unusable.

Apart from the normal monitoring parameters that you should have for every box, there are a few checks you can run to monitor your end user's perspective of your Citrix server.

op5 Monitor includes a plugin called check_citrix, which allows you to monitor your Citrix servers and check that they are correctly responding to client requests.

There is also a third-party plugin called check_ctx_licensserver which allows you to monitor your Citrix server's license usage.

To be able to complete this how-to you will need the following:

• op5 Monitor installed correctly
• a Citrix server or server(s) to monitor
• optionally, check_ctx_licensserver from http://exchange.nagios.org/directory/Plugins/Remote-Access/Citrix-License-Server-Check/details

The best way to make sure that your Citrix solution is usable is to check whether or not you get a correct 'published applications' list as a reply to a broadcast request.

This allows you to get notified by op5 Monitor and not your end users should your Citrix server stop responding to requests.

command_name    check_citrix_broadcast
command_line    $USER1$/check_citrix -B -W "$ARG1$" -P "$ARG2$"

This command will broadcast a request to find the address of a Citrix browser to query.

-W is a comma-separated list of published applications that _should_ be in the list of applications returned by the Citrix master browser, otherwise the check will return WARNING.

-P is a comma-separated list of published applications that _must_ be in the list of applications returned by the Citrix master browser, otherwise the check will return CRITICAL.

If the plugin is unable to get a response to its broadcasted request for a Citrix master browser it will return CRITICAL.

This check can be added to a host representing your Citrix cluster.

The same options as the above apply, but instead of having the check broadcast for a browser you point it to a specific server.

command_name    check_citrix_broadcast
command_line    $USER1$/check_citrix -C $HOSTADDRESS$ -W "$ARG1$" -P "$ARG2$"

This check should be added to each of your Citrix server hosts.

• op5 System 2.8
• op5 Monitor version 3.0.2
• op5 Statistics 2.8
• op5 LogServer 1.2

• upgrade op5 Statistics from 2.8 to 2.10

• upgrade op5 LogServer from 1.2 to 1.4

• Follow "op5_System_3.0_upgrade-guide". ! Don't forget to move the backup-file created by the  prepare-script away from the local filesystem before overwriting the whole filesystem.

• upgrade op5 Statistics from 2.10.4 to 3.0.latest
• upgrade op5 Monitor from 3.0.6 to 4.1.4 to 4.2.latest

• re-install the LogServer if possible, it will save you a lot of time and data conversions, or upgrade op5 LogServer 2.0.0 to 2.1.0 to 3.0.4 to 3.2.4 to 3.3.0 to 3.4.0 to 3.5.latest

• upgrade op5 System 3.0.0 to 3.3.0

op5 Monitor includes support for graphing what's known as "performance data" returned by check plugins that support this feature.

Performance data can be anything that gives a more detailed picture of a particular check's performance characteristics than the OK/WARNING/CRITICAL levels that Monitor reacts to.

For example, check_ping returns performance data for packet loss and round trip times. This data is stored by Monitor and used to create graphs for different time periods, such as the last 24 hours and past week. This feature can be very helpful in identifying trends or potential problems in a network.

The purpose of this article is to describe how one can add support for new plugins to Monitor's graph system, and create what is known as "templates" for the grapher.

To be able to complete this how-to you will need the following:

• op5 Monitor installed correctly
• a host to which you can add a service for testing purposes
• a check returning performance data that you want to graph (or use the dummy check)

Execute the check and observe its output, from the command line or "Test this service"

This is the output of check_ping:

OK - 127.0.0.1: rta 0.007ms, lost 0%|
rta=0.007ms;200.000;500.000;0; pl=0%;40;80;;

The data in the output _after_ the | (pipe) sign is performance data. If your check output contains something like this, it supports performance data and can be graphed.

For demonstration purposes we will use a dummy check that we will create a graph template for. If you have a check in your system already that returns performance data which you want to create a graph for, feel free to skip this section and replace references to check_dummy_howto with your own check name.

Create the file /opt/plugins/check_dummy_howto and fill it with the below script.

#!/bin/bash
RANGE=100
DS1=$RANDOM
let "DS1 %= $RANGE"
RANGE=500
DS2=$RANDOM
let "DS2 %= $RANGE"
/opt/plugins/check_dummy 0 \
"Perfdata graph dummy check|pct=$DS1%;80;95;0; val=$DS2;350;450;0;"

Make this script executable by everyone:

chmod a+x /opt/plugins/check_dummy_howto

Try executing the file, you should see output similar to this (all on one line):

# /opt/plugins/check_dummy_howto
OK: Perfdata graph dummy check|pct=11%;80;95;;
val=139;350;450;;

Now, open up op5 Monitor's web interface and go to Configure. Add a new check command with the below settings:

command_name: check_dummy_howto
command_line: $USER1$/check_dummy_howto

And save the command. Then add a service to a non-critical host using this check_command. Remember to verify that the check works as expected by using "Test this service"

If everything checks out, save the configuration. Monitor will now start to execute check_dummy_howto with its regular check interval, and save the performance data returned by the script. Now is a good time for a coffee break, to let Monitor gather some data to work with.

If you go to the dummy service we created above in the web interface, and click the statistics icon to show the graphs of performance data, you will see that Monitor is already graphing the results of the plugin. However, in the low right corner of the graphs it will say "Default Template", because Monitor does not know what kind of values it is plotting which means that the graphs are very generic.

Let's build a custom template to display this performance data in a more effective and prettier way. Our new template should:

• combine the two series of data our plugin generates into a single graph
• use separate colors to differentiate between the data series
• have custom titles and legends describing the graphed data.

It doesn't sound like much, but it makes a lot of difference when you see the graph!

op5 Monitor includes many graph templates that support different plugins included in the distribution. You can find the included template files in this directory:

/opt/monitor/op5/pnp/templates.dist

Templates are PHP files that are named after the check plugin they work with. So for example a check_command called check_ping has a template with the name check_plugin.php.

When creating a custom template file you place it in this directory:

/opt/monitor/op5/pnp/templates

Let's go ahead and create an empty .php file named after the plugin we want to make a custom graph for. If you are using the dummy plugin we created earlier, call it:

/opt/monitor/op5/pnp/templates/check_dummy_howto.php

Remember to start incapsulate the code between:

The only thing a graph template file should do is to set two variables. It must not produce any output, and it must be valid PHP (which is not that hard to do!).

The two variables are arrays named $opt and $def. It is optional to set a value in $opt, but $def always needs to be defined. Each graph has exactly one value in each of these arrays. So, if you wanted to create three graphs, you would have three values in $opt and three values in $def. The values are simply long text strings that contain arguments and data to the graph engine.

Our dummy plugin returns two performance data values, two series of data, and a default template would create one graph per series. But we want to show both series of data in one graph, so we will only have one value in each of $opt and $def.

Let's start by adding our first data series:

$def[1] = "";

$def[1] .= "DEF:ds1=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .= "LINE:ds1#00FF00:\"Percentage points \" " ;

Let's go over what these lines do: the first line only initializes $def[1] with an empty string, so that we don't have to remember to change the assignment operator in later lines.

The second line DEFines a data series (or data source), calls it ds1 and gives it the value of the first data series returned by our plugin. AVERAGE only means that the graph engine uses the AVERAGE of the values returned, if they are several. The dummy plugin we are using only returns one value per run and series, so this has no effect for us.

The third line tells the graph engine how to draw the data we just defined. We tell it to draw a line in the color #00FF00 (which is the hexadecimal RGB value for green) and give the line a legend with the name "Percentage points".

That was pretty simple, right? You can go ahead and take a look at the statistics screen of your service in Monitor now. If everything is working as it should, you should now see a clean graph with a single green line showing the first of the data series returned by your plugin.

We can add the second data series to the graph by simply adding these two lines:

$def[1] .= "DEF:ds2=$rrdfile:$DS[2]:AVERAGE " ;
$def[1] .= "LINE:ds2#0000FF:\"Pirate population\" " ;

Set up a good title for your graph and y-axis with this (all on one line):

$opt[1] = "--title \"Pirate population / global temperature change\"
 --vertical-label \"pirates / degrees\"";

Try changing the second data series to draw an AREA instead of a LINE:

$def[1] .= "AREA:ds2#0000FF:\"Pirate population\" " ;

Have a look at your graphs now. Notice that the first line has disappeared behind the blue area. This is because graph objects are drawn in the order they are specified in the template file, so the first line is drawn and then the blue area is drawn over it.

If you change places between the first and second statements, you can see that the green line is now drawn over the blue area.

If you want, you can change the first line to be an area as well:

$def[1] .= "AREA:ds1#00FF00:\"Global temperature increase in %\":STACK " ;

That last part, ":STACK", means that the second area will be stacked on top of the first. This is a useful way to show values that add up to a total, such as bandwidth usage per customer, or memory usage per application.

This how-to only scratches the surface of what is possible to do with graph templates in op5 Monitor. If you want to learn more, you can find documentation for the components here:

RRDTool: http://oss.oetiker.ch/rrdtool/doc/index.en.html

Since SSL certificates created for your op5 system have an expiration date, you might be required to renew it when it expires to avoid web browser error messages.

This can easily be done with just a few commands.

• Root command line access to the op5 server
• Web browser to verify the new certificate

First, log in as root using either some form of ssh client, or directly onto the console of the server.

We begin with creating a certificate request. Below the commands I have chosen to include the actual questions that you will be asked during creation of the request, together with some example answers in bold text. The ones left blank in the example can just be skipped by simply pressing [Enter]. Also note, that if you have added an entry in your DNS for the op5 server that you use to reach it, for example monitor.op5.com, this should be entered when asked for Common Name.

# cd /etc/pki/tls
# openssl req -new -key private/localhost.key -out /tmp/localhost.csr

Country Name (2 letter code) [GB]:SE
State or Province Name (full name) [Berkshire]:Vastra Gotalands Lan
Locality Name (eg, city) [Newbury]:Gothenburg
Organization Name (eg, company) [My Company Ltd]:op5 AB
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:monitor.op5.com
Email Address []:root@op5.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Next step is to sign the request you just created. Here you have the chance to specify for how long the certificate should be valid. In the example below I chose one year, but if you feel you want a certificate to last for longer before renewing it again, feel free to change the number after the -days parameter to something you find more suitable.

# openssl x509 -req -days 365 -in /tmp/localhost.csr \
-signkey private/localhost.key -out /tmp/localhost.crt

Finally, we make a backup of our old certificate, copy our new certificate into place, restart the apache web server, and clean up /tmp.

# cp certs/localhost.crt certs/localhost.crt.old
# cp /tmp/localhost.crt certs/localhost.crt
# service httpd restart
# rm /tmp/localhost.csr

When done with the above steps, check and verify the new certificate with your web browser.