Every computer system has the need to detect problems (ideally before they occur). Diagnostic information aids in researching the root causes of problems. This is addressed in both legacy and modern systems to varying degrees, based on the monitoring capabilities that have been instrumented in the system.
Many computer systems have been in place for a number of years and would benefit from incremental modernization using open source technologies, lowering total cost of ownership (TCO) and leaving core applications in place.
Logging is one facet of monitoring that will be explored in this month’s blog using an example of integrating error detection in Tomcat access logs in a Zabbix dashboard to detect web user errors.
Modern systems embrace the concept of centralized logging, providing benefits including:
Central place for logs (e.g., Elasticsearch)
Consistent format for output by use of a common library across applications (e.g., log4j)
Log correlation for transaction systems through use of identifiers (e.g., Fluentd filters, Elasticsearch)
Visualization dashboards (e.g., Grafana, Kibana, Zabbix)
Problem diagnosis in legacy and less modern systems require skilled technical people to find logs, analyze and correlate data.
To varying degrees, legacy and less modern systems:
Maintain centralized logs in one place.
Provide consistent format of content of log files
Provide log rollover
Use consistent log file name conventions
Our approach is to use a central tool to analyze the existing log files in real-time in order to alert system administrators to problems and provide visualizations. This includes custom techniques to extract log data for visualization and log correlation.
We use the open source Zabbix monitoring tool, chosen because of its maturity (since 2001), ease of integration and adaptability to a variety of different environments.
We show here an example of adding a simple log item to a Zabbix dashboard.
This utilizes the Tomcat log to identify user transaction errors.
Overview of steps:
An item in Zabbix is an object that contains data that is being monitored which is then used for various purposes in Zabbix, including visualization and alert triggering.
The steps to accomplish this are quite simple:
Create a new item in Zabbix to hold the log data
Add the item to a Zabbix dashboard.
Step 1: Create item: We start by logging into the Zabbix web console and selecting Configuration -> Hosts:
We then select Items on the host on which the Zabbix agent is running:
Click Create item on next screen to add a new item with the following data:
Step 2: Add item: Add the item to the Zabbix dashboard and save the change:
We’re done! Let’s go to our dashboard and check out our change:
Check back next month for Part 2: Custom Centralized Visualization.
Comments