猿教程 Logo

如何使用Icinga在Ubuntu 16.04上监控主机和服务 (How To Monitor Hosts and Services with Icinga on Ubuntu 16.04)


Introduction

Icinga is an open-source monitoring system used to monitor the health of networked hosts and services. In this tutorial we will use Icinga to set up two different kinds of monitoring configurations. The first is based on simple network checks of your host's external services, such as making a periodic HTTP request to your website. The other configuration uses a software agent running on the host to gather more detailed system information such as load and number of running processes.


Prerequisites

Before starting this tutorial, you should have completed the previous tutorial in this series, How To Install Icinga and Icinga Web on Ubuntu 16.04. This will leave you with the Icinga core and Icinga Web interface running on a single host, which we'll refer to as the icinga-master node throughout.

You will also need some servers to monitor. We will use two Ubuntu 16.04 servers with Apache installed for our examples. You can use just the Apache portion of the LAMP tutorial mentioned above to set these up.


Step 1 – Setting up Simple Host Monitoring

One simple way to monitor a server with Icinga is to set up a regular check of its externally available services. So for a web host, we'd regularly ping the server's IP address and also try to access a web page. This will tell us if the host is up, and if the web server is functioning correctly.

Let's set up monitoring for a web server. Pick one of the Apache servers mentioned as a prerequisite and make sure the default Apache placeholder page is being served properly. We will call this server web-1.example.com. We won't need to log into it at all, all the health checks will be configured and executed on the master node.

Note: Icinga always defaults to using the Fully Qualified Domain Name (FQDN) of any host it's dealing with. A FQDN is a hostname plus its domain name, so web-1.example.com, for example. If you don't have a proper domain set up for a host, the FQDN will be something like web-1.localdomain. These are fine to use, just be consistent, and if you don't have a "real" FQDN always use the server's IP address in any Icinga address field you configure.

Log into the master node. To add a new host, we need to edit Icinga's hosts.conf file. Open it in a text editor:

$ sudo nano /etc/icinga2/conf.d/hosts.conf
sudo nano /etc/icinga2/conf.d/hosts.conf

This will open a file with some explanatory comments and a single host block defined. The existing object Host NodeName configuration block defines the icinga-master host, which is the host we installed Icinga and Icinga Web on. Position your cursor at the bottom of the file, and add a new host:

. . .
object Host "web-1.example.com" {
  import "generic-host"
  address = "web-1_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
}

This defines a host called web-1.example.com, imports some default host configs from a template called generic-host, points Icinga to the correct IP address, and then defines a few variables that tell Icinga to check for an HTTP response at the root (/) URL and notify the icingaadmins group via email when problems occur.

Save and close the file, then restart Icinga:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

Switch back to the Icinga Web interface in your browser. The interface updates itself fairly rapidly, so you don't need to refresh the page. The new host information should populate in short order, and the health checks will change from Pending to Ok once Icinga gathers enough information.

This is a great way to monitor external services on a host, and there are other checks available for SSH servers, SMTP, and so on. But it'd also be nice to know more details about the internal health of the servers we're monitoring.

Next, we'll set up monitoring via an Icinga agent, so we can keep an eye on more detailed system information.


Step 2 – Setting up Agent-based Monitoring

Icinga provides a mechanism for securely communicating between a master and client node in order to run more extensive remote health checks. Instead of only knowing that our web server is successfully serving pages, we could also monitor CPU load, number of process, disk space and so on.

We're going to set up a second server to monitor. We'll call it web-2.example.com. We need to install the Icinga software on the remote machine, run some setup wizards to make the connection, then update some configuration files on the Icinga master node.

Note: There are many ways to architect an Icinga installation, complete with multiple tiers of master/satellite/client nodes, high-availability failover, and multiple ways to share configuration details between nodes. We are setting up a simple two tier structure with one master node and multiple client nodes. Further, all configuration will be done on the master node, and our health check commands will be scheduled on the master node and pushed to the clients. The Icinga project calls this setup Top Down Command Endpoint mode.


Set up the Master Node

First, we need to set up the master node to make client connections. We do that by running the node setup wizard on our master node:

$ sudo icinga2 node wizard
sudo icinga2 node wizard

This will start a script that asks us a few questions, and sets things up for us. In the following, we hit ENTER to accept most defaults. Non-default answers are highlighted. Some informational output has been removed for clarity:

Output
       Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specify the common name (CN) [icinga-master.example.com]: ENTER to accept the default, or type your FQDN
Checking for existing certificates for common name 'icinga-master.example.com'...
Certificates not yet generated. Running 'api setup' now.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
Please specify the API bind host/port (optional): ENTER
Bind Host []: ENTER
Bind Port []: ENTER
Done.

Now restart your Icinga 2 daemon to finish the installation!

Restart Icinga to finish updating the configuration:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

Open up a firewall port to allow external connections to Icinga:

$ sudo ufw allow 5665
sudo ufw allow 5665

Now we'll switch to the client node, install Icinga and run the same wizard.


Set up the Client Node

Log into the server we're calling web-2.example.com. We need to install the Icinga repository again, and then install Icinga itself. This is the same procedure we used on the master node. First install the key:

$ curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -
curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -

Open the icinga.list file:

$ sudo nano /etc/apt/sources.list.d/icinga.list
sudo nano /etc/apt/sources.list.d/icinga.list

Paste in the repository details:

deb https://packages.icinga.com/ubuntu icinga-xenial main

Save and close the file, then update the package cache:

$ sudo apt-get update
sudo apt-get update

Then, install icinga2. Note that we do not need the icinga2-ido-mysql package that we installed on the master node:

$ sudo apt-get install icinga2
sudo apt-get install icinga2

Now we run through the node wizard on this server, but we do a satellite config instead of master:

$ sudo icinga2 node wizard
sudo icinga2 node wizard
Output
       Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: y
Starting the Node setup routine...
Please specify the common name (CN) [web-2.example.com]: ENTER
Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga-master.example.com
Do you want to establish a connection to the master from this node? [Y/n]: y
Please fill out the master connection information:
Master endpoint host (Your master's IP address or FQDN): icinga-master_server_ip
Master endpoint port [5665]: ENTER
Add more master endpoints? [y/N]: ENTER
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [icinga-master_server_ip]: ENTER
Port [5665]: ENTER

The wizard will now fetch the public certificate from our master node and show us its details. Confirm the information, then continue:

Output
       . . .
Is this information correct? [y/N]: y
information/cli: Received trusted master certificate.

Please specify the request ticket generated on your Icinga 2 master.
 (Hint: # icinga2 pki ticket --cn 'web-2.example.com'):

At this point, switch back to your master server and run the command the wizard prompted you to. Don't forget sudo in front of it:

$ sudo icinga2 pki ticket --cn 'web-2.example.com'
sudo icinga2 pki ticket --cn 'web-2.example.com'

The command will output a key. Copy it to your clipboard, then switch back to the client node, paste it in and hit ENTER to continue with the wizard.

Output
       . . .
information/cli: Requesting certificate with ticket '5f53864a504a1c42ad69faa930bffa3a12600546'.

Please specify the API bind host/port (optional):
Bind Host []: ENTER
Bind Port []: ENTER
Accept config from master? [y/N]: n
Accept commands from master? [y/N]: y
Done.

Now restart your Icinga 2 daemon to finish the installation!

Now open up the Icinga port on your firewall:

$ sudo ufw allow 5665
sudo ufw allow 5665

And restart Icinga to fully update the configuration:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

You can verify there's a connection between the two servers with netstat:

$ netstat | grep :5665
netstat | grep :5665

It might take a moment for the connection to be made. Eventually netstat will output a line showing an ESTABLISHED connection on the correct port.

Output
       tcp        0      0 web-2_server_ip:58188     icinga-master_server_ip:5665     ESTABLISHED

This shows that our servers have connected and we’re ready to configure the client checks.


Configure the Agent Monitoring

Even though the master and client are now connected, there's still some setup to do on the master to enable monitoring. We need to set up a new host file. Switch back to the master node.

One important level of organization in an Icinga install is the concept of a zone. All client nodes must create their own zone, and report to a parent zone, in this case our master node. By default our master node's zone is named after its FQDN. We'll make a directory named after our master zone within Icinga's zones.d directory. This will hold the information for all the master zone's clients.

Create the zone directory:

$ sudo mkdir /etc/icinga2/zones.d/icinga-master.example.com
sudo mkdir /etc/icinga2/zones.d/icinga-master.example.com

We're going to create a services configuration file. This will define some service checks that we'll perform on any remote client node. Open the file now:

$ sudo nano /etc/icinga2/zones.d/icinga-master.example.com/services.conf
sudo nano /etc/icinga2/zones.d/icinga-master.example.com/services.conf

Paste in the following, then save and close:

apply Service "load" {
  import "generic-service"
  check_command = "load"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

apply Service "procs" {
  import "generic-service"
  check_command = "procs"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

This defines two service checks. The first will report on the CPU load, and the second will check the number of processes on the server. The last two lines of each service definition are important. command_endpoint tells Icinga that this service check needs to be sent to a remote command endpoint. The assign where line automatically assigns the service check to any host that has a client_endpoint variable defined.

Let's create such a host now. Open a new file in the zone directory we previously created. Here we've named the file after the remote host:

$ sudo nano /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com<^>.conf
sudo nano /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com<^>.conf

Paste in the following configuration, then save and close the file:

object Zone "web-2.example.com" {
  endpoints = [ "web-2.example.com" ]
  parent = "icinga-master.example.com"
}

object Endpoint "web-2.example.com" {
  host = "web-2_server_ip"
}

object Host "web-2.example.com" {
  import "generic-host"
  address = "web-2_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.client_endpoint = name
}

This file defines a zone for our remote host, and ties it back to the parent zone. It also defines the host as an endpoint, and then defines the host itself, importing some default rules from the generic-host template. It also sets some vars to create an HTTP check and enable email notifications. Note that because this host has vars.client_endpoint = name defined, it will also be assigned the service checks we just defined in services.conf.

Restart Icinga to update the config:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

Switch back to the Icinga Web interface, and the new host will show up with checks Pending. After a few moments, those checks should turn Ok. This means our client node is successfully running checks for the master node.


Conclusion

In this tutorial we set up two different types of monitoring with Icinga, external service checks and agent-based host checks. There's much more to learn about configuring and working with Icinga, so you'll likely want to dive deeper into Icinga's extensive documentation.

Note that if you get to a point where you require custom check commands, you'll need to sync these from the master to the client nodes using a global configuration zone. You can find out more about this specific feature here.

Finally, if you have a large number of servers to monitor, you might look into using configuration management software to automate your Icinga configuration updates. Our tutorial series Getting Started with Configuration Management give a comprehensive overview of the concepts and software involved.

介绍

Icinga是一个开源监控系统,用于监控网络主机和服务的健康状况。 在本教程中,我们将使用Icinga来设置两种不同类型的监视配置。 第一个是基于对主机外部服务的简单网络检查,例如定期向您的网站发送HTTP请求。 另一个配置使用在主机上运行的软件代理来收集更详细的系统信息,例如负载和正在运行的进程数。


先决条件

在开始本教程之前,您应该已经完成了本系列的上一个教程,如何在Ubuntu 16.04上安装Icinga和Icinga Web。 这将使您能够在单个主机上运行Icinga内核和Icinga Web界面,我们将其称为icinga-master节点。

您还需要一些服务器进行监控。 我们将使用两台安装Apache的Ubuntu 16.04服务器作为示例。 您可以使用上面提到的LAMP教程的Apache部分来设置这些。


步骤1 - 设置简单的主机监控

使用Icinga监控服务器的一个简单方法是定期检查其外部可用的服务。 因此,对于网络托管商,我们会定期ping服务器的IP地址,并尝试访问网页。 这将告诉我们主机是否启动,以及Web服务器是否正常运行。

我们设置一个Web服务器的监控。 选择其中一个提到的Apache服务器作为先决条件,并确保正确提供默认的Apache占位符页面。 我们会将此服务器称为web-1.example.com。 我们完全不需要登录,所有的健康检查都将在主节点上进行配置和执行。

注意:Icinga总是默认使用其处理的任何主机的完全限定域名(FQDN)。 FQDN是一个主机名及其域名,例如web-1.example.com。 如果您没有为主机设置适当的域,则FQDN将类似于web-1.localdomain。 这些都可以使用,只是一致,如果没有“真正的”FQDN,在您配置的任何Icinga地址字段中始终使用服务器的IP地址。

登录到主节点。 要添加新主机,我们需要编辑Icinga的hosts.conf文件。 在文本编辑器中打开它:

$ sudo nano /etc/icinga2/conf.d/hosts.conf
sudo nano /etc/icinga2/conf.d/hosts.conf

这将打开一个文件,并提供一些解释性注释,并定义单个主机块。 现有对象Host NodeName配置块定义了icinga-master主机,这是我们安装Icinga和Icinga Web的主机。 将光标放在文件的底部,并添加一个新的主机:

. . .
object Host "web-1.example.com" {
  import "generic-host"
  address = "web-1_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
}

这定义了名为web-1.example.com的主机,从名为generic-host的模板导入一些默认的主机配置,将Icinga指向正确的IP地址,然后定义一些变量,指示Icinga检查HTTP响应 根(/)URL,并在发生问题时通过电子邮件通知icingaadmins组。

保存并关闭文件,然后重新启动Icinga:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

切换回浏览器中的Icinga Web界面。 界面更新本身相当快速,所以你不需要刷新页面。 新的主机信息应该按顺序填写,一旦Icinga收集足够的信息,健康状况检查将从待处理状态更改为OK。

这是监控主机外部服务的好方法,还有其他可用于SSH服务器,SMTP等的检查。 但是,了解有关我们正在监控的服务器的内部运行状况的更多细节也是很好的。

接下来,我们将通过Icinga代理设置监控,因此我们可以关注更详细的系统信息。


步骤2 - 设置基于代理的监控

Icinga提供了一种在主节点和客户机节点之间进行安全通信的机制,以便运行更广泛的远程健康检查。 我们也可以监视CPU负载,进程数,磁盘空间等,而不是只知道我们的Web服务器成功地提供了页面。

我们要设置第二台服务器进行监控。 我们称它为web-2.example.com。 我们需要在远程机器上安装Icinga软件,运行一些安装向导进行连接,然后更新Icinga主节点上的一些配置文件。

注意:有许多方法构建Icinga安装,包括多层主/卫星/客户端节点,高可用性故障切换以及在节点之间共享配置详细信息的多种方法。 我们正在建立一个简单的两层结构,一个主节点和多个客户端节点。 此外,所有配置将在主节点上完成,我们的健康检查命令将在主节点上调度并推送到客户端。 Icinga项目调用此设置自顶向下命令端点模式。


设置主节点

首先,我们需要设置主节点进行客户端连接。 我们通过在主节点上运行节点设置向导来执行此操作:

$ sudo icinga2 node wizard
sudo icinga2 node wizard

这将开始一个脚本,问我们几个问题,并为我们设置了一些事情。 在下文中,我们按ENTER键接受大多数默认值。 非默认答案突出显示。 为清楚起见,删除了一些信息输出:

Output
       Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specify the common name (CN) [icinga-master.example.com]: ENTER to accept the default, or type your FQDN
Checking for existing certificates for common name 'icinga-master.example.com'...
Certificates not yet generated. Running 'api setup' now.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
Please specify the API bind host/port (optional): ENTER
Bind Host []: ENTER
Bind Port []: ENTER
Done.

Now restart your Icinga 2 daemon to finish the installation!

重新启动Icinga以完成更新配置:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

打开防火墙端口以允许外部连接到Icinga:

$ sudo ufw allow 5665
sudo ufw allow 5665

现在我们将切换到客户端节点,安装Icinga并运行相同的向导。


设置客户端节点

登录到我们称之为web-2.example.com的服务器。 我们需要重新安装Icinga存储库,然后安装Icinga本身。 这是我们在主节点上使用的相同的过程。 首先安装密钥:

$ curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -
curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -

打开icinga.list文件:

$ sudo nano /etc/apt/sources.list.d/icinga.list
sudo nano /etc/apt/sources.list.d/icinga.list

粘贴在存储库中的详细信息:

deb https://packages.icinga.com/ubuntu icinga-xenial main

保存并关闭文件,然后更新包缓存:

$ sudo apt-get update
sudo apt-get update

然后,安装icinga2。 请注意,我们不需要在主节点上安装的icinga2-ido-mysql包:

$ sudo apt-get install icinga2
sudo apt-get install icinga2

现在我们运行这个服务器上的节点向导,但是我们做一个卫星配置而不是主机:

$ sudo icinga2 node wizard
sudo icinga2 node wizard
Output
       Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: y
Starting the Node setup routine...
Please specify the common name (CN) [web-2.example.com]: ENTER
Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga-master.example.com
Do you want to establish a connection to the master from this node? [Y/n]: y
Please fill out the master connection information:
Master endpoint host (Your master's IP address or FQDN): icinga-master_server_ip
Master endpoint port [5665]: ENTER
Add more master endpoints? [y/N]: ENTER
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [icinga-master_server_ip]: ENTER
Port [5665]: ENTER

该向导现在将从主节点获取公共证书,并向我们显示其详细信息。 确认信息,然后继续:

Output
       . . .
Is this information correct? [y/N]: y
information/cli: Received trusted master certificate.

Please specify the request ticket generated on your Icinga 2 master.
 (Hint: # icinga2 pki ticket --cn 'web-2.example.com'):

此时,切换回主服务器并运行该向导提示您的命令。 不要在前面忘记sudo:

$ sudo icinga2 pki ticket --cn 'web-2.example.com'
sudo icinga2 pki ticket --cn 'web-2.example.com'

该命令将输出一个键。 将其复制到剪贴板,然后切换回客户端节点,将其粘贴并按ENTER键继续向导。

Output
       . . .
information/cli: Requesting certificate with ticket '5f53864a504a1c42ad69faa930bffa3a12600546'.

Please specify the API bind host/port (optional):
Bind Host []: ENTER
Bind Port []: ENTER
Accept config from master? [y/N]: n
Accept commands from master? [y/N]: y
Done.

Now restart your Icinga 2 daemon to finish the installation!

现在打开防火墙上的Icinga端口:

$ sudo ufw allow 5665
sudo ufw allow 5665

并重新启动Icinga以完全更新配置:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

您可以使用netstat验证两台服务器之间的连接:

$ netstat | grep :5665
netstat | grep :5665

连接可能需要一些时间。 最终,netstat将输出一条在正确端口上显示ESTABLISHED连接的行。

Output
       tcp        0      0 web-2_server_ip:58188     icinga-master_server_ip:5665     ESTABLISHED

这表明我们的服务器已连接,我们已准备好配置客户端检查。


配置代理监控

即使主服务器和客户端现在已连接,仍然有一些设置可以在主服务器上进行监控。 我们需要设置一个新的主机文件。 切换回主节点。

Icinga安装中的一个重要组织层次是区域的概念。 所有客户机节点必须创建自己的区域,并报告给父区域,在本例中为我们的主节点。 默认情况下,我们的主节点的区域以其FQDN命名。 我们将在Icinga的zones.d目录中创建一个以我们的主区域命名的目录。 这将保存所有主区客户的信息。

创建区域目录:

$ sudo mkdir /etc/icinga2/zones.d/icinga-master.example.com
sudo mkdir /etc/icinga2/zones.d/icinga-master.example.com

我们要创建一个服务配置文件。 这将定义我们将在任何远程客户机节点上执行的一些服务检查。 立即打开文件:

$ sudo nano /etc/icinga2/zones.d/icinga-master.example.com/services.conf
sudo nano /etc/icinga2/zones.d/icinga-master.example.com/services.conf

贴在下面,然后保存并关闭:

apply Service "load" {
  import "generic-service"
  check_command = "load"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

apply Service "procs" {
  import "generic-service"
  check_command = "procs"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

这定义了两个服务检查。 第一个将报告CPU负载,第二个将检查服务器上的进程数。 每个服务定义的最后两行很重要。 command_endpoint告诉Icinga该服务检查需要发送到远程命令端点。 线路自动将服务检查分配给定义了client_endpoint变量的任何主机。

现在我们来创建一个这样的主机。 在我们以前创建的区域目录中打开一个新文件。 这里我们在远程主机之后命名了该文件:

$ sudo nano /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com<^>.conf
sudo nano /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com<^>.conf

粘贴在以下配置中,然后保存并关闭文件:

object Zone "web-2.example.com" {
  endpoints = [ "web-2.example.com" ]
  parent = "icinga-master.example.com"
}

object Endpoint "web-2.example.com" {
  host = "web-2_server_ip"
}

object Host "web-2.example.com" {
  import "generic-host"
  address = "web-2_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.client_endpoint = name
}

此文件为我们的远程主机定义了一个区域,并将其绑定到父区域。 它还将主机定义为端点,然后定义主机本身,从通用主机模板导入一些默认规则。 它还设置一些var来创建HTTP检查并启用电子邮件通知。 请注意,因为这个主机已经定义了vars.client_endpoint = name,它也将被分配给我们刚刚在services.conf中定义的服务检查。

重新启动Icinga来更新配置:

$ sudo systemctl restart icinga2
sudo systemctl restart icinga2

切换回Icinga Web界面,新的主机将显示“Pending”。 过了一会儿,那些检查应该会转好了。 这意味着我们的客户端节点成功运行主节点的检查。


结论

在本教程中,我们使用Icinga,外部服务检查和基于代理的主机检查来设置两种不同类型的监视。 要了解如何配置和使用Icinga,您可能希望深入了解Icinga的大量文档。

请注意,如果您到达需要自定义检查命令的点,则需要使用全局配置区将这些从主服务器同步到客户端节点。 您可以在这里找到关于这个特定功能的更多信息。

最后,如果要监视大量的服务器,可以查看使用配置管理软件来自动执行Icinga配置更新。 我们的教程系列配置管理入门概述了所涉及的概念和软件。