Setup Grafana monitoring with Prometheus
Architecture
We have three servers we are going to be working with:
Server Name IP
grafana 192.168.2.1
db-test-1 192.168.2.10
db-test-2 192.168.2.11
Grafana and prometheus will be installed on the grafana server. The grafana server will also have node_exporter installed.
For db-test-1 and 2, we will only install node_exporter and mysqld_exporter. These two are database servers. If you have a server you want to monitor that was not a database server, then just install node_exporter as is gets system wide stats only.
Node_exporter is a prometheus exporter for server metrics. It collects stats such as CPU, load, Memory, Disk performance, disk utilization etc.
Mysqld_exporter collects MySQL metrics such as InnoDB metrics, replication, table statistics, user statistics, cluster overview etc.
Prometheus is basically an open source systems and service monitoring system. It collects metrics from other servers e.g, web servers, database servers etc, and uses graph monitoring systems like grafana to present these metrics in a meaningful way through dashboards that users can easily understand.
Prometheus Installation
First, download prometheus from the downloads page https://prometheus.io/download/wget https://github.com/prometheus/prometheus/releases/download/v1.1.2/prometheus-1.1.2.linux-amd64.tar.gz
sudo mkdir ~/Prometheus
sudo tar zxf prometheus-1.1.2.linux-amd64.tar.gz -C ~/Prometheus --strip-components=1
Config Prometheus
Now the next thing to do is get Prometheus listening to the metrics from the node exporter, for this the first thing that is needed is to create a configuration file for Prometheus.sudo vim /etc/prometheus.yml
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['192.168.2.1:9090']
- job_name: grafana-linux
static_configs:
- targets: ['192.168.2.1:9100']
labels:
alias: grafana-1
- job_name: db-test-1-linux
static_configs:
- targets: ['192.168.2.10:9100']
labels:
alias: db-test-1
- job_name: db-test-1-mysql
static_configs:
- targets: ['192.168.2.10:9104']
labels:
alias: db-test-1
- job_name: db-test-2-linux
static_configs:
- targets: ['192.168.2.11:9100']
labels:
alias: db-test-2
- job_name: db-test-1-mysql
static_configs:
- targets: ['192.168.2.11:9104']
labels:
alias: db-test-2
Config startup script for Prometheus
To enable prometheus to start at start time, configure the following /etc/init/prometheus.conf#Prometheus Upstart Script
start on startup
script
/usr/bin/prometheus -config.file /etc/prometheus.yml > /var/log/prometheus.log 2>&1 &
end script
Start prometheus
sudo service prometheus startDownload and Install Grafana
Donwload grafana from the download page http://grafana.org/download/
wget https://grafanarel.s3.amazonaws.com/builds/grafana_3.1.0-1468321182_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_3.1.0-1468321182_amd64.deb
Edit sudo vim /etc/grafana/grafana.ini (scroll to the bottom of the page and find the section below)
[dashboards.json]
enabled = true
path = /var/lib/grafana/dashboards
Fix Percona APP bugs
Run the two commands below to fix bugssudo sed -i 's/expr=\(.\)\.replace(\(.\)\.expr,\(.\)\.scopedVars\(.*\)var \(.\)=\(.\)\.interval/expr=\1.replace(\2.expr,\3.scopedVars\4var \5=\1.replace(\6.interval, \3.scopedVars)/' /usr/share/grafana/public/app/plugins/datasource/prometheus/datasource.js
sudo sed -i 's/,range_input/.replace(\/"{\/g,"\\"").replace(\/}"\/g,"\\""),range_input/; s/step_input:""/step_input:this.target.step/' /usr/share/grafana/public/app/plugins/datasource/prometheus/query_ctrl.js
Install Percona plugin
grafana-cli plugins install percona-percona-appStart grafana
sudo service grafana-server start# Log onto grafana server
http://ip-address:3000
username: admin
password: admin
To enable the Percona plugin, log into the grafana web monitoring, click on the icon on top left, then Plugins, then App tab, and enable the Percona App which should be listened there.
Installing node_exporter and mysqld_exporter
On all three servers, install node_exporter:
Download the latest version of node_exporter and mysqld_exporter from here
mkdir ~/Prometheus
wget https://github.com/prometheus/node_exporter/releases/download/0.12.0rc3/node_exporter-0.12.0rc3.linux-amd64.tar.gz
wget https://github.com/prometheus/mysqld_exporter/releases/download/0.7.1/mysqld_exporter-0.7.1.linux-amd64.tar.gz
mkdir /opt/prometheus_exporters
tar zxf node_exporter-0.12.0rc3.linux-amd64.tar.gz -C ~/Prometheus
tar zxf mysqld_exporter-0.7.1.linux-amd64.tar.gz -C ~/Prometheus
Create MySQL user account
On the db servers, create a mysql user account for mysqld_exporter to use when getting metrics from MySQl. Be careful with the permissions granted to to this user account.
mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'prometheus'@'localhost' identified by 'promPass987';
mysql> GRANT SELECT ON performance_schema.* TO 'prometheus'@'localhost';
mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'prometheus'@'localhost' identified by 'promPass987';
mysql> GRANT SELECT ON performance_schema.* TO 'prometheus'@'localhost';
Then create a file to put these credentials for mysqld_exporter that we just created
cd ~/Prometheus
vim .my.cnf
cd ~/Prometheus
vim .my.cnf
And past the following in the cnf file:
[client]
user=prometheus
password=promPass987
[client]
user=prometheus
password=promPass987
Create node_exporter startup script
Now we need to create an upstart script to start the node_exporter service at bootup. So lets create a link for the node exporter in /usr/bin
sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin
Now lets create the upstart script:
sudo vim /etc/init/node_exporter.conf
Then paste the following lines into that file:
#Prometheus Node Exporter Upstart script
start on startup
script
/usr/bin/node_exporter -collectors.enabled="diskstats,filesystem,loadavg,meminfo,netdev,stat,time,uname,vmstat"
end script
Now lets start the script with the following command:
service node_exporter start
Create mysqd_exporter startup script
Now we need to create an upstart script to start the mysqld_exporter service at bootup. So lets create a link for the node exporter in /usr/bin
sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin
Now lets create the upstart script:
sudo vim /etc/init/mysqld_exporter.conf
Then paste the following lines into that file:
#Prometheus Node Exporter Upstart script
start on startup
script
/usr/bin/node_exporter -collectors.enabled="diskstats,filesystem,loadavg,meminfo,netdev,stat,time,uname,vmstat"
end script
george.chilumbu@db-kdb-slave-1 [~] [10:37] less /etc/init/mysqld_exporter.conf
#Prometheus Mysqld Exporter Upstart script
start on startup
script
/usr/bin/mysqld_exporter -collect.perf_schema.eventsstatements=true -collect.perf_schema.indexiowaits=true -collect.binlog_size=true -collect.perf_schema.tableiowaits=true -collect.perf_schema.tablelocks=true -collect.perf_schema.file_events=true -collect.perf_schema.eventswaits=true -collect.info_schema.processlist=true -collect.info_schema.userstats=true -collect.info_schema.query_response_time=true -collect.info_schema.tables=true -collect.info_schema.tablestats=true -config.my-cnf="/opt/Prometheus/prometheus_exporters/.my.cnf"
end script
You can start the script with the following command:
service mysqld_exporter start
Finally, you should be able to view dashboard in griffin as demonstrated below. The screenshot below is the grapheme monitoring system that i setup for our team at KKBOX.
The print screen below also shows how we use griffin/prometheus/node_exporter to monitor metrics on our redis servers:
REFERENCES
[1] https://blog.100tb.com/how-to-set-up-prometheus-to-monitor-your-ubuntu-server[2] https://www.percona.com/blog/2016/02/29/graphing-mysql-performance-with-prometheus-and-grafana/
[3] https://grafana.net/plugins/percona-percona-app
No comments:
Post a Comment