HA Proxy for Redis Cluster
Introduction
HAProxy is a network software application that offers high availability, load balancing, and proxying for TCP and HTTP network applications. For our Redis cluster, we will be using HAProxy for the main purpose of high availability.HAProxy 1.5+ comes with a new built-in TCP health check feature for Redis to peform an automatic failover. To void having to change Redis IP/Port in the front-end client application after each failover, we are going to setup HAProxy with the TCP health check to test if a Redis instance is a master or slave to make sure it only proxy connections to an active master instance.
Our current architecture looks like this (click image above to view). Please note that the design may change as new ideas come in)
Proxys
redis-proxy-1
redis-proxy-2
Virtual IP
float_ip
Redis Nodes:
Master: Redis-1-1 -> Slave: Redis-1-2
Master: Redis-2-1 -> Slave: Redis-2-2
Master: Redis-3-1 -> Slave: Redis-3-2
Sentinel:
Sentinel 1: Installed on the same machine as Redis-1-1
Sentinel 2: Installed on the same machine as Redis-2-1
Sentinel 3: Installed on the same machine as Redis-3-1
##NOTE - Each sentinel instance monitors all three Redis Masters in the Cluster.
#Info about their slaves and other monitoring sentinel instances are auto-discovered.
KeepAlived:
Keepalived 1: Installed on the same machine as redis-proxy-1
Keepalived 2: Installed on the same machine as redid-proxy-2
Installation
Ubuntu 14.04 does not ship with HAProxy 1.5 (well, it currently does not yet). So will are going to download our HAProxy from a PPA and install it using apt-get. On redid-proxy-1 and redid-proxy-2, run the following commands:
sudo add-apt-repository ppa:vbernat/haproxy-1.5
sudo apt-get update
apt-get dist-upgrade
Now we can install HAProxy on servers redis-proxy-1 and redis-proxy-2 with the following command:
sudo apt-get install haproxy
If all went well, we should have a successful HAProxy version 1.5 installation that works fine on Ubuntu 14.04.
Configure HAProxy for Redis
The first thing we are going to do is to replace the default harpy.cfg file with a new file and config. Do this on both proxy servers:
sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.original
On both proxy servers, lets create a new /etc/haproxy/haproxy.cfg file and enter the following config:
defaults REDIS
mode tcp
timeout connect 3s
timeout server 6s
timeout client 6s
# Specifies listening socket for accepting client connections using the default
# REDIS TCP timeout and backend bk_redis tcp health check.
frontend ft_redis
bind *:6379 name redis "This is where the float ip will be bound
default_backend bk_redis
# Specifies the backend redis proxy server tcp health settings
# to make sure it only forward incoming connections to reachable a master.
backend bk_redis
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
server redis-1-1 ip_here:6379 check inter 1s
server redis-2-1 ip_here:6379 check inter 1s
server redis-3-1 ip_here:6379 check inter 1s
Restart the harpy with the following command:
#/etc/init.d/haproxy restart
Install Keepalived
The keepalived will help us monitor if the active HAProxy currently using the float-ip is down. If it , then it will transfer the float ip to the backup HAPRoxy server. If the primary HAProxy happens to be back up and running, the float ip will be automatically be transferred back to it.
Install keepalived on both the proxy server instances using the following command:
sudo apt-get install keepalived
After installation, we should create a new keepalived config file (/etc/keepalived/keepalived.conf) on the primary (active) node:
vrrp_script haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101 # 101 on master, 100 on backup
virtual_ipaddress {
float_ip/23 dev eth0 label eth0:0
}
track_script {
haproxy
}
NOTE: Change the ethernet interfaces appropriately.
Restart keepalived with the following command:
# /etc/init.d/keepalived restart
That is basically it with the high availability setup for Redis using HAproxy and Keepalived and a float IP
REFERENCES
[1] https://www.vultr.com/docs/installing-and-configuring-haproxy-on-ubuntu-14-04
[2] https://support.pivotal.io/hc/en-us/articles/205309388-How-to-setup-HAProxy-and-Redis-Sentinel-for-automatic-failover-between-Redis-Master-and-Slave-servers
[3] http://www.101tech.net/2014/08/08/highly-available-redis-cluster/
[4] https://robertianhawdon.me.uk/2014/02/11/sysops-installing-a-high-availability-redis-service-on-centos-6-x-in-windows-azure/
[5] http://redis.io/topics/sentinel-clients
No comments:
Post a Comment