High Availability Software Based Load Balancer for CockroachDB

Recently, I was building an environment for a series of tests that used CockroachDB as the underlying data store.  This happens all the time, but in this instance I had the need to also build a high availability (HA) load balancer (LB) configuration.  The need was for the environment to continue to function even if one of the software based load balancers failed.  I will explain how to build redundant load balancers using keepalived and haproxy in this post.


To start out, I built the DB cluster on a number of nodes, spread across multiple availability zones and multiple geographic regions.  For this, I followed the basic instructions from the CockroachDB documentation (https://www.cockroachlabs.com/docs/v21.1/install-cockroachdb-linux).  One of the huge benefits of CRDB is that it provides redundancy and distributed execution of queries across the DB cluster.  What it does not include is any kind of load balancing, as that is beyond its purpose.  If we were to connect to any node in our cluster at this point, we could view the DBConsole or issue sql queries.  But if the node our client is connected to were to experience an infrastructure failure, our workload would stop running.



CockroachDB will generate a haproxy configuration for us if we ask it, using the command: ./cockroach gen haproxy --insecure --host $IP_OF_A_NODE_IN_THE_CLUSTER


This will create an example haproxy.cfg file with a decent set of defaults to use for our CRDB cluster.  This file can be transferred to another node that will run haproxy -f haproxy.cfg, and it will act as a basic software load balancer in front of our CRDB cluster.  This is the start of our HA software based LB configuration.  If we use this configuration file our haproxy node will act as a load balancer and begin distributing connections across all the nodes in the CRDB cluster.  This scenario is much better, as if one of the CRDB nodes has an infrastructure failure, our workload will continue to run.  But if the HAProxy node were to fail, we're dead in the water again.



If we take this configuration for haproxy and copy it to a second node running haproxy, we now have a warm spare.  This will offer us a very minimal amount of redundancy, but getting the 2nd haproxy node in place would take manual effort.  



But what if there was a way for the 2nd haproxy node to automatically take over LB duties if the first one were to fail?  If we incorporate keepalived into our design, it can make this happen for us.  In short, keepalived creates a virtual IP that you would use to access the LB.  Keepalived will watch the LB node that is offering up the VIP, and if that node were to stop responding, keepalived will recreate that VIP on the second LB node.  This gives us a redundant LB configuration within our overall system.  Of course, separate network paths, power, and storage will also play into the overall resiliency of the system.



A basic keepalived.conf file would look like below.  From one LB node to the next, the unicast_src_ip and unicast_peer will be flipped.  In this example, the VIP that is being presented is 10.13.1.40.  The config below is for my second LB node, which has an IP of 10.13.1.39 and the first LB node has an IP of 10.13.1.38.  A configuration for the first LB node would be the same with the exception of 10.13.1.39 and 10.13.1.38 being swapped.


global_defs {

  notification_email {

  }

  router_id LVS_DEVEL

  vrrp_skip_check_adv_addr

  vrrp_garp_interval 0

  vrrp_gna_interval 0

}

   

vrrp_script chk_haproxy {

  script "killall -0 haproxy"

  interval 2

  weight 2

}

   

vrrp_instance haproxy-vip {

  state BACKUP

  priority 100

  interface eth0             # Network card

  virtual_router_id 60

  advert_int 1

  authentication {

    auth_type PASS

    auth_pass 1111

  }

  unicast_src_ip 10.13.1.39  # The IP address of this machine

  unicast_peer {

    10.13.1.38               # The IP address of peer machines

  }

   

  virtual_ipaddress {

    10.13.1.40/32            # The VIP address

  }

   

  track_script {

    chk_haproxy

  }

}

Comments

Post a Comment

Popular posts from this blog

CockroachDB Backups, Exports, and Archives

Today I would like to talk about the differences between backups, exports, and archives as they relate to CockroachDB.  Let's start by deciding, in general, what these three ideas are supposed to accomplish. Backing up data is a concept that nearly everyone in the IT field has encountered at some point.  Many of us have had to work with various backup systems, differing capabilities, and various requirements.  Backups exist to satisfy the use case where we need to restore from some form of disaster or loss and get a system back to an operational state.  There can be various requirements in the form of encryption, retention, frequency, and so forth.  CockroachDB is highly performant in this situation as both backups and restores are executed in a parallel manner, with work spread through the cluster and there is the ability to backup the MVCC history as well.  The syntax for executing backups and restores can be found in the excellent CRDB documentation....
Read more

Data Center Failures and CRDB Replication

Image
Data Center Failures and CRDB Replication David Lukens, 2021-04-03 Executive Summary The purpose of this document is to explain how replication of data within CockroachDB can be used to tolerate various failure scenarios and the impacts to query execution times. This simulation will explore the effect of different replication factors on a database cluster that spans multiple Availability Zones in multiple Regions. Overall, more replicas allows for more nodes to be down at the same time, while both R=5 and R=7 allow for the loss of an entire region. But, counterintuitively, R=7 only allows for a single Availability Zone to be lost, while R=5 allows for two. Description In a discussion, the question came up, “What happens to geographically diverse replicas when something fails?” The result was a simulation of geo-partitioned replicas and creation of a visual representation to understand this. This was for a 15 node CockroachDB cluster, with five Availability Zones (AZs or Data Cent...
Read more

Bulk Update of data in CockroachDB

  Working with customers always provides a large number of topics to discuss; however, in recent weeks the same type of scenario has surfaced a number of times.  "I have a very large table and I want to update a large portion of these records.  I wrote an UPDATE sql statement but it never finishes." So how does this situation come about?  There are three main scenarios that have repeatedly led to the need to bulk update data within CockroachDB. Data from a legacy datastore has been imported into CockroachDB without any sanitization.  Ideally, some form of ETL workflow has been utilized and only clean data is being stored in CockroachDB. An application inserts data over a period of time that also hasn't been properly sanitized. A business use case arises that entails updating data in place.   Now, once the malformed data is in CockroachDB, something needs to be done to fix the data. Let's take an example table like the one below.   table_name ...
Read more