High Availability¶
MapD provides support for High Availability (HA). The goal is to meet service level agreements for performance and uptime appropriate to your business case.
MapD HA provides redundancy to eliminate single points of failure and crossover systems that continue operations when a component goes offline.
When a server is running in HA mode, it is assigned to a specific HA group. All servers in the HA group eventually receive all updates applied to any server in the same HA group.
Redundancy¶
Incoming bulk data is seamlessly distributed to multiple MapD databases. As HA group members receive each update, backend synchronization orchestrates and manages replication, then updates the MapD servers in the HA group using Kakfa topics as a distributed resilient logging system.
To avoid double handling of larger file-based bulk operations such as COPY FROM, MapD HA uses a distributed redundant file system to store data. All nodes in the HA group must have access to the distributed file system. By default, the GlusterFS distributed file system is installed, but in principle you can use any fully featured DFS.
Load Balancing and Failure Detection¶
The MapD HA server uses load balancing to maximize performance and reliability. A load balancer distributes users across the available MapD Servers, allowing improved concurrency and throughput as more servers are added. If a MapD server becomes unavailable, the load balancer redirects traffic to a different available MapD server, preserving availability with a reduction in capacity.
Load balancing can be done with purpose-build hardware such as F5 or mid-tier application servers such as haproxy or node.js. In some circumstances, the application might beneficially take into consideration data locality, routing similar requests to the same server to improve performance or reduce GPU memory usage.
Currently MapD does not natively support in-session recover; a new session must be established after failure. Typically this can be handled within mid-tier application servers by retrying a request on detecting a connection failure.
HA and Distributed¶
MapD High Availability can be used in combination with MapD Distributed, allowing horizontal scale-out. Each MapD Server referred to in this topic can be configured as a cluster of machines.
MapD HA Example¶
When starting a MapD Core Database instance as part of a HA group, you must specify the following options.
| Option | Description |
|---|---|
ha-group-id |
Identifies the name of the HA group this server belongs to. This identifies the Kafka topic to be used. The topic must exist in Kafka before this server can start. |
ha-unique-server-id |
A unique id that uniquely identifies this server in the HA group. This should never be changed for the lifetime of the MapD Core Database instance. |
ha-brokers |
A list used to identify the available Kafka brokers to manage the system. |
ha-shared-data |
The parent directory of the distributed file system. Used to check that bulk load files are placed in a directory available to all HA group members. |
A mapd.conf file would look like this.
port = 9091
http-port = 9090
data = "/home/ec2-user/prod/mapd-storage/data"
read-only = false
quiet = false
ha-group-id=hag-group-1
ha-unique-server-id=mymachine
ha-brokers=mykafkabrokerlist
ha_shared_data=/mnt/dfs/mapd-storage/
[web]
port = 9092
frontend = "/home/ec2-user/prod/mapd/frontend"
Implementing HA¶
Contact MapD support for assistance with MapD HA implementation.