MapD Distributed Configuration¶

MapD supports distributed configuration. Distributed configuration allows single queries to span more than one physical host when the scale of the data is too large to fit on a single machine.

Aside from providing the capacity for additional scale, a distributed setup has other advantages. Writes to the database can be distributed across the nodes, thus speeding up import. Reads from disk are similarly accelerated. For clients who have performance requirements beyond the already high levels of performance a single node can provide, the additional GPUs possible in a distributed cluster can significantly increase read performance in many usage scenarios (with performance scaling linearly or near linearly with the number of GPUs for simple queries requiring little intercommunication between servers).

GPU processing is distributed in a similar fashion. Multiple GPUs across the cluster query data on their local hosts. This allows processing of larger datasets, distributed across multiple servers.

The MapD Distributed Cluster¶

A MapD distributed database consists of three components:

An aggregator, which is a specialized MapD Core Database instance for managing the cluster
One or more leaf nodes, each leaf being a complete MapD Core Database instance for storing and querying data
A String Dictionary Server, which is a centralized repository for all dictionary-encoded items

Conceptually, a MapD distributed database is horizontally sharded across n leaf nodes. Each leaf node holds one nth of the total dataset. Sharding currently is round-robin only. Queries and responses are orchestrated by a MapD Aggregator server.

The MapD Aggregator¶

Clients interact with the aggregator. The aggregator orchestrates execution of a query across the appropriate leaf nodes. The aggregator composes the steps of the query execution plan to send to each leaf node, and manages their results. The full query execution might require multiple iterations between the aggregator and leaf nodes before returning a result to the client.

One of the core features of MapD Core Database is backend, GPU-based rendering for data-rich charts such as point maps. When running as a distributed cluster, the backend rendering is distributed across all leaf nodes and the aggregator composes the final image.

String Dictionary Server¶

The String Dictionary Server is a new component for the MapD Distributed System. It manages and allocates IDs for dictionary encoded fields, ensuring that these IDs are consistent across the entire cluster.

The server creates a new ID for each new encoded value. For queries returning results from encoded fields, the IDs are automatically converted to the original values by the aggregator. Leaf nodes use the string dictionary for processing joins on encoded columns.

For moderate-sized configurations, the String Dictionary Server can share a host with a leaf node. For larger clusters, this service can be configured to run on a small, separate CPU-only server.

Replicated Tables¶

As noted earlier, a table is split by default to 1/nth of the complete dataset. When you create a table to be used to provide dimension information, you can improve performance by replicating its contents onto every leaf node using the partitions property. For example:

CREATE TABLE flights … WITH (PARTITIONS=’REPLICATED’)

This reduces the distribution overhead during query execution in cases where sharding is not possible or appropriate. This is most useful for relatively small, heavily used dimension tables.

Data Loading¶

Data loading on a MapD Distributed Cluster can be completed in two ways.

You can execute a COPY FROM statement to the aggregator exactly as with MapD single-node processing. The aggregator distributes data evenly across the leaf nodes.

You can also execute COPY FROM statements directly to the leaf nodes. The additional I/O and CPU resources available to load are significantly increased, potentially providing linear speed increase on the load. The downside is that you must manually manage the balanced division of data between leaf nodes.

An Example MapD Distributed Cluster¶

Assume four GPU-based machines, each with a combination of one or more CPUs and GPUs.

Hostname	IP	Purpose
Node1	10.10.10.1	Leaf Aggregator
Node2	10.10.10.2	Leaf String Dictionary Server
Node3	10.10.10.3	Leaf
Node4	10.10.10.4	Leaf

Install MapD server on each node as normal. For larger deployments, you can have the install on a shared drive.

Set up the configuration file for the entire cluster. This file is the same for all nodes.

Cluster.conf

[
  {
    "host": "node1",
    "port": 19091,
    "role": "dbleaf"
  },
  {
    "host": "node2",
    "port": 19091,
    "role": "dbleaf"
  },
 {
    "host": "node3",
    "port": 19091,
    "role": "dbleaf"
  },
  {
    "host": "node4",
    "port": 19091,
    "role": "dbleaf"
  },

  {
    "host": "node2",
    "port": 10301,
    "role": "string"
  }
]

In the Cluster.conf file, the location of each leaf node is identified as well as the location of the String Dictionary server. dbleaf is a leaf node, string is the String Dictionary Server. The port each node is listening on is also identified. These ports must match what the individual server are configured as.

Each leaf node requires a mapd.conf configuration file.

port = 19091
http-port = 19090
data = "<location>/mapd-storage/nodeLocal/data"
read-only = false
quiet = false
string-servers = "<location>/mapd-storage/cluster.conf"

The new parameter string-servers identifies the file containing the cluster configuration, to tell the leaf node where the String Dictionary Server is.

The aggregator node needs a slightly different mapd.conf.

port = 9091
http-port = 9090
data = "<location>/mapd-storage/nodeLocalAggregator/data"
read-only = false
quiet = false
cluster = "<location>/mapd-storage/cluster.conf"

[web]
port = 9092
frontend = "<location>/prod/mapd/frontend"

The new parameter cluster tells the MapD Core Database instance that it is an aggregator node, and where to find the rest of its cluster.

Implementing a MapD Distributed Cluster¶

Contact MapD support for assistance with MapD Distributed Cluster implementation.