The previous post discussed about the mutable data in Ampool Active Data Store (ADS). Here we will discuss about how Ampool ADS enables you to deal with very large volume of immutable data. The FTable stands for “flow-table” and it enables fast ingestion of very large amount of immutable data (aka facts data). The data is internally stored in multiple storage tiers and moved across seamlessly whenever required. The storage tiers are typically arranged in the ascending order of latency and decreasing order of the cost (per GB) associated with it. The newer data is relatively accessed much more frequently as compared to the older data. Thus, the high-demand data is stored in a tier with lowest latency whereas the data with lower demand is stored in the tiers with higher latency but lower cost.

The Ampool FTable nicely complements MTable in data warehousing use cases where mutable dimension data stored in MTable vs large ever growing facts data is stored in FTable with tiered storage. Separating these two types of tables and keeping the FTable primarily immutable we avoid the need for background data compaction and thus use SSD/Flash storage more effectively by eliminating write amplification phenomenon.

Similar to MTable, FTable also provides partitioningredundancypersistence and support for table schema with basic and complex data types. The data is partitioned across the cluster nodes using hash based partitioning using specified column values as partition key. The table could also be configured with redundancy for availability & fault tolerance. It also provides you with an option to persist the in-memory data to a persistent local disk storage so that it can be recovered in case of node failure/restarts.

Being immutable table, at high level it only supports the append and scan operations (Note: Although, to curate occasional data inconsistencies it does provide administrative operations like bulk delete/update with arbitrary filter criteria). The append enables you to ingest the data at very high ingestion rates. Either single record or bulk insertion of records is possible. A typical key-value store has a constant overhead per entry. The FTable employs block strategy for it’s in-memory layer that groups multiple records together and minimizes per record overhead and the overall memory footprint. Thus, for a typical append-only kind of tables it helps you to make optimal usage of available memory. .

FTable support INSERT_TIME as an in-built column for users to efficiently retrieve the data over a specified time range without having to scan the entire table. When the data is ingested, Ampool records the insertion-time along with each record. This insertion-time is used internally as an implicit order to store the data within a partition. This helps to boost the real-time range queries based on the insertion-time e.g. get all data for past N hours.

With FTables you could configure a hierarchy of tiers. In a recommended configuration first tier is always a memory-tier, second tier is shared nothing local disk/SSD based, while third tier is an archive tier using shared stores such as HDFS/S3 etc. FTable can also be configured to move the data from one tier to next tier based on specific policies. Both time and space-usage based policies could be configured per tier, per table. As data exceeds the time or space threshold, it is automatically moved to next tier. This allows you to make more optimal usage of the available resources in the respective tiers. Ampool FTable uses open data storage formats e.g. ORC, Parquet to store the data on persistent tiers such as local disk, HDFS/S3 etc.

The scan is another key operation on FTable. It allows you to retrieve the selected records by applying a set of arbitrary filters based on specific columns. The scan is a seamless operation that spans across all the configured tiers, as applicable. It retrieves the data from all the tiers transparently. The scan operation utilizes the data locality by applying the filter where the data is and only the matching results are sent over the network.

Modern applications that supports both real-time access and historical analysis, the recent data is accessed more frequently in a real-time manner as compared to the older data that gets utilized in batch oriented way. FTable’s tiered storage policy serves both types of workloads very effectively eliminating the need of multiple types of data stores used in a lambda architectural patten.

Similar to MTable, FTable also supports MASH Shell, Java APIs and connectors with multiple compute frameworks such as Apache SparkApache Hive for programatic access to tiered data store. In subsequent posts we will deep dive into details of these access mechanisms.

Continuing from the previous post, we now introduce MTable, one of the mechanisms to store/Query/Scan data from Ampool’s Active Data Store (ADS). MTable stands for mutable table. In subsequent posts we will learn more about MTable but at the core, it is a distributed tabular data structure that is partitioned based on a primary key. MTable allows quick lookup & update of individual table rows based on row-key but at the same time also supports very efficient scan operations and range queries based on row keys.

MTable ( Mutable In-Memory Tables ) is a collection of Rows, Row is defined as a tuple of Row-Key and set of Columns.
Each row is uniquely identified by its Row-Key. Each Column is defined by its name and type. All primitive types, Date, TimesStamp, Map, Struct and Union Types are supported with MTable.

MTables comes with two types ORDERED_VERSIONED and UNORDERED

MTable of Type ORDERED_VERSIONED store Rows in sorted order of Row Key. Row Key for the table is of type byte[] and will be stored in lexicographic order. ORDERED_VERSIONED MTable is a good fit when you want to do range queries.

MTable of Type UNORDERED stores the data in a hash map. This type is suitable for faster GETs, which is nearly equal to O(1).

MTable supports multiple versions for each Row. If the version is explicitly specified, the row is stored with that version, otherwise, the current timestamp in milliseconds is used. One can easily access the instance of the row at a particular version, which helps in implementing the features such as transactional consistency, snapshot isolation, auditing etc.

MTable is partitioned & distributed across all the members of Ampool Cluster. MTable data is partitioned into number of buckets based on the row-key. ORDERED_VERSIONED type table uses range partitioning scheme while UNORDERED table uses hash partitioning. MTable also supports colocation of partitions from different MTables that have related data entries. This helps in running the efficient join queries. Also MTable supports configurable redundancy, so that one or more copies of each bucket are stored on different nodes to achieve high availability in various node failure scenarios.

MTable optionally provides an ability to persist the data on the local disk for recovery purposes. In case of power outage or node restart data in memory is restored from the local disk storage. Ampool Data Store supports two types of persistence, Synchronous and Asynchronous i.e. when data is ingested into MTable it is synchronously or asynchronously written to local disk for recovery. Asynchronous persistence is often used as it offers the best ingestion performance.

MTable does not limit the size of data to only available memory but allows overflow of data to local disk, if memory is full. Eviction threshold is specified for each member server in the cluster such that %memory occupied is more than eviction threshold then server evicts the column values based on LRU algorithm. The row keys and pointer to column values spilled on disk are retained in memory for faster access.

Ampool Data Store also provides a mechanism for CDC ( Change Data Capture ). Using this Applications can listen to MTable events like insert, delete, and updates. These events are delivered in the order of their occurrence. Application can implement CDCEventListener to listen and take actions on those events, typically applying these events to the target store for data replication.

MTable supports data-aware computation using Co-Processor Mechanism. It provides two types of Co-Processors resembling stored procedures or database triggers.

  1. EndPoint
  2. Observer

One can invoke an Endpoint co-processor at any time from the client. The Endpoint co-processor code is executed remotely on the server close to target MTable, and results are returned to the client. Observer Co-processors are similar to database triggers i.e. they execute user provided custom code before/after the insert/update events on the MTable.

What Operations can be performed on MTable

  1. Single and Batch Put of Rows
  2. Single and Batch Get of Rows
  3. Delete a Single Row
  4. Check and Put or Delete
  5. Scan a Table with Filter
  6. Coprocessor Execution

In the subsequent posts, We will deep dive into using MTable with example code.

As mentioned in previous posts, Ampool is a memory-centric, data aware, distributed store which extends the proven open source Apache Geode project to support multiple types of application workloads. Apache Geode is a distributed in-memory key-value store which primarily supports the low latency transactional workloads using the data storage abstraction called “Region”. A Region is essentially a distributed hash map. Application specific domain objects are serialized and stored as keys & values. There are multiple types of regions primarily based on the system properties such as if data stored in the region is partitioned or fully replicated, is it persistent, does Region support overflow of values etc. Check out Geode Region Types for more details.

Ampool Active Data Store (ADS) extends Geode platform by introducing additional data storage abstractions (or data structures) so that multiple types of applications can make use of them storing and retrieving the data efficiently and also collaborate effectively. Traditionally applications cache the data in their local memory/heap from the underlying data store in various types of suitable data structures. Ampool eliminates the burden of such explicit memory management from applications and virtually provides unlimited memory space using its memory-centric Active Data Store. Applications can directly interact and manipulate the data in the data store using suitable data structures at memory-speed.

As of today, Ampool ADS adds two more data abstractions namely MTable & FTable. These abstractions primarily extend Geode’s functionality to support machine learning & analytics workloads along with OLTP.

  • MTable stands for mutable table. In subsequent posts we will learn more about MTable but at the core, it is a distributed tabular data structure that is partitioned based on a primary key. MTable allows quick lookup & update of individual table rows based on row-key but at the same time also supports very efficient scan operations and range queries based on row keys.
  • FTable stands for flow table also sometimes we refer to it as facts table as it is useful for storing immutable event log, metrics, or audit trail data referred to as “facts data” in the data warehousing context. FTable primarily supports “Append” and “Scan” operations and internally keeps records sorted based on the insertion time and partition key (which is typically one of the table columns).

In Ampool we believe that the table abstraction is much more efficient for most common operations, rather than raw key-value maps because it enables support for simple and complex typed column values, and also allows efficient projection, filter, and computation pushdowns for efficient data access and manipulation.

Both MTable and FTable provide persistence for data stored in memory as well as ability to extend it using persistent storage when data is too big to fit in available memory. Data can be evicted to disk-based storage based on configured eviction policies while data access operations transparently fetch data irrespective of where it is stored. Thus, it avoids the need for applications to have two different stores for short term vs long term data storage.

Embedding Apache Geode, Ampool ADS provides HTAP (Hybrid Transactional and Analytical Processing) capabilities. We purposely separate MTable and FTable as two separate types of tables for efficiently storing and accessing large data used in the context of warehousing resides in the Fact table and is primarily immutable. For dimension data, which is relatively smaller and often mutable, can be efficiently stored and accessed through MTable. As it changes more frequently, dimension data require periodic imports in the warehouse and adds to overall latency to the analytical processing. The ability of MTable for continuous in-place updates can significantly lower the overall latency for real-time analytics. Also, large facts table data stored in FTable spreads across multiple tiers of storage such as hot data in memory vs warm/cold data in SSD, LocalDisk or long term archival storage like S3/HDFS. Keeping data immutable avoids the costly process of compaction and re-write on the disk storage and thus enable effective usage of SSDs for faster data access as the next tier to data stored in memory.

Through connector framework, both MTable and FTable support multiple compute and streaming analytics engines on top such as Apache Spark, Apache Hive, Apache Apex as well as ability to ingest data from distributed message queue platforms such as Apache Kafka. Ampool ADS also supports native Java interface for these table abstractions so applications can directly interact with Ampool or can easily write connectors for external systems to make use of Ampool ADS for efficient storage, retrieval and manipulation of data.

In subsequent posts, we will provide details of MTable and FTable including various features, configuration options, and policies.