Module fdb_rl::split_helper

source ·
Expand description

Helper module for splitting records across multiple key-value pairs.

FoundationDB has key-value size limitation of 10KB and 100KB respectively. While we do not explicitly check the size of our keys, this module provides functions that when required will split serialized record into multiple 100KB chunks and spreads them across multiple key-value pairs.

This is done by adding a suffix to the key. The keyspace for a record is organized as shown below.

|------------------------------+---------------------|
| Key                          | Value               |
|------------------------------+---------------------|
| (Subspace, Primary Key, -1,) | Record header       |
|------------------------------+---------------------|
| (Subspace, Primary Key, 0,)  | 100KB chunk or less |
|------------------------------+---------------------|
| (Subspace, Primary Key, 1,)  | 100KB chunk or less |
|------------------------------+---------------------|
| (Subspace, Primary Key, 2,)  | 100KB chunk or less |
|------------------------------+---------------------|
| ...                          | ...                 |
| ...                          | ...                 |
|------------------------------+---------------------|

There is a record header at suffix of -1. The record header is a tuple of the form:

(header_version, data_splits, incarnation, versionstamp)

  • The header_version is a number describing the version of the header. The form described above is header_version of 0.

  • data_splits is the number of splits of data that is contained within this record. At a minimum, a record will have atleast one data split. This is true even in case of an empty record.

  • incarnation is the incarnation of the record. Incarnation is managed using RecordContext and is incremented each time a record is migrated between FoundationDB clusters.

  • The versionstamp contains information about RecordVersion’s global version and local version.

There is no tuple encoding for the data. That is 100KB chunk or less value is stored in the raw format. This is because tuple encoding would introduce escape sequences which depending on the data might exceed the 100KB limit.

Note: In the key that gets generated, we flatten the primary key tuple when suffix is appended to it. The primary key is not a nested tuple.

A minimal record will be of the form

|------------------------------+---------------|
| Key                          | Value         |
|------------------------------+---------------|
| (Subspace, Primary Key, -1,) | Record header |
|------------------------------+---------------|
| (Subspace, Primary Key, 0,)  | ""            |
|------------------------------+---------------|

The functions load, save and delete does not take a value of RawRecordPrimaryKeySchema as its argument. However its implementation assumes that you are adhering to primary key schema constraint as mentioned in the documentation for RawRecord.

Warning: Functions in this module are not meant to be public. We need to make functions in this module public to support integration tests. Do not use functions in this module in your code.

Modules

  • error 🔒
    Provides error constants for split helper module.

Structs

Constants

  • SPLIT_RECORD_SIZE 🔒
    If a record is greater than this size (in bytes) it will be split into multiple key-value pairs.

Functions

  • delete
    Delete the serialized representation of a record.
  • delete_unchecked 🔒 ⚠
    Delete the serialized representation of a record without any safety checks.
  • load
    Load serialized byte array that may be split among several keys.
  • save
    Save serialized representation using multiple keys if necessary.