Types¶
Velox supports scalar types and complex types. Scalar types are categorized into a fixed set of physical types, and an extensible set of logical types. Physical types determine the in-memory layout of the data. Logical types add additional semantics to a physical type.
Physical Types¶
Each physical type is implemented using a C++ type. The table below shows the supported physical types, their corresponding C++ type, and fixed-width bytes required per value.
Physical Type |
C++ Type |
Fixed Width (bytes) |
---|---|---|
BOOLEAN |
bool |
0.125 (i.e. 1 bit) |
TINYINT |
int8_t |
1 |
SMALLINT |
int16_t |
2 |
INTEGER |
int32_t |
4 |
BIGINT |
int64_t |
8 |
HUGEINT |
int128_t |
16 |
REAL |
float |
4 |
DOUBLE |
double |
8 |
TIMESTAMP |
struct Timestamp |
16 |
VARCHAR |
struct StringView |
16 |
VARBINARY |
struct StringView |
16 |
OPAQUE |
std::shared_ptr<void> |
16 |
UNKNOWN |
struct UnknownValue |
0 |
All physical types except VARCHAR and VARBINARY have a one-to-one mapping with their C++ types. The C++ type is also used as a template parameter for vector classes. For example, vector of 64-bit integers is represented as FlatVector<int64_t> whose type is BIGINT.
OPAQUE type can be used to define custom types. An OPAQUE type must be specified wih an unique std::type_index. Values for this type must be provided as std::shared_ptr<T> where T is a C++ type. More details on when to use an OPAQUE type to define a custom type are given below.
VARCHAR, VARBINARY, OPAQUE use variable number of bytes per value. These types store a fixed-width part in the C++ type and a variable-width part elsewhere. All other types use fixed-width bytes per value as shown in the above table. For example: VARCHAR and VARBINARY FlatVectors store the fixed-width part in a StringView for each value. StringView is a struct that contains a 4-byte size field, a 4-byte prefix field, and an 8-byte field pointer that points to variable-width part. The variable-width part for each value is store in stringBuffers. OPAQUE types store variable-width parts outside of the FlatVector.
UNKNOWN type is used to represent an empty or all nulls vector of unknown type. For example, SELECT array() returns an ARRAY(UNKNOWN()) because it is not possible to determine the type of the elements. This works because there are no elements.
TIMESTAMP type is used to represent a specific point in time. A TIMESTAMP is defined as the sum of seconds and nanoseconds since UNIX epoch. struct Timestamp contains one 64-bit signed integer for seconds and another 64-bit unsigned integer for nanoseconds. Nanoseconds represent the high-precision part of the timestamp, which is less than 1 second. Valid range of nanoseconds is [0, 10^9). Timestamps before the epoch are specified using negative values for the seconds. Examples: * Timestamp(0, 0) represents 1970-01-0 T00:00:00 (epoch). * Timestamp(10*24*60*60 + 125, 0) represents 1970-01-11 00:02:05 (10 days 125 seconds after epoch). * Timestamp(19524*24*60*60 + 500, 38726411) represents 2023-06-16 08:08:20.038726411
(19524 days 500 seconds 38726411 nanoseconds after epoch).
Timestamp(-10*24*60*60 - 125, 0) represents 1969-12-21 23:57:55 (10 days 125 seconds before epoch).
Timestamp(-5000*24*60*60 - 1000, 123456) represents 1956-04-24 07:43:20.000123456 (5000 days 1000 seconds before epoch plus 123456 nanoseconds).
Logical Types¶
Logical types are backed by a physical type and include additional semantics. There can be multiple logical types backed by the same physical type. Therefore, knowing the C++ type is not sufficient to infer a logical type. The table below shows the supported logical types, and their corresponding physical type.
Logical Type |
Physical Type |
---|---|
DATE |
INTEGER |
DECIMAL |
BIGINT if precision <= 18, HUGEINT if precision >= 19 |
INTERVAL DAY TO SECOND |
BIGINT |
INTERVAL YEAR TO MONTH |
INTEGER |
DECIMAL type carries additional precision, and scale information. Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2. DECIMAL types are backed by BIGINT and HUGEINT physical types, which store the unscaled value. For example, the unscaled value of decimal 123.45 is 12345. BIGINT is used upto 18 precision, and has a range of [-10^{18} + 1, +10^{18} - 1]. HUGEINT is used starting from 19 precision upto 38 precision, with a range of [-10^{38} + 1, +10^{38} - 1].
All the three values, precision, scale, unscaled value are required to represent a decimal value.
Custom Types¶
Most custom types can be represented as logical types and can be built by extending the existing physical types. For example, Presto Types described below are implemented by extending the physical types. An OPAQUE type must be used when there is no physical type available to back the logical type.
Complex Types¶
Velox supports the ARRAY, MAP, and ROW complex types. Complex types are composed of scalar types and can be nested with other complex types.
For example: MAP<INTEGER, ARRAY<BIGINT>> is a complex type whose key is a scalar type INTEGER and value is a complex type ARRAY with element type BIGINT.
Array type contains its element type. Map type contains the key type and value type. Row type contains its field types along with their names.
Presto Types¶
Velox supports a number of Presto-specific logical types. The table below shows the supported Presto types.
Presto Type |
Physical Type |
---|---|
HYPERLOGLOG |
VARBINARY |
JSON |
VARCHAR |
TIMESTAMP WITH TIME ZONE |
ROW<BIGINT, SMALLINT> |
TIMESTAMP WITH TIME ZONE represents a time point in milliseconds precision
from UNIX epoch with timezone information. Its physical type contains one 64-bit
signed integer for milliseconds and another 16-bit signed integer for timezone ID.
Valid range of timezone ID is [1, 1680], its definition can be found in TimeZoneDatabase.cpp
.