From d003d8dd8b19436c98f818409b2b3fce25177111 Mon Sep 17 00:00:00 2001 From: lenkis Date: Mon, 18 May 2026 19:24:36 +0300 Subject: [PATCH 1/9] Add a draft of 3.7.0 what's new --- doc/release/3.7.0.rst | 104 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 doc/release/3.7.0.rst diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst new file mode 100644 index 0000000000..8dc487937d --- /dev/null +++ b/doc/release/3.7.0.rst @@ -0,0 +1,104 @@ +Tarantool 3.7 +============= + +Release date: 2026-04-12 + +Releases on GitHub: :tarantool-release:`3.6.0` + +The 3.7 release of Tarantool adds the following main product features and improvements +for the Community and Enterprise editions: + +* **Community Edition (CE)** + + * Multiple minor improvements and bug fixes. + +* **Enterprise Edition (EE)** + + * Failover coordinator: safe SSL configuration. + * MemCS: dictionary encoding and `lz4` compression. + +.. _3-7-failover-ssl-params: + +[EE] Failover coordinator: safe SSL cofiguration +------------------------------------------------ + +Tarantool failover coordinator now safely supports TLS-encrypted and mTLS-authenticated connections. +SSL certificates and Certificate Authority can be configured via a dedicated failover-specific +configuration section. + +Previuosly, the coordinator relied on SSL parameters taken from instance URIs. +In case of SSL + CA (mTLS), this approach might require the coordinator host to have access +to private keys/certificates of the instances it connected to. + +Now the `failover.iproto.ssl` configuration section is a source of SSL client parameters +for outgoing `net.box` connections initiated by the coordinator. The parameters are injected +into URI parameters passed to `net.box.connect()` only for URIs with `params.transport: ssl`, +and do not affect non-SSL connections. + +Example: + +.. code-block:: yaml + + failover: + mode: stateful + iproto: + listen: + uri: '127.0.0.1:3302' + params: + transport: ssl + ssl: + ca_file: './certs/rootCA.pem' + ssl_cert: './certs/coordinator.crt' + ssl_key: './certs/coordinator.key' + # Optional: + # ssl_ciphers: 'ECDHE-RSA-AES256-GCM-SHA384:...' + # ssl_password: 'secret' + # ssl_password_file: './certs/key_passwords.txt' + +.. _3-7-memcs-memory-improvements: + +[EE] MemCS: dictionary encoding and lz4 compression +--------------------------------------------------- + +This release brings two major memory utilization improvements in the MemCS engine: +* Arrow dictionary-encoded layout is now supported for string columns. +* It is now possible to compress column data with `lz4`. + +When combined, these features result in 8-13x reduction in occupied space compared +to uncompressed data. + +Dictionary encoding is a data representation technique to store values by integers +referencing a dictionary (hash-table) consisting of unique values. This is effective +when a column contains data with many repeated values. +??? why did we choose Arrow? +??? Select size of the ID. E.g. can we limit the number of unique values to 256 and use 1-byte id? +??? Is it possible to return only a pointer to the full dictionary of a column, without creating a subset of a dictionary for each record batch? +??? Block size. Min/max block size is configured per-index per-column via space:create_index() - what's the syntax? defaults? + Block size is set in the number of elements, not bytes, which is natural for Arrow and will allow to unite all columns in a common tree. +??? what other implementation details from RFC can we disclose? (allocation & readview, multithreading, memcs_column.[ch], bps_vec.h, balancing, last block optimization) +??? what new params do we have in config? + +The `lz4` codec provides is a universal algorythm with optimal compression speed and rate +and configurable acceleration level within 100 to 10000. +??? what new params do we have in config? + +Compression is configured per column using the 'compression' parameter. +You can enable/disable compression, specify the algorythm and its parameters. + +Examples: + +- compression: + +.. code-block:: lua + + box.schema.create_space('test', { + engine = 'memcs', field_count = 3, format = { + {name = 'c1', type = 'uint64'}, + {name = 'c2', type = 'uint64', compression = 'lz4'}, + {name = 'c3', type = 'string', compression = {method?codec?algo?algorithm?nothing? = 'lz4', acceleration = 1000}}, + } + }) + +- dictionary encoding: + +??? From 463b2a1fc9c6d5698f8250ee0a8f427c297db779 Mon Sep 17 00:00:00 2001 From: mar Date: Wed, 20 May 2026 21:58:43 +0300 Subject: [PATCH 2/9] doc: write release note for tarantool 3.7 (memcs: dictionary encoding and lz4 compression --- doc/release/3.7.0.rst | 105 +++++++++++++++++++++++++++++------------- doc/release/index.rst | 7 +++ 2 files changed, 79 insertions(+), 33 deletions(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 8dc487937d..7113a80c17 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -19,14 +19,14 @@ for the Community and Enterprise editions: .. _3-7-failover-ssl-params: -[EE] Failover coordinator: safe SSL cofiguration ------------------------------------------------- +[EE] Failover coordinator: safe SSL configuration +------------------------------------------------- Tarantool failover coordinator now safely supports TLS-encrypted and mTLS-authenticated connections. SSL certificates and Certificate Authority can be configured via a dedicated failover-specific configuration section. -Previuosly, the coordinator relied on SSL parameters taken from instance URIs. +Previously, the coordinator relied on SSL parameters taken from instance URIs. In case of SSL + CA (mTLS), this approach might require the coordinator host to have access to private keys/certificates of the instances it connected to. @@ -61,44 +61,83 @@ Example: --------------------------------------------------- This release brings two major memory utilization improvements in the MemCS engine: -* Arrow dictionary-encoded layout is now supported for string columns. -* It is now possible to compress column data with `lz4`. -When combined, these features result in 8-13x reduction in occupied space compared -to uncompressed data. +* Dictionary encoding for string columns (Arrow dictionary-encoded layout). +* Column-level compression with the `lz4` codec. -Dictionary encoding is a data representation technique to store values by integers -referencing a dictionary (hash-table) consisting of unique values. This is effective -when a column contains data with many repeated values. -??? why did we choose Arrow? -??? Select size of the ID. E.g. can we limit the number of unique values to 256 and use 1-byte id? -??? Is it possible to return only a pointer to the full dictionary of a column, without creating a subset of a dictionary for each record batch? -??? Block size. Min/max block size is configured per-index per-column via space:create_index() - what's the syntax? defaults? - Block size is set in the number of elements, not bytes, which is natural for Arrow and will allow to unite all columns in a common tree. -??? what other implementation details from RFC can we disclose? (allocation & readview, multithreading, memcs_column.[ch], bps_vec.h, balancing, last block optimization) -??? what new params do we have in config? +In typical workloads, using these features together provides an **8–13x reduction in memory footprint** compared to uncompressed storage. -The `lz4` codec provides is a universal algorythm with optimal compression speed and rate -and configurable acceleration level within 100 to 10000. -??? what new params do we have in config? +.. _3-7-memcs-memory-improvements_dict_encoding: -Compression is configured per column using the 'compression' parameter. -You can enable/disable compression, specify the algorythm and its parameters. +Dictionary encoding for string columns +====================================== -Examples: +MemCS can store string values in a **dictionary-encoded** form: unique strings are kept in a shared dictionary, while rows store integer IDs referencing dictionary entries. +This is most effective for columns with many repeated values (for example, statuses, categories, event types). -- compression: +Dictionary encoding is enabled via the index `layout` option: .. code-block:: lua - box.schema.create_space('test', { - engine = 'memcs', field_count = 3, format = { - {name = 'c1', type = 'uint64'}, - {name = 'c2', type = 'uint64', compression = 'lz4'}, - {name = 'c3', type = 'string', compression = {method?codec?algo?algorithm?nothing? = 'lz4', acceleration = 1000}}, - } - }) + local s = box.schema.create_space('test', { + engine = 'memcs', format = format, field_count = field_count, + }) + local pk = s:create_index('pk', {layout = 'dict'}) -- dictionary encoding: -??? +Limitations and behavior +~~~~~~~~~~~~~~~~~~~~~~~~ + +* Only **non-key string columns** can use dictionary encoding. Other columns silently ignore `layout = 'dict'`. +* Dictionary IDs are stored as `uint16` (2 bytes per row). +* The maximum number of unique values per dictionary-encoded column is `UINT16_MAX` (65536). When the dictionary is full, inserting a new unique value fails with an error. +* Approximate memory usage for such a column is `2 * space_size + dict_size bytes`. +* Dictionary memory is included in `space:bsize()` statistics. +* The implementation follows **Apache Arrow dictionary-encoded layout**, since Arrow is the standard interchange format for MemCS. + +When reading via ArrowStream, MemCS always returns dictionary-encoded columns in Arrow dictionary-encoded layout: + +* the dictionary is returned as **string-view** +* indices are `uint16` + +Additional notes: + +* If no new unique values are inserted, all record batches share the same dictionary. +* Dictionaries are returned by reference (no copying), so ArrowArray export has low overhead. +* Dictionaries only grow (values are not removed), so after a dictionary update it remains compatible with batches produced with the previous dictionary. + +.. _3-7-memcs-memory-improvements_lz4_column_compression: + +LZ4 column compression +====================== + +MemCS now supports column-level compression with `lz4 `_. +Compression is applied to ranges of consecutive values, stored as compressed blocks. + +Compression is configured per column in the space format using the `compression` attribute and can be enabled only at space creation time: + +.. code-block:: lua + + local format = { + {'c1', 'uint64'}, + {'c2', 'string', is_nullable = true, compression = 'lz4'}, + {'c3', 'int32', compression = {type = 'lz4', acceleration = 1000}}, + } + box.schema.create_space('test', { + engine = 'memcs', field_count = #format, format = format, + }) + + +Compression settings: + +* Algorithm: `lz4` +* `acceleration` – LZ4-specific configuration parameter: + * allowed range: 1…65537 + * recommended: 10…1000 + * higher values improve compression and decompression speed, but reduce the compression ratio + +Limitations +~~~~~~~~~~~ + +* Only non-indexed columns can be compressed. +* A column of any type supports compression, however "external" string values (strings longer than 12 characters) are currently not subject to compression. diff --git a/doc/release/index.rst b/doc/release/index.rst index 633b8b61ad..22a5798137 100644 --- a/doc/release/index.rst +++ b/doc/release/index.rst @@ -63,6 +63,12 @@ For information about earlier versions, see :doc:`eos_versions`. - End of support - Versions + * - :doc:`3.7` + - ** 2026** + - **Not planned yet** + - **Not planned yet** + - | :tarantool-release:`3.7.0` + * - :doc:`3.6 ` - **December 12, 2025** - **Not planned yet** @@ -99,6 +105,7 @@ For information about earlier versions, see :doc:`eos_versions`. :maxdepth: 1 policy + 3.7.0 3.6.0 3.5.0 3.4.0 From 0b0b7694d5ce955327f0670b0b6580b93f57a8d8 Mon Sep 17 00:00:00 2001 From: mar Date: Wed, 20 May 2026 22:22:07 +0300 Subject: [PATCH 3/9] doc: add some corrections --- doc/release/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release/index.rst b/doc/release/index.rst index 22a5798137..7d6fc378d9 100644 --- a/doc/release/index.rst +++ b/doc/release/index.rst @@ -64,7 +64,7 @@ For information about earlier versions, see :doc:`eos_versions`. - Versions * - :doc:`3.7` - - ** 2026** + - **April 22, 2026** - **Not planned yet** - **Not planned yet** - | :tarantool-release:`3.7.0` From 012373c1088a26b2749fa1c8aa610db8728ae87c Mon Sep 17 00:00:00 2001 From: mar Date: Thu, 21 May 2026 11:57:25 +0300 Subject: [PATCH 4/9] doc: add some corrections --- doc/release/3.7.0.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 7113a80c17..30c3d957fc 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -70,7 +70,7 @@ In typical workloads, using these features together provides an **8–13x reduct .. _3-7-memcs-memory-improvements_dict_encoding: Dictionary encoding for string columns -====================================== +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ MemCS can store string values in a **dictionary-encoded** form: unique strings are kept in a shared dictionary, while rows store integer IDs referencing dictionary entries. This is most effective for columns with many repeated values (for example, statuses, categories, event types). @@ -86,7 +86,7 @@ Dictionary encoding is enabled via the index `layout` option: Limitations and behavior -~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ * Only **non-key string columns** can use dictionary encoding. Other columns silently ignore `layout = 'dict'`. * Dictionary IDs are stored as `uint16` (2 bytes per row). @@ -109,7 +109,7 @@ Additional notes: .. _3-7-memcs-memory-improvements_lz4_column_compression: LZ4 column compression -====================== +~~~~~~~~~~~~~~~~~~~~~~ MemCS now supports column-level compression with `lz4 `_. Compression is applied to ranges of consecutive values, stored as compressed blocks. @@ -137,7 +137,7 @@ Compression settings: * higher values improve compression and decompression speed, but reduce the compression ratio Limitations -~~~~~~~~~~~ +^^^^^^^^^^^ * Only non-indexed columns can be compressed. * A column of any type supports compression, however "external" string values (strings longer than 12 characters) are currently not subject to compression. From b12e9a0aaf7a9946cef50d985ad8d20e81bfc6ad Mon Sep 17 00:00:00 2001 From: mar Date: Thu, 21 May 2026 12:06:06 +0300 Subject: [PATCH 5/9] doc: add some corrections --- doc/release/3.7.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 30c3d957fc..59f2abcfde 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -3,7 +3,7 @@ Tarantool 3.7 Release date: 2026-04-12 -Releases on GitHub: :tarantool-release:`3.6.0` +Releases on GitHub: :tarantool-release:`3.7.0` The 3.7 release of Tarantool adds the following main product features and improvements for the Community and Enterprise editions: From e720a3eb64ca039859d9ab774449b10801370734 Mon Sep 17 00:00:00 2001 From: mar Date: Tue, 26 May 2026 11:47:16 +0300 Subject: [PATCH 6/9] doc: add release calendar image --- doc/release/3.7.0.rst | 2 +- doc/release/_images/releases_calendar.svg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 59f2abcfde..5724a42ea9 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -75,7 +75,7 @@ Dictionary encoding for string columns MemCS can store string values in a **dictionary-encoded** form: unique strings are kept in a shared dictionary, while rows store integer IDs referencing dictionary entries. This is most effective for columns with many repeated values (for example, statuses, categories, event types). -Dictionary encoding is enabled via the index `layout` option: +Dictionary encoding is enabled via the :ref:`layout ` option: .. code-block:: lua diff --git a/doc/release/_images/releases_calendar.svg b/doc/release/_images/releases_calendar.svg index 4192490bd1..2a7157710a 100644 --- a/doc/release/_images/releases_calendar.svg +++ b/doc/release/_images/releases_calendar.svg @@ -1,4 +1,4 @@ -
2024
2024
2025
2025
2026
2026
2027
2027
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
3.0
3.0
3.0.1
3.0.1
Release
Release
Updates and fixes
Updates and fixes
Support
Support
x.y
x.y
EOL series
EOL series
x.y
x.y
Current series
Current series
3.1
3.1
3.1.0
3.1.0
3.0.2
3.0.2
3.1.1
3.1.1
3.2
3.2
3.2.0
3.2.0
Jan
Jan
3.1.2
3.1.2
3.2.1
3.2.1
3.3
3.3
3.3.0
3.3.0
3.3.1
3.3.1
3.4
3.4
3.4.0
3.4.0
Apr
Apr
Jul
Jul
3.0.0
3.0.0
3.5
3.5
3.5.0
3.5.0
3.6
3.6
3.6.0
3.6.0Text is not SVG - cannot display \ No newline at end of file +
2024
2024
2025
2025
2026
2026
2027
2027
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
3.0
3.0
3.0.1
3.0.1
Release
Release
Updates and fixes
Updates and fixes
Support
Support
x.y
x.y
EOL series
EOL series
x.y
x.y
Current series
Current series
3.1
3.1
3.1.0
3.1.0
3.0.2
3.0.2
3.1.1
3.1.1
3.2
3.2
3.2.0
3.2.0
Jan
Jan
3.1.2
3.1.2
3.2.1
3.2.1
3.3
3.3
3.3.0
3.3.0
3.3.1
3.3.1
3.4
3.4
3.4.0
3.4.0
Apr
Apr
Jul
Jul
3.0.0
3.0.0
3.5
3.5
3.5.0
3.5.0
3.6
3.6
3.6.0
3.6.0
3.7
3.7
3.7.0
3.7.0
Oct
Oct
2028
2028
Jan
Jan
Apr
AprText is not SVG - cannot display \ No newline at end of file From 2042bef14ccfa8a18eac2ea25f95857fcd00e988 Mon Sep 17 00:00:00 2001 From: mar Date: Wed, 27 May 2026 15:42:33 +0300 Subject: [PATCH 7/9] doc: add notes --- doc/release/3.7.0.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 5724a42ea9..422d6263c9 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -85,6 +85,11 @@ Dictionary encoding is enabled via the :ref:`layout ` option: local pk = s:create_index('pk', {layout = 'dict'}) + .. NOTE:: + + Currently the `layout` option can be specified only during the space creation. It's not possible to change it for the existing space. + + Limitations and behavior ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -128,6 +133,11 @@ Compression is configured per column in the space format using the `compression` }) + .. NOTE:: + + Currently the `compression` attribute can be specified only during the space creation. It's not possible to change it for the existing space. + + Compression settings: * Algorithm: `lz4` From a4624f3fda3fdcd97a1a84deb1f640cb1aad415a Mon Sep 17 00:00:00 2001 From: mar Date: Wed, 27 May 2026 16:39:50 +0300 Subject: [PATCH 8/9] doc: add notes --- doc/release/3.7.0.rst | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 422d6263c9..5406f36de1 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -84,12 +84,6 @@ Dictionary encoding is enabled via the :ref:`layout ` option: }) local pk = s:create_index('pk', {layout = 'dict'}) - - .. NOTE:: - - Currently the `layout` option can be specified only during the space creation. It's not possible to change it for the existing space. - - Limitations and behavior ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -100,6 +94,10 @@ Limitations and behavior * Dictionary memory is included in `space:bsize()` statistics. * The implementation follows **Apache Arrow dictionary-encoded layout**, since Arrow is the standard interchange format for MemCS. + .. NOTE:: + + Currently the `layout` option can be specified only during the space creation. It's not possible to change it for the existing space. + When reading via ArrowStream, MemCS always returns dictionary-encoded columns in Arrow dictionary-encoded layout: * the dictionary is returned as **string-view** @@ -133,11 +131,6 @@ Compression is configured per column in the space format using the `compression` }) - .. NOTE:: - - Currently the `compression` attribute can be specified only during the space creation. It's not possible to change it for the existing space. - - Compression settings: * Algorithm: `lz4` @@ -151,3 +144,7 @@ Limitations * Only non-indexed columns can be compressed. * A column of any type supports compression, however "external" string values (strings longer than 12 characters) are currently not subject to compression. + + .. NOTE:: + + Currently the `compression` attribute can be specified only during the space creation. It's not possible to change it for the existing space. From 9e305300ff7cfbddc541a1c2f6f9577128254e67 Mon Sep 17 00:00:00 2001 From: mar Date: Wed, 27 May 2026 17:24:25 +0300 Subject: [PATCH 9/9] doc: add notes --- doc/release/3.7.0.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/release/3.7.0.rst b/doc/release/3.7.0.rst index 5406f36de1..b1c064fde2 100644 --- a/doc/release/3.7.0.rst +++ b/doc/release/3.7.0.rst @@ -94,9 +94,9 @@ Limitations and behavior * Dictionary memory is included in `space:bsize()` statistics. * The implementation follows **Apache Arrow dictionary-encoded layout**, since Arrow is the standard interchange format for MemCS. - .. NOTE:: + .. IMPORTANT:: - Currently the `layout` option can be specified only during the space creation. It's not possible to change it for the existing space. + Currently the `layout` option can be specified only during the space creation. It's not possible to change it for the existing space. When reading via ArrowStream, MemCS always returns dictionary-encoded columns in Arrow dictionary-encoded layout: @@ -145,6 +145,6 @@ Limitations * Only non-indexed columns can be compressed. * A column of any type supports compression, however "external" string values (strings longer than 12 characters) are currently not subject to compression. - .. NOTE:: + .. IMPORTANT:: - Currently the `compression` attribute can be specified only during the space creation. It's not possible to change it for the existing space. + Currently the `compression` attribute can be specified only during the space creation. It's not possible to change it for the existing space.