Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
0540e3d
initial proposal
polyzos Apr 28, 2026
673cb76
add some improvements
polyzos Apr 28, 2026
2756a65
improve light mode
polyzos Apr 29, 2026
d865d04
improve the diagram
polyzos Apr 29, 2026
f5a88d6
update landing page
polyzos May 7, 2026
2617cd5
landing pages updates
polyzos May 7, 2026
54b0be8
fix landings page diagram
polyzos May 7, 2026
64fa351
finalize landing page
polyzos May 7, 2026
f5c6ee3
small updates to the landing page
polyzos May 8, 2026
99bfea2
update main blog page
polyzos May 8, 2026
6d84175
enable again the dark mode
polyzos May 12, 2026
7932295
add some link syntax highlighting
polyzos May 12, 2026
5a93ff2
address comments from michael and copilot
polyzos May 12, 2026
66167af
update diagrams
polyzos May 12, 2026
44ffc04
update visual backgrounds
polyzos May 13, 2026
1ec2872
remove redundant edit
polyzos May 13, 2026
7a7d1df
update from jark
wuchong May 16, 2026
281e638
fix link color on downloads/talks/videos page
wuchong May 16, 2026
1b0e134
address comments
polyzos May 16, 2026
39660dd
fix navbar
polyzos May 16, 2026
5ddd7a6
small fixes and kafka comparison section
polyzos May 16, 2026
33b38fe
ensure text consistency
polyzos May 16, 2026
21e3828
fix code color style and some display problems
wuchong May 18, 2026
1afde50
refine sidebar menu styles for consistent hierarchy and active state
wuchong May 18, 2026
0b32225
remove unnecessary inline keyboard hint from search button for cleane…
wuchong May 18, 2026
9f1fb37
refactor navbar styles and color mode toggle behavior for improved la…
wuchong May 18, 2026
fd1a1c1
update delta join image
polyzos May 18, 2026
0cdb2db
make more mobile friendly
polyzos May 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ $ cd docker/fluss
docker/fluss $ docker buildx build --push --platform linux/arm64/v8,linux/amd64 --tag apache/fluss:${RELEASE_VERSION}-rc${RC_NUM} .
```

Then, run the following commands to build and push the flink a customized Apache Flink image that includes all necessary libraries and JARs for the Fluss Quickstart guide:
Then, run the following commands to build and push the flink, a customized Apache Flink image that includes all necessary libraries and JARs for the Fluss Quickstart guide:

```bash
docker/fluss $ cd ../quickstart-flink
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ And then you can use the staged maven artifacts as dependencies in the project a

For any user-facing feature included in a release, we aim to ensure it is functional, usable, and well-documented for the Fluss community.

To support this, release managers can create and assign cross-team testing issues that outline key scenarios to validate. These issues are open toand encouraged forall community members to pick up and help verify.
To support this, release managers can create and assign cross-team testing issues that outline key scenarios to validate. These issues are open to, and encouraged for, all community members to pick up and help verify.

A great way to get started is by walking through the official Quickstart Guide: https://fluss.apache.org/docs/quickstart/flink/ (please switch to the documentation version currently under release).

Expand Down
Binary file modified website/docs/assets/architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/docs/assets/data_organization.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed website/docs/assets/delta_join.jpg
Binary file not shown.
Binary file added website/docs/assets/delta_join.png
Comment thread
polyzos marked this conversation as resolved.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/docs/assets/deployment_overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/docs/assets/streamhouse.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/docs/assets/tiered-storage.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion website/docs/engine-flink/delta-joins.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Starting with **Apache Fluss 0.8**, streaming join jobs running on **Flink 2.1 o

Traditional streaming joins in Flink require maintaining both input sides entirely in state to match records across streams. Delta join, by contrast, uses a **index-key lookup mechanism** to transform the behavior of querying data from the state into querying data from the Fluss source table, thereby avoiding redundant storage of the same data in both the Fluss source table and the state. This drastically reduces state size and improves performance for many streaming analytics and enrichment workloads.

![](../assets/delta_join.jpg)
![](../assets/delta_join.png)

## Example: Delta Join in Flink 2.1

Expand Down
41 changes: 0 additions & 41 deletions website/docs/intro.md

This file was deleted.

60 changes: 60 additions & 0 deletions website/docs/intro.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: Introduction
sidebar_position: 1
slug: /
---

# What is Apache Fluss?

**One platform. Streams, tables, and the lakehouse.** Apache Fluss is an open-source streaming storage system for real-time analytics and AI, designed to serve as the real-time data layer of a Lakehouse architecture.

Fluss collapses the message broker, the online key-value store, the stream-processing state backend, and the lakehouse cold store into a single coherent foundation, so the same table can be read at sub-second freshness for analytics, looked up by primary key for features, and tiered into open formats like Apache Iceberg, Apache Paimon, or Lance for historical scans.

![arch](/img/fluss.png)

It integrates first-class with **Apache Flink** and **Apache Spark**, supports streaming reads and writes with sub-second latency, stores data in a columnar layout for projection and predicate pushdown, and offers two table types (append-only **Log Tables** and updatable **PrimaryKey Tables**) to cover both event-stream and database-style workloads.

**Fluss (German: river, pronounced `/flus/`)** carries streaming data continuously into lakes, like a river 🌊

## Use cases

Fluss targets workloads where the same data must be read at multiple freshness layers (stream, point-lookup, and lake) without being copied across systems. Six use cases drive most deployments today.

### 1. Real-time feature stores for ML serving and training

The single largest use case. A Fluss **PrimaryKey Table** holds the live feature row in RocksDB on the leader for sub-millisecond key-value lookup at serving time, and the ordered changelog for training reads. Both views read the same underlying storage, so training/serving skew is structurally eliminated rather than monitored after the fact. Replaces the typical Redis-online-store plus Iceberg-offline-store pair with a single substrate.

### 2. Real-time risk, fraud detection, and decision audit

PrimaryKey Tables emit a complete `+I / -U / +U / -D` changelog for every entity. When a real-time decision fires (decline a transaction, flag an account, deny a request), the exact feature values that drove it trace directly back to a specific changelog record, so the decision is fully reconstructible from the log itself, with no secondary audit pipeline.

### 3. Real-time entity profiles and Customer 360

The **Aggregation Merge Engine** combined with **Roaring Bitmaps** lets the leader maintain large entity sets such as *"users who clicked in the last ten minutes"* or *"accounts that crossed a usage threshold today"*, via at-write-time read-modify-write. Profile composition reduces to set algebra (`AND` / `OR` / `AND NOT`) over bitmap columns. Multiple producers can write disjoint columns of the same wide row independently via partial updates.

### 4. AI agent memory and context engineering for LLM systems

Four distinct memory primitives live in the same substrate under a consistent schema surface:

- **Session memory**: conversation logs
- **Entity memory**: live key-value facts
- **Behavioral memory**: streaming features
- **Semantic memory**: vector store via Lance

An agent reads all four concurrently to assemble a grounded prompt, with no cross-system synchronization.

### 5. Real-time operational analytics and OLAP hot-store replacement

The columnar Arrow log plus server-side compound pruning (partition pruning, predicate pushdown, column projection) lets analytical engines such as Flink batch, Spark, Trino, StarRocks, and DuckDB query the hot tier directly. The result is order-of-magnitude reductions in I/O, network, and deserialization cost. Replaces the typical "OLAP hot store" sitting alongside Kafka.

### 6. Streaming ETL with externalized state

Stateful streaming ETL pipelines (joins, rolling aggregations, deduplication, reference-data enrichment, wide-row assembly across multiple producers) traditionally accumulate gigabytes of operator state inside Flink that must be checkpointed, recovered, and rebalanced on scale-out. With Fluss, that state moves into **PrimaryKey Tables** on the leader and the Flink job becomes stateless: dual-stream joins collapse into stateless index-key lookups via **delta joins**, rolling counts and velocity signals collapse into writes against the **Aggregation Merge Engine**, and multi-producer wide-row updates collapse into **partial-updates** against a shared row.

## Where to go next?

- [QuickStart](quickstart/flink.md): Get started with Fluss in minutes.
- [Architecture](concepts/architecture.md): Learn about Fluss's architecture.
- [Table Design](table-design/overview.md): Explore Fluss's table types, partitions and buckets.
- [Lakehouse](streaming-lakehouse/overview.md): Integrate Fluss with your Lakehouse to bring low-latency data to your Lakehouse analytics.
- [Development](/community/dev/ide-setup): Set up your development environment and contribute to the community.
2 changes: 1 addition & 1 deletion website/docs/maintenance/tiered-storage/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ in the well-known open data lake format for better analytics performance. Curren

The overall tiered storage architecture is shown in the following diagram:

<img width="600px" src={require('../../assets/tiered-storage.png').default} />
<img width="900px" src={require('../../assets/tiered-storage.png').default} />
142 changes: 117 additions & 25 deletions website/docusaurus.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,81 @@
* limitations under the License.
*/

import {themes as prismThemes} from 'prism-react-renderer';
import type {Config} from '@docusaurus/types';
import type * as Preset from '@docusaurus/preset-classic';
import lightTheme from './src/utils/prismLight';
import darkTheme from './src/utils/prismDark';
import versionReplace from './src/plugins/remark-version-replace/index';
import { loadVersionData } from './src/utils/versionData';
const { versionsMap, latestVersion } = loadVersionData();

const config: Config = {
title: 'Apache Fluss™ (Incubating)',
tagline: 'Streaming Storage for Real-Time Analytics & AI',
tagline: 'The streaming storage layer for real-time analytics and the lakehouse',
favicon: 'img/logo/fluss_favicon.svg',

headTags: [
{
tagName: 'meta',
attributes: {
name: 'description',
content:
'Apache Fluss is an open-source columnar streaming storage system. Sub-second freshness, primary-key tables, first-class Apache Flink integration, and native tiering to Apache Iceberg and Apache Paimon.',
},
},
{
tagName: 'meta',
attributes: {
property: 'og:title',
content: 'Apache Fluss · Streaming Storage for the Real-Time Lakehouse',
},
},
{
tagName: 'meta',
attributes: {
property: 'og:description',
content:
'Open-source columnar streaming storage with sub-second freshness, primary-key tables, Flink integration, and native tiering to Iceberg and Paimon.',
},
},
{
tagName: 'meta',
attributes: {
property: 'og:type',
content: 'website',
},
},
{
tagName: 'meta',
attributes: {
name: 'twitter:card',
content: 'summary_large_image',
},
},
{
tagName: 'meta',
attributes: {
name: 'twitter:title',
content: 'Apache Fluss · Streaming Storage for the Real-Time Lakehouse',
},
},
{
tagName: 'meta',
attributes: {
name: 'twitter:description',
content:
'Open-source columnar streaming storage with sub-second freshness, primary-key tables, Flink integration, and native tiering to Iceberg and Paimon.',
},
},
{
tagName: 'meta',
attributes: {
name: 'theme-color',
content: '#102856',
},
},
],

// Set the production url of your site here
url: 'https://fluss.apache.org/',
// Set the /<baseUrl>/ pathname under which your site is served
Expand Down Expand Up @@ -67,8 +129,6 @@ const config: Config = {
{
docs: {
sidebarPath: './sidebars.ts',
editUrl: ({docPath}) =>
`https://github.com/apache/fluss/edit/main/website/docs/${docPath}`,
remarkPlugins: [versionReplace],
lastVersion: latestVersion,
versions: versionsMap
Expand Down Expand Up @@ -99,10 +159,9 @@ const config: Config = {
path: 'community',
routeBasePath: 'community',
sidebarPath: './sidebarsCommunity.js',
editUrl: ({docPath}) => {
return `https://github.com/apache/fluss/edit/main/website/community/${docPath}`;
},
// ... other options
// editUrl intentionally omitted so the "Edit this page" link does
// not appear at the bottom of community pages (mirrors the docs and
// blog presets, which also leave editUrl unset).
},
],
[
Expand All @@ -116,16 +175,16 @@ const config: Config = {
[
'@docusaurus/plugin-pwa',
{
debug: true,
debug: false,
offlineModeActivationStrategies: [
'appInstalled',
'standalone',
'queryString',
],
pwaHead: [
{ tagName: 'link', rel: 'icon', href: '/img/logo.svg' },
{ tagName: 'link', rel: 'icon', href: '/img/logo/svg/colored_logo.svg' },
{ tagName: 'link', rel: 'manifest', href: '/manifest.json' },
{ tagName: 'meta', name: 'theme-color', content: '#0071e3' },
{ tagName: 'meta', name: 'theme-color', content: '#102856' },
],
},
],
Expand Down Expand Up @@ -160,13 +219,15 @@ const config: Config = {
image: 'img/logo/png/colored_logo.png',
colorMode: {
defaultMode: 'light',
disableSwitch: true,
disableSwitch: false,
respectPrefersColorScheme: false,
},
navbar: {
title: '',
logo: {
alt: 'Fluss',
src: 'img/logo/svg/colored_logo.svg',
src: 'img/logo/svg/white_color_logo.svg',
srcDark: 'img/logo/svg/white_color_logo.svg',
},
items: [
{
Expand Down Expand Up @@ -194,17 +255,6 @@ const config: Config = {
{to: '/community/welcome', label: 'Community', position: 'left'},
{to: '/roadmap', label: 'Roadmap', position: 'left'},
{to: '/downloads', label: 'Downloads', position: 'left'},
{
label: 'ASF', position: 'right', items: [
{to: 'https://www.apache.org/', label: 'Foundation'},
{to: 'https://www.apache.org/licenses/', label: 'License'},
{to: 'https://events.apache.org', label: 'Events'},
{to: 'https://www.apache.org/foundation/sponsorship.html', label: 'Donate'},
{to: 'https://www.apache.org/foundation/thanks.html', label: 'Sponsors'},
{to: 'https://www.apache.org/security/', label: 'Security'},
{to: 'https://privacy.apache.org/policies/privacy-policy-public.html', label: 'Privacy'}
]
},
{
type: 'docsVersionDropdown',
position: 'right',
Expand All @@ -220,6 +270,48 @@ const config: Config = {
},
footer: {
style: 'dark',
links: [
{
title: 'Product',
items: [
{label: 'Documentation', to: '/docs/quickstart/flink'},
{label: 'Quickstart', to: '/docs/quickstart/flink'},
{label: 'Roadmap', to: '/roadmap'},
{label: 'Downloads', to: '/downloads'},
{label: 'Blog', to: '/blog'},
],
},
{
title: 'Community',
items: [
{label: 'GitHub', href: 'https://github.com/apache/fluss'},
{label: 'Slack', href: 'https://join.slack.com/t/apache-fluss/shared_invite/zt-33wlna581-QAooAiCmnYboJS8D_JUcYw'},
{label: 'Welcome', to: '/community/welcome'},
{label: 'Contribute', to: '/community/welcome'},
],
},
{
title: 'Resources',
items: [
{label: 'Talks', to: '/learn/talks'},
{label: 'Videos', to: '/learn/videos'},
{label: 'Issues', href: 'https://github.com/apache/fluss/issues'},
{label: 'Releases', href: 'https://github.com/apache/fluss/releases'},
],
},
{
title: 'Apache',
items: [
{label: 'Foundation', href: 'https://www.apache.org/'},
{label: 'License', href: 'https://www.apache.org/licenses/'},
{label: 'Events', href: 'https://events.apache.org'},
{label: 'Donate', href: 'https://www.apache.org/foundation/sponsorship.html'},
{label: 'Sponsors', href: 'https://www.apache.org/foundation/thanks.html'},
{label: 'Security', href: 'https://www.apache.org/security/'},
{label: 'Privacy', href: 'https://privacy.apache.org/policies/privacy-policy-public.html'},
],
},
],
logo: {
width: 200,
src: "/img/apache-incubator.svg",
Expand All @@ -232,7 +324,7 @@ const config: Config = {
},
prism: {
theme: lightTheme,
darkTheme: prismThemes.dracula,
darkTheme: darkTheme,
additionalLanguages: ['java', 'bash', 'scala']
},
algolia: {
Expand Down
Loading