-
Notifications
You must be signed in to change notification settings - Fork 542
[Docs] Website Redesign #3226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[Docs] Website Redesign #3226
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
0540e3d
initial proposal
polyzos 673cb76
add some improvements
polyzos 2756a65
improve light mode
polyzos d865d04
improve the diagram
polyzos f5a88d6
update landing page
polyzos 2617cd5
landing pages updates
polyzos 54b0be8
fix landings page diagram
polyzos 64fa351
finalize landing page
polyzos f5c6ee3
small updates to the landing page
polyzos 99bfea2
update main blog page
polyzos 6d84175
enable again the dark mode
polyzos 7932295
add some link syntax highlighting
polyzos 5a93ff2
address comments from michael and copilot
polyzos 66167af
update diagrams
polyzos 44ffc04
update visual backgrounds
polyzos 1ec2872
remove redundant edit
polyzos 7a7d1df
update from jark
wuchong 281e638
fix link color on downloads/talks/videos page
wuchong 1b0e134
address comments
polyzos 39660dd
fix navbar
polyzos 5ddd7a6
small fixes and kafka comparison section
polyzos 33b38fe
ensure text consistency
polyzos 21e3828
fix code color style and some display problems
wuchong 1afde50
refine sidebar menu styles for consistent hierarchy and active state
wuchong 0b32225
remove unnecessary inline keyboard hint from search button for cleane…
wuchong 9f1fb37
refactor navbar styles and color mode toggle behavior for improved la…
wuchong fd1a1c1
update delta join image
polyzos 0cdb2db
make more mobile friendly
polyzos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| --- | ||
| title: Introduction | ||
| sidebar_position: 1 | ||
| slug: / | ||
| --- | ||
|
|
||
| # What is Apache Fluss? | ||
|
|
||
| **One platform. Streams, tables, and the lakehouse.** Apache Fluss is an open-source streaming storage system for real-time analytics and AI, designed to serve as the real-time data layer of a Lakehouse architecture. | ||
|
|
||
| Fluss collapses the message broker, the online key-value store, the stream-processing state backend, and the lakehouse cold store into a single coherent foundation, so the same table can be read at sub-second freshness for analytics, looked up by primary key for features, and tiered into open formats like Apache Iceberg, Apache Paimon, or Lance for historical scans. | ||
|
|
||
|  | ||
|
|
||
| It integrates first-class with **Apache Flink** and **Apache Spark**, supports streaming reads and writes with sub-second latency, stores data in a columnar layout for projection and predicate pushdown, and offers two table types (append-only **Log Tables** and updatable **PrimaryKey Tables**) to cover both event-stream and database-style workloads. | ||
|
|
||
| **Fluss (German: river, pronounced `/flus/`)** carries streaming data continuously into lakes, like a river 🌊 | ||
|
|
||
| ## Use cases | ||
|
|
||
| Fluss targets workloads where the same data must be read at multiple freshness layers (stream, point-lookup, and lake) without being copied across systems. Six use cases drive most deployments today. | ||
|
|
||
| ### 1. Real-time feature stores for ML serving and training | ||
|
|
||
| The single largest use case. A Fluss **PrimaryKey Table** holds the live feature row in RocksDB on the leader for sub-millisecond key-value lookup at serving time, and the ordered changelog for training reads. Both views read the same underlying storage, so training/serving skew is structurally eliminated rather than monitored after the fact. Replaces the typical Redis-online-store plus Iceberg-offline-store pair with a single substrate. | ||
|
|
||
| ### 2. Real-time risk, fraud detection, and decision audit | ||
|
|
||
| PrimaryKey Tables emit a complete `+I / -U / +U / -D` changelog for every entity. When a real-time decision fires (decline a transaction, flag an account, deny a request), the exact feature values that drove it trace directly back to a specific changelog record, so the decision is fully reconstructible from the log itself, with no secondary audit pipeline. | ||
|
|
||
| ### 3. Real-time entity profiles and Customer 360 | ||
|
|
||
| The **Aggregation Merge Engine** combined with **Roaring Bitmaps** lets the leader maintain large entity sets such as *"users who clicked in the last ten minutes"* or *"accounts that crossed a usage threshold today"*, via at-write-time read-modify-write. Profile composition reduces to set algebra (`AND` / `OR` / `AND NOT`) over bitmap columns. Multiple producers can write disjoint columns of the same wide row independently via partial updates. | ||
|
|
||
| ### 4. AI agent memory and context engineering for LLM systems | ||
|
|
||
| Four distinct memory primitives live in the same substrate under a consistent schema surface: | ||
|
|
||
| - **Session memory**: conversation logs | ||
| - **Entity memory**: live key-value facts | ||
| - **Behavioral memory**: streaming features | ||
| - **Semantic memory**: vector store via Lance | ||
|
|
||
| An agent reads all four concurrently to assemble a grounded prompt, with no cross-system synchronization. | ||
|
|
||
| ### 5. Real-time operational analytics and OLAP hot-store replacement | ||
|
|
||
| The columnar Arrow log plus server-side compound pruning (partition pruning, predicate pushdown, column projection) lets analytical engines such as Flink batch, Spark, Trino, StarRocks, and DuckDB query the hot tier directly. The result is order-of-magnitude reductions in I/O, network, and deserialization cost. Replaces the typical "OLAP hot store" sitting alongside Kafka. | ||
|
|
||
| ### 6. Streaming ETL with externalized state | ||
|
|
||
| Stateful streaming ETL pipelines (joins, rolling aggregations, deduplication, reference-data enrichment, wide-row assembly across multiple producers) traditionally accumulate gigabytes of operator state inside Flink that must be checkpointed, recovered, and rebalanced on scale-out. With Fluss, that state moves into **PrimaryKey Tables** on the leader and the Flink job becomes stateless: dual-stream joins collapse into stateless index-key lookups via **delta joins**, rolling counts and velocity signals collapse into writes against the **Aggregation Merge Engine**, and multi-producer wide-row updates collapse into **partial-updates** against a shared row. | ||
|
|
||
| ## Where to go next? | ||
|
|
||
| - [QuickStart](quickstart/flink.md): Get started with Fluss in minutes. | ||
| - [Architecture](concepts/architecture.md): Learn about Fluss's architecture. | ||
| - [Table Design](table-design/overview.md): Explore Fluss's table types, partitions and buckets. | ||
| - [Lakehouse](streaming-lakehouse/overview.md): Integrate Fluss with your Lakehouse to bring low-latency data to your Lakehouse analytics. | ||
| - [Development](/community/dev/ide-setup): Set up your development environment and contribute to the community. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.