docs: update README and ARCHITECTURE to reflect current codebase (#97)

Author: zhexuany

ARCHITECTURE.md

@@ -27,10 +27,11 @@ Roboflow is a distributed data transformation pipeline that converts robotics ba
 |-------|---------|-----------|
 | `roboflow-core` | Foundation types, error handling, registry | `RoboflowError`, `CodecValue`, `TypeRegistry` |
 | `roboflow-storage` | Storage abstraction layer | `Storage`, `LocalStorage`, `S3Storage`, `StorageFactory` |
-| `roboflow-dataset` | Dataset format writers | `LerobotWriter`, `DatasetWriter`, `ImageData` |
-| `roboflow-distributed` | Distributed coordination via TiKV | `TiKVClient`, `BatchController`, `Worker`, `Catalog` |
-| `roboflow-sources` | Data source implementations | `BagSource`, `McapSource`, `RrdSource` |
-| `roboflow-sinks` | Data sink implementations | `LerobotSink`, `ZarrSink`, `DatasetFrame` |
+| `roboflow-executor` | Stage-based task executor | `StageExecutor`, `Pipeline`, `ExecutionPolicy`, `SlotPool` |
+| `roboflow-media` | Image and video encoding/decoding | `ImageData`, `VideoEncoder`, `ConcurrentVideoEncoder` |
+| `roboflow-dataset` | Dataset format writers and sources | `LerobotWriter`, `DatasetWriter`, `Source`, `BagSource`, `McapSource` |
+| `roboflow-pipeline` | Pipeline execution and stages | `DatasetPipelineExecutor`, `DiscoverStage`, `ConvertStage`, `MergeStage` |
+| `roboflow-distributed` | Distributed coordination via TiKV | `TiKVClient`, `BatchController`, `Worker`, `Scanner`, `Finalizer` |
 
 ## Core Abstractions
 
@@ -55,21 +56,33 @@ trait SeekableStorage: Storage {
 - **S3**: AWS S3-compatible storage
 - **OSS**: Alibaba Cloud Object Storage
 
-### Pipeline Stages
+### Source/Sink Pattern
 
 ```rust
 trait Source: Send + Sync {
     async fn initialize(&mut self, config: &SourceConfig) -> SourceResult<SourceMetadata>;
     async fn read_batch(&mut self, size: usize) -> SourceResult<Option<Vec<TimestampedMessage>>>;
     async fn finalize(&mut self) -> SourceResult<SourceStats>;
 }
+```
+
+**Supported sources:**
+- **MCAP**: Streaming and memory-mapped reads
+- **ROS1 Bag**: Legacy bag format support
+- **RRD**: Rerun data format
+
+### Pipeline Stages
 
-trait Sink: Send + Sync {
-    async fn initialize(&mut self, config: &SinkConfig) -> SinkResult<()>;
-    async fn write_frame(&mut self, frame: DatasetFrame) -> SinkResult<()>;
-    async fn flush(&mut self) -> SinkResult<()>;
-    async fn finalize(&mut self) -> SinkResult<SinkStats>;
-    fn supports_checkpointing(&self) -> bool;
+```rust
+// Stage-based execution inspired by Spark
+pub struct DiscoverStage;
+pub struct ConvertStage;
+pub struct MergeStage;
+
+// Pipeline executor
+pub struct DatasetPipelineExecutor {
+    writer: Box<dyn DatasetWriter>,
+    config: DatasetPipelineConfig,
 }
 ```
 
@@ -117,18 +130,19 @@ The distributed system uses a Kubernetes-inspired design with TiKV as the contro
 | kubelet heartbeat | HeartbeatManager | Worker liveness |
 | Finalizers | Finalizer controller | Cleanup handling |
 | Job/CronJob | BatchSpec, WorkUnit | Work scheduling |
+| Scheduler | Scanner | File discovery and job creation |
 
 ### Batch State Machine
 
 ```
 ┌──────────┐    ┌─────────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
 │ Pending  │───▶│ Discovering │───▶│ Running  │───▶│ Merging  │───▶│ Complete │
 └──────────┘    └─────────────┘    └──────────┘    └──────────┘    └──────────┘
-                                      │
-                                      ▼
-                                   ┌──────────┐
-                                   │ Failed   │
-                                   └──────────┘
+                                       │
+                                       ▼
+                                    ┌──────────┐
+                                    │ Failed   │
+                                    └──────────┘
 ```
 
 ### TiKV Key Structure
@@ -142,6 +156,27 @@ roboflow/worker/{pod_id}/lock      → LockRecord
 roboflow/worker/{pod_id}/checkpoint→ CheckpointState
 ```
 
+## CLI Commands
+
+The unified `roboflow` binary provides all operations:
+
+```bash
+# Run unified service (default: all roles)
+roboflow run
+
+# Run specific roles
+roboflow run --role worker
+roboflow run --role finalizer
+
+# Job management
+roboflow submit s3://bucket/file.bag --output s3://bucket/out/
+roboflow jobs list
+roboflow batch list
+
+# Health check
+roboflow health
+```
+
 ## Dataset Writing
 
 ### LeRobot Format
@@ -151,23 +186,30 @@ struct LerobotConfig {
     pub dataset: DatasetConfig,
     pub mappings: Vec<Mapping>,
     pub video: VideoConfig,
-    pub flushing: FlushingConfig,  // Incremental flushing
+    pub flushing: FlushingConfig,
+    pub streaming: StreamingConfig,
 }
 
-struct FlushingConfig {
-    pub max_frames_per_chunk: usize,  // Default: 1000
-    pub max_memory_bytes: usize,      // Default: 2GB
-    pub incremental_video_encoding: bool,
+struct StreamingConfig {
+    pub finalize_metadata_in_coordinator: bool,
 }
 ```
 
-### Incremental Flushing
+### Video Encoding
 
-To prevent OOM on long recordings, the writer processes data in chunks:
+```rust
+// Concurrent video encoder for parallel chunk encoding
+pub struct ConcurrentVideoEncoder {
+    config: ConcurrentEncoderConfig,
+}
 
-1. **Frame-based**: Flush after N frames (configurable, default 1000)
-2. **Memory-based**: Flush when memory exceeds threshold (default 2GB)
-3. **Output structure**: `data/chunk-000/`, `data/chunk-001/`, etc.
+pub struct ConcurrentEncoderConfig {
+    pub storage: Arc<dyn Storage>,
+    pub key_prefix: String,
+    pub codec: VideoCodec,
+    pub crf: u8,
+}
+```
 
 ### Upload Coordinator
 
@@ -176,7 +218,6 @@ struct EpisodeUploadCoordinator {
     pub storage: Arc<dyn Storage>,
     pub config: UploadConfig,
     pub progress: Option<UploadProgress>,
-    // Worker pool for parallel uploads
 }
 
 struct UploadConfig {
@@ -213,7 +254,7 @@ let data = arena.alloc_vec::<u8>(size);
 
 ```toml
 [source]
-type = "mcap"  # or "bag", "rrd", "hdf5"
+type = "mcap"  # or "bag", "rrd"
 path = "s3://bucket/path/to/data.mcap"
 
 # Optional: topic filtering
@@ -325,17 +366,15 @@ enum CircuitState {
 
 | Flag | Purpose |
 |------|---------|
-| `distributed` | TiKV distributed coordination (always enabled) |
-| `dataset-hdf5` | HDF5 dataset format support |
-| `dataset-parquet` | Parquet dataset format support |
-| `cloud-storage` | S3/OSS cloud storage support |
-| `gpu` | GPU compression (Linux only) |
 | `jemalloc` | jemalloc allocator (Linux only) |
 | `cli` | CLI support for binaries |
+| `profiling` | Profiling support for profiler binary |
+| `cpuid` | CPU-aware WindowLog detection (x86_64 only) |
+| `io-uring-io` | io_uring support for Linux 5.6+ |
 
 ## See Also
 
 - `CLAUDE.md` - Developer guidelines and conventions
-- `tests/s3_pipeline_tests.rs` - Integration tests
-- `crates/roboflow-dataset/src/lerobot/` - Dataset writer implementation
+- `tests/` - Integration and E2E tests
+- `crates/roboflow-dataset/src/` - Dataset writer and source implementations
 - `crates/roboflow-distributed/src/` - Distributed coordination

CLAUDE.md

@@ -14,15 +14,17 @@ Roboflow: Distributed data transformation pipeline converting robotics bag/MCAP
 
 ## Workspace Structure
 
-The project uses a Cargo workspace with 5 crates:
+The project uses a Cargo workspace with 7 crates:
 
 | Crate | Purpose |
 |-------|---------|
 | `roboflow-core` | Error types, registry, values |
 | `roboflow-storage` | S3, OSS, Local storage (always available) |
-| `roboflow-dataset` | KPS, LeRobot, streaming converters |
-| `roboflow-distributed` | TiKV client, catalog, circuit breaker |
-| `roboflow-hdf5` | Optional HDF5 format support |
+| `roboflow-executor` | Stage-based task executor for distributed pipelines |
+| `roboflow-media` | Image and video encoding/decoding |
+| `roboflow-dataset` | Dataset writers, sources (MCAP, bag), streaming converters |
+| `roboflow-pipeline` | Pipeline execution and stages for dataset processing |
+| `roboflow-distributed` | TiKV client, catalog, circuit breaker, worker coordination |
 
 **Import patterns:**
 - Use facade re-exports from `roboflow`: `use roboflow::{Robocodec, DatasetWriter, ...}`

README.md

@@ -2,6 +2,7 @@
 
 [![License: MulanPSL-2.0](https://img.shields.io/badge/License-MulanPSL--2.0-blue.svg)](http://license.coscl.org.cn/MulanPSL2)
 [![Rust](https://img.shields.io/badge/rust-1.80%2B-orange.svg)](https://www.rust-lang.org)
+[![codecov](https://codecov.io/gh/archebase/roboflow/branch/main/graph/badge.svg)](https://codecov.io/gh/archebase/roboflow)
 
 [English](README.md) | [简体中文](README_zh.md)
 
@@ -73,46 +74,52 @@ Roboflow uses a **Kubernetes-inspired distributed control plane** for fault-tole
 |-------|---------|
 | `roboflow-core` | Error types, registry, values |
 | `roboflow-storage` | S3, OSS, Local storage (always available) |
-| `roboflow-dataset` | KPS, LeRobot, streaming converters |
-| `roboflow-distributed` | TiKV client, catalog, circuit breaker |
-| `roboflow-hdf5` | Optional HDF5 format support |
+| `roboflow-executor` | Stage-based task executor for distributed pipelines |
+| `roboflow-media` | Image and video encoding/decoding for robotics datasets |
+| `roboflow-dataset` | KPS, LeRobot, streaming converters, data sources |
+| `roboflow-pipeline` | Pipeline execution and stages for dataset processing |
+| `roboflow-distributed` | TiKV client, catalog, circuit breaker, worker coordination |
 
 ## Quick Start
 
-### Submit a Conversion Job
-
-```bash
-roboflow submit \
-  --input s3://bucket/input.bag \
-  --output s3://bucket/output/ \
-  --config lerobot_config.toml
-```
-
-### Run a Worker
+### Run the Unified Service
 
 ```bash
+# Set environment variables
 export TIKV_PD_ENDPOINTS="127.0.0.1:2379"
 export AWS_ACCESS_KEY_ID="your-key"
 export AWS_SECRET_ACCESS_KEY="your-secret"
 
-roboflow worker
+# Run unified service (scanner + worker + finalizer + reaper)
+roboflow run
 ```
 
-### Run a Scanner
+### Run Specific Roles
 
 ```bash
-export SCANNER_INPUT_PREFIX="s3://bucket/input/"
-export SCANNER_OUTPUT_PREFIX="s3://bucket/jobs/"
+# Worker only - processes work units
+roboflow run --role worker
 
-roboflow scanner
+# Finalizer only - merges completed batches
+roboflow run --role finalizer
+
+# With custom pod ID
+roboflow run --pod-id my-pod-1
 ```
 
-### List Jobs
+### Submit a Conversion Job
+
+```bash
+roboflow submit s3://bucket/input.bag --output s3://bucket/output/
+```
+
+### Manage Jobs
 
 ```bash
 roboflow jobs list
 roboflow jobs get <job-id>
-roboflow jobs retry <job-id>
+roboflow batch list
+roboflow batch get <batch-id>
 ```
 
 ## Installation
@@ -166,6 +173,7 @@ encoding = "cdr"
 | `WORKER_POLL_INTERVAL_SECS` | Job poll interval | `5` |
 | `WORKER_MAX_CONCURRENT_JOBS` | Max concurrent jobs | `1` |
 | `SCANNER_SCAN_INTERVAL_SECS` | Scan interval | `60` |
+| `FINALIZER_POLL_INTERVAL_SECS` | Finalizer poll interval | `30` |
 
 ## Development
 
@@ -188,6 +196,22 @@ cargo fmt
 cargo clippy --all-targets -- -D warnings
 ```
 
+### Development Infrastructure
+
+Start required services with Docker Compose:
+
+```bash
+docker compose up -d       # Start all services (MinIO, TiKV, PD)
+docker compose down        # Stop all services
+```
+
+**Services:**
+| Service | Purpose | Ports |
+|---------|---------|-------|
+| MinIO | S3-compatible object storage | 9000 (API), 9001 (Console) |
+| TiKV | Distributed KV storage | 20160 |
+| PD | TiKV placement driver | 2379, 2380 |
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.