Clean up docs, ensure examples for YAML, add in advanced SQL section

Author: pflooky

CLAUDE.md

@@ -0,0 +1,166 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Data Caterer is a test data management tool built with Scala and Apache Spark that provides automated data generation, validation, and cleanup capabilities. It supports multiple data sources including databases, files, messaging systems, and HTTP APIs.
+
+## Build System & Common Commands
+
+The project uses Gradle with Kotlin DSL and follows a multi-module structure:
+- **Root module**: Configuration and project orchestration
+- **api**: Builder patterns and models for programmatic usage  
+- **app**: Core execution engine, Spark integration, and UI server
+- **example**: Sample implementations and Docker configurations
+
+### Essential Commands
+
+```bash
+# Build the entire project
+./gradlew build
+
+# Build individual modules
+./gradlew :app:build
+./gradlew :api:build
+
+# Run tests (use exact class names, NOT wildcards)
+./gradlew :app:test --tests "io.github.datacatering.datacaterer.core.ui.plan.PlanRepositoryTest" --info
+./gradlew :api:test
+
+# Generate test coverage with Scoverage
+./gradlew reportScoverage
+
+# Create fat/shadow JAR for distribution
+./gradlew :app:shadowJar
+
+# Run specific configurations from IDE
+./gradlew :app:run --args="DataCatererUI"
+```
+
+### Important Test Running Notes
+
+ScalaTest with JUnit Platform has limitations with Gradle's `--tests` filtering:
+- ✅ Use exact class names: `--tests "io.github.datacatering.datacaterer.core.ui.plan.PlanRepositoryTest"`
+- ❌ Do NOT use wildcards: `--tests "*PlanRunTest*"` (runs ALL tests instead of filtering)
+
+## Architecture Overview
+
+### Core Domain Concepts
+
+- **Plans**: High-level configuration defining data operations to perform
+- **Tasks**: Individual data sources (databases, files, messaging systems, HTTP)
+- **Steps**: Sub-operations within tasks (tables, topics, file paths)
+- **Fields**: Individual data field configurations with generation rules
+- **Validations**: Data quality checks and assertions
+
+### Module Structure
+
+```
+api/                          # Builder API and models
+├── model/                    # Core data models and types
+├── connection/              # Data source connection builders
+└── validation/              # Validation builders
+
+app/                          # Core application
+├── core/
+│   ├── generator/           # Data generation engine
+│   ├── validator/           # Data validation engine
+│   ├── sink/               # Data output processors
+│   ├── metadata/           # Metadata discovery and integration
+│   ├── ui/                 # Web UI server components
+│   └── util/               # Utilities and helpers
+└── main/resources/          # Configuration files and UI assets
+```
+
+### Key Architectural Patterns
+
+**Builder Pattern**: All configuration uses immutable builders with method chaining
+```scala
+postgres("customer_postgres", "jdbc:postgresql://localhost:5432/customer")
+  .table("accounts")  
+  .fields(field.name("account_id").regex("ACC[0-9]{10}").unique(true))
+```
+
+**Case Class Data Models**: Immutable data structures with Jackson JSON serialization
+```scala
+@JsonIgnoreProperties(ignoreUnknown = true)
+case class DataSource(
+  name: String,
+  `type`: String,
+  options: Map[String, String] = Map(),
+  enabled: Boolean = true
+)
+```
+
+**Spark Integration**: Uses Apache Spark for distributed data processing and Spark SQL for data operations
+
+## Development Patterns
+
+### Code Style Requirements
+
+- Use `com.softwaremill.quicklens.ModifyPimp` for immutable updates in builders
+- Always provide parameterless constructors: `def this() = this(DefaultValue())`
+- Use `@JsonIgnoreProperties(ignoreUnknown = true)` for JSON serialization compatibility
+- Use `Option[T]` instead of `null` for optional values
+- Follow package structure under `io.github.datacatering.datacaterer`
+
+### Builder Implementation Pattern
+
+```scala
+case class TaskBuilder(task: Task = Task()) {
+  def this() = this(Task())
+  
+  def name(name: String): TaskBuilder = 
+    this.modify(_.task.name).setTo(name)
+    
+  def option(option: (String, String)): TaskBuilder =
+    this.modify(_.task.options)(_ ++ Map(option))
+}
+```
+
+### Environment Configuration
+
+Runtime behavior is controlled via environment variables:
+- `ENABLE_GENERATE_DATA`: Enable/disable data generation
+- `ENABLE_DELETE_GENERATED_RECORDS`: Enable cleanup mode
+- `PLAN_FILE_PATH`: Path to YAML plan configuration
+- `TASK_FOLDER_PATH`: Directory containing task definitions
+- `APPLICATION_CONFIG_PATH`: Custom application configuration
+
+### Data Source Support
+
+The system supports:
+- **Databases**: Postgres, MySQL, Cassandra, BigQuery
+- **Files**: CSV, JSON, Parquet, Delta Lake, Iceberg, ORC
+- **Messaging**: Kafka, RabbitMQ, Solace
+- **HTTP**: REST APIs with OpenAPI/Swagger integration
+- **Metadata Sources**: Great Expectations, JSON Schema, Data Contract CLI, OpenMetadata, Marquez
+
+## UI and API Integration
+
+The application includes a web UI server that provides:
+- Connection management and testing
+- Interactive plan creation
+- Execution history tracking
+- Real-time results viewing
+
+The UI is implemented as a separate module with React frontend and Scala backend using HTTP4S.
+
+## Testing Strategy
+
+- Use ScalaTest for unit testing
+- Test both API builders and core application logic
+- Mock external dependencies (databases, file systems)
+- Use exact class names for test filtering, not wildcards
+- Leverage the example module for integration testing
+
+## Key Dependencies
+
+- **Scala**: 2.12.x
+- **Apache Spark**: 3.5.x
+- **Jackson**: JSON serialization
+- **Quicklens**: Immutable data updates
+- **ScalaTest**: Testing framework
+- **HTTP4S**: Web server framework
+- **Logback/Log4j**: Logging

docs/docs/configuration.md

@@ -31,6 +31,7 @@ Flags are used to control which processes are executed when you run Data Caterer
 | `enableRecordTracking`         | false   | Enable/disable which data records have been generated for any data source                                                                                                                                                   |
 | `enableDeleteGeneratedRecords` | false   | Delete all generated records based off record tracking (if `enableRecordTracking` has been set to true)                                                                                                                     |
 | `enableGenerateValidations`    | false   | If enabled, it will generate validations based on the data sources defined.                                                                                                                                                 |
+| `enableFastGeneration`         | false   | Enable fast generation to maximize throughput. This automatically disables slower features and applies runtime optimizations for maximum performance |
 
 === "Java"
 
@@ -96,6 +97,40 @@ Flags are used to control which processes are executed when you run Data Caterer
       enableGenerateValidations = ${?ENABLE_GENERATE_VALIDATIONS}
       enableAlerts = false
       enableAlerts = ${?ENABLE_ALERTS}
+      # Fast generation disables slower features for maximum throughput
+      enableFastGeneration = false
+      enableFastGeneration = ${?ENABLE_FAST_GENERATION}
+    }
+    ```
+
+### Fast generation mode
+
+Enable fast generation to maximize throughput. This automatically disables slower features (record tracking, count,
+sink metadata, unique checks, save reports, validations, alerts) and applies runtime optimizations (e.g. lower shuffle
+partitions, enable AQE, Kryo serializer) and increases `numRecordsPerBatch`.
+
+[:material-run-fast: Scala Example](https://github.com/data-catering/data-caterer-example/blob/main/src/main/scala/io/github/datacatering/plan/FastGenerationAndReferencePlanRun.scala) | [:material-coffee: Java Example](https://github.com/data-catering/data-caterer-example/blob/main/src/main/java/io/github/datacatering/plan/FastGenerationAndReferenceJavaPlanRun.java)
+
+=== "Java"
+
+    ```java
+    configuration()
+      .enableFastGeneration(true);
+    ```
+
+=== "Scala"
+
+    ```scala
+    configuration
+      .enableFastGeneration(true)
+    ```
+    
+=== "application.conf"
+
+    ```
+    flags {
+      enableFastGeneration = true
+      enableFastGeneration = ${?ENABLE_FAST_GENERATION}
     }
     ```
 
@@ -216,6 +251,11 @@ when analysing the generated data if the number of records generated is large.
       oneOfDistinctCountVsCountThreshold = 0.2
       numGeneratedSamples = 10
     }
+
+    uniqueCheck {
+      uniqueBloomFilterNumItems = 100000
+      uniqueBloomFilterFalsePositiveProbability = 0.1
+    }
     ```
 
 ## Generation
@@ -289,6 +329,35 @@ Configurations to alter how validations are executed.
     }
     ```
 
+### Unique generation tuning
+
+If `enableUniqueCheck` is enabled, you can tune the underlying Bloom filter used for uniqueness checks to balance memory usage and false positive probability.
+
+=== "Java"
+
+    ```java
+    configuration()
+      .uniqueBloomFilterNumItems(100000L)
+      .uniqueBloomFilterFalsePositiveProbability(0.1);
+    ```
+
+=== "Scala"
+
+    ```scala
+    configuration
+      .uniqueBloomFilterNumItems(100000L)
+      .uniqueBloomFilterFalsePositiveProbability(0.1)
+    ```
+    
+=== "application.conf"
+
+    ```
+    uniqueCheck {
+      uniqueBloomFilterNumItems = 100000
+      uniqueBloomFilterFalsePositiveProbability = 0.1
+    }
+    ```
+
 ## Runtime
 
 Given Data Caterer uses Spark as the base framework for data processing, you can configure the job as to your 

docs/docs/generator/count.md

@@ -413,7 +413,7 @@ It can generate a dataset like below where all combinations of `debit_credit` an
     csv("transactions", "app/src/test/resources/sample/csv/transactions")
       .fields(
         field().name("account_id"),
-        field().name("debit_creidt").oneOf("D", "C"),
+        field().name("debit_credit").oneOf("D", "C"),
         field().name("status").oneOf("open", "closed", "suspended")
       )
       .allCombinations(true);
@@ -425,7 +425,7 @@ It can generate a dataset like below where all combinations of `debit_credit` an
     csv("transactions", "app/src/test/resources/sample/csv/transactions")
       schema(
         field.name("account_id"),
-        field.name("debit_creidt").oneOf("D", "C"),
+        field.name("debit_credit").oneOf("D", "C"),
         field.name("status").oneOf("open", "closed", "suspended")
       )
       .allCombinations(true)