Feature/real time sink patterns (#120)

* Update version to 0.18.0, add new execution strategies including duration-based and ramp load patterns, enhance validation capabilities, and improve performance metrics collection. Introduce new test plans and examples for various execution strategies, along with updates to documentation and Docker configurations.

* feat(foreign-key): implement strategies and utilities for foreign keys

- Add CardinalityConfigBuilder and NullabilityConfigBuilder for configuring cardinality and nullability in relationships
- Introduce new processors for foreign key uniqueness and connection resolution
- Add comprehensive tests for new features and update existing test cases
- Enhance documentation with new YAML examples and integration tests for execution strategies, including performance test examples

This implementation enables flexible foreign key management in data generation, supporting various relationship patterns and improving data integrity validation.

* Fix cardinality and nullability config to be at foreign key target level and not top level of foreign keys

* fix: revert back plan models changes and unified yaml, cleanup imports

* feat: enhance CI/CD workflows for multi-architecture builds

- Updated the build workflow to support both amd64 and arm64 architectures for packaging the application as a Debian package.
- Introduced a new workflow for testing Linux ARM64 builds, including setup for QEMU and Docker Buildx for cross-platform compatibility.
- Adjusted artifact naming conventions to clearly indicate architecture in the output files.

* fix: update CI/CD workflows to install fakeroot before jpackage

- Modified the build and test workflows to include the installation of fakeroot prior to executing the jpackage command, ensuring successful packaging of the application for multi-architecture builds.

* fix: remove OS specific installation, fix integration test, update quick start

* Enhance foreign key strategies with deterministic behavior using hash-based approaches

- Updated ForeignKeyEndToEndIntegrationTest to assert expected null rows for foreign keys based on deterministic hash values.
- Modified CardinalityStrategy, DistributedSamplingStrategy, GenerationModeStrategy, and NullabilityStrategy to utilize hash-based methods for consistent results across different Spark environments.
- Introduced SimplePercentileCalculator for efficient percentile calculations in performance metrics, replacing the deprecated T-Digest.
- Added tests for new deterministic behaviors in foreign key strategies and updated existing tests for consistency.
- Improved DataGenerator to support deterministic SQL generation with seed-based hash functions.

* Refactor YAML plan and task handling, enhance foreign key processing

- Removed obsolete integration test steps from GitHub Actions workflow.
- Improved logging in StepDataCoordinator for better debugging during record generation.
- Updated CardinalityCountAdjustmentProcessor to ensure only foreign key target steps are modified, preventing unintended changes.
- Added new YAML plan and task files for account balances and transactions, including validation of foreign key relationships.
- Introduced integration tests for YAML plan execution to verify record counts and foreign key integrity.

Author: pflooky

.github/workflows/build.yml

@@ -52,221 +52,3 @@ jobs:
           ORG_GRADLE_PROJECT_signingKey: ${{ secrets.ORG_GRADLE_PROJECT_SIGNINGKEY }}
           ORG_GRADLE_PROJECT_signingKeyId: ${{ secrets.ORG_GRADLE_PROJECT_SIGNINGKEYID }}
           ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.ORG_GRADLE_PROJECT_SIGNINGPASSWORD }}
-      - name: Upload jpackage jar
-        uses: actions/upload-artifact@v4
-        with:
-          name: jars
-          path: "app/build/libs/data-caterer.jar"
-          overwrite: true
-
-  osx:
-    needs: build
-    strategy:
-      matrix:
-        include:
-          - runner: macos-13  # Intel x64
-            arch: x64
-            arch_name: x86_64
-          - runner: macos-14  # Apple Silicon arm64
-            arch: aarch64
-            arch_name: aarch64
-    runs-on: ${{ matrix.runner }}
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2
-      - name: Set version
-        run: |
-          BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
-          COMMIT_HASH=$(git rev-parse --short HEAD)
-
-          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
-            APP_VERSION="$BASE_VERSION"
-          else
-            APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
-          fi
-
-          echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
-          echo "Building version: $APP_VERSION"
-      - uses: actions/setup-java@v4
-        with:
-          java-version: '21'
-          java-package: jdk
-          architecture: ${{ matrix.arch }}
-          distribution: oracle
-      - name: Download fat jar
-        uses: actions/download-artifact@v4
-        with:
-          name: jars
-          path: app/build/libs/
-      - name: Package jar as dmg installer
-        run: 'jpackage --main-jar data-caterer.jar "@misc/jpackage/jpackage.cfg" "@misc/jpackage/jpackage-mac.cfg"'
-      - name: Rename DMG with version and architecture
-        run: mv DataCaterer-*.dmg DataCaterer-${{ env.APP_VERSION }}-macos-${{ matrix.arch_name }}.dmg
-      - name: Upload dmg
-        uses: actions/upload-artifact@v4
-        with:
-          name: data-caterer-macos-${{ matrix.arch_name }}
-          path: "DataCaterer-${{ env.APP_VERSION }}-macos-${{ matrix.arch_name }}.dmg"
-          overwrite: true
-
-  windows:
-    needs: build
-    runs-on: [windows-latest]
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2
-      - name: Set version
-        run: |
-          $BASE_VERSION = (Get-Content gradle.properties | Select-String '^version=' | ForEach-Object { $_ -replace 'version=','' }).Trim()
-          $COMMIT_HASH = git rev-parse --short HEAD
-
-          if ("${{ github.ref }}" -eq "refs/heads/main") {
-            $APP_VERSION = $BASE_VERSION
-          } else {
-            $APP_VERSION = "$BASE_VERSION-$COMMIT_HASH"
-          }
-
-          echo "APP_VERSION=$APP_VERSION" >> $env:GITHUB_ENV
-          Write-Output "Building version: $APP_VERSION"
-      - uses: actions/setup-java@v4
-        with:
-          java-version: '21'
-          java-package: jdk
-          architecture: x64
-          distribution: oracle
-      - name: Download fat jar
-        uses: actions/download-artifact@v4
-        with:
-          name: jars
-          path: app/build/libs/
-      - name: Package jar as exe
-        run: 'jpackage --main-jar data-caterer.jar "@misc/jpackage/jpackage.cfg" "@misc/jpackage/jpackage-windows.cfg"'
-      - name: Rename EXE with version and architecture
-        run: mv DataCaterer-*.exe DataCaterer-$env:APP_VERSION-windows-x86_64.exe
-      - name: Upload installer
-        uses: actions/upload-artifact@v4
-        with:
-          name: data-caterer-windows-x86_64
-          path: "DataCaterer-${{ env.APP_VERSION }}-windows-x86_64.exe"
-          overwrite: true
-
-  linux-amd64:
-    needs: build
-    runs-on: [ubuntu-latest]
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2
-      - name: Set version
-        run: |
-          BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
-          COMMIT_HASH=$(git rev-parse --short HEAD)
-
-          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
-            APP_VERSION="$BASE_VERSION"
-          else
-            APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
-          fi
-
-          echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
-          echo "Building version: $APP_VERSION"
-      - uses: actions/setup-java@v4
-        with:
-          java-version: '21'
-          java-package: jdk
-          architecture: ${{ matrix.arch }}
-          distribution: oracle
-      - name: Download fat jar
-        uses: actions/download-artifact@v4
-        with:
-          name: jars
-          path: app/build/libs/
-      - name: Package jar as debian package (amd64)
-        run: 'jpackage --main-jar data-caterer.jar "@misc/jpackage/jpackage.cfg" "@misc/jpackage/jpackage-linux.cfg"'
-      - name: Rename deb with version
-        run: |
-          DEB_FILE=$(ls datacaterer_*_amd64.deb 2>/dev/null | head -n 1)
-          if [ -n "$DEB_FILE" ]; then
-            echo "Found deb file: $DEB_FILE"
-            mv "$DEB_FILE" datacaterer_${{ env.APP_VERSION }}_amd64.deb
-            echo "Renamed to: datacaterer_${{ env.APP_VERSION }}_amd64.deb"
-          else
-            echo "No deb file found"
-            echo "Current directory:"
-            pwd
-            echo "Files in current directory:"
-            ls -lart
-            exit 1
-          fi
-      - name: Upload deb (amd64)
-        uses: actions/upload-artifact@v4
-        with:
-          name: data-caterer-linux-amd64
-          path: "datacaterer_${{ env.APP_VERSION }}_amd64.deb"
-          overwrite: true
-
-  linux-arm64:
-    needs: build
-    runs-on: [ubuntu-latest]
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2
-      - name: Set version
-        run: |
-          BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
-          COMMIT_HASH=$(git rev-parse --short HEAD)
-
-          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
-            APP_VERSION="$BASE_VERSION"
-          else
-            APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
-          fi
-
-          echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
-          echo "Building version: $APP_VERSION"
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: arm64
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-      - name: Download fat jar
-        uses: actions/download-artifact@v4
-        with:
-          name: jars
-          path: app/build/libs/
-      - name: Package jar as debian package (arm64)
-        run: |
-          docker run --rm --platform linux/arm64 \
-            -v $(pwd):/workspace \
-            -w /workspace \
-            arm64v8/eclipse-temurin:21-jdk \
-            bash -c "apt-get update && apt-get install -y fakeroot && jpackage --main-jar data-caterer.jar '@misc/jpackage/jpackage.cfg' '@misc/jpackage/jpackage-linux.cfg'"
-      - name: Rename output to indicate version and architecture (arm64)
-        run: |
-          DEB_FILE=$(ls datacaterer_*_arm64.deb 2>/dev/null | head -n 1)
-          if [ -n "$DEB_FILE" ]; then
-            echo "Found deb file: $DEB_FILE"
-            mv "$DEB_FILE" datacaterer_${{ env.APP_VERSION }}_arm64.deb
-            echo "Renamed to: datacaterer_${{ env.APP_VERSION }}_arm64.deb"
-          else
-            echo "No deb file found"
-            echo "Current directory:"
-            pwd
-            echo "Files in current directory:"
-            ls -lart
-            exit 1
-          fi
-      - name: Upload deb (arm64)
-        uses: actions/upload-artifact@v4
-        with:
-          name: data-caterer-linux-arm64
-          path: "datacaterer_${{ env.APP_VERSION }}_arm64.deb"
-          overwrite: true

.github/workflows/check.yml

@@ -23,17 +23,3 @@ jobs:
       - name: Run gradle integration tests
         run: |
           ./gradlew :app:integrationTest --info
-      - name: Run intsa-integration tests
-        id: tests
-        uses: data-catering/insta-integration@v4
-      - name: Print results
-        run: |
-          echo "Records generated:         ${{ steps.tests.outputs.num_records_generated }}"
-          echo "Successful validations:    ${{ steps.tests.outputs.num_success_validations }}"
-          echo "Failed validations:        ${{ steps.tests.outputs.num_failed_validations }}"
-          echo "Number of validations:     ${{ steps.tests.outputs.num_validations }}"
-          echo "Validation success rate:   ${{ steps.tests.outputs.validation_success_rate }}"
-
-          if [ "${{ steps.tests.outputs.num_failed_validations }}" -gt 0 ]; then
-            exit 1
-          fi

README.md

@@ -34,35 +34,39 @@ and deep dive into issues [from the generated report](https://data.catering/late
 
 ![Basic flow](misc/design/basic_data_caterer_flow_medium.gif)
 
-## Quick start
-
-1. Docker
-   ```shell
-   docker run -d -i -p 9898:9898 -e DEPLOY_MODE=standalone --name datacaterer datacatering/data-caterer:0.17.3
-   ```
-   [Open localhost:9898](http://localhost:9898).
-1. [Run Scala/Java examples](#run-scalajava-examples)
-   ```shell
-   git clone git@github.com:data-catering/data-caterer-example.git
-   cd data-caterer-example && ./run.sh
-   #check results under docker/sample/report/index.html folder
-   ```
-1. UI App Downloads (Nightly builds from `main` branch)
-   - **macOS**:
-     - [Intel (x86_64)](https://nightly.link/data-catering/data-caterer/workflows/build/main/data-caterer-macos-x86_64.zip)
-     - [Apple Silicon (M1/M2/M3)](https://nightly.link/data-catering/data-caterer/workflows/build/main/data-caterer-macos-aarch64.zip)
-   - **Windows**:
-     - [x64](https://nightly.link/data-catering/data-caterer/workflows/build/main/data-caterer-windows-x86_64.zip)
-       1. After downloading, go to 'Downloads' folder and 'Extract All' from data-caterer-windows-x86_64
-       1. Double-click the installer to install Data Caterer
-       1. Click on 'More info' then at the bottom, click 'Run anyway'
-       1. Go to '/Program Files/DataCaterer' folder and run DataCaterer application
-       1. If your browser doesn't open, go to [http://localhost:9898](http://localhost:9898) in your preferred browser
-   - **Linux**:
-     - [amd64](https://nightly.link/data-catering/data-caterer/workflows/build/main/data-caterer-linux-amd64.zip)
-     - [arm64](https://nightly.link/data-catering/data-caterer/workflows/build/main/data-caterer-linux-arm64.zip)
-
-[Follow quick start instructions from here if you want more details](https://data.catering/latest/get-started/quick-start/).
+## Quick Start
+
+### Java/Scala API (Recommended)
+
+```shell
+git clone git@github.com:data-catering/data-caterer.git
+cd data-caterer/example
+./run.sh
+```
+
+It will run the [`DocumentationPlanRun`](example/src/main/scala/io/github/datacatering/plan/DocumentationPlanRun.scala) class.
+Press Enter to run the default example. Check results at `docker/sample/report/index.html`.
+
+### YAML
+
+```shell
+git clone git@github.com:data-catering/data-caterer.git
+cd data-caterer/example
+./run.sh csv.yaml
+```
+
+It will run the [`csv.yaml`](example/docker/data/custom/plan/csv.yaml) plan file and the [`csv_transaction_file`](example/docker/data/custom/task/file/csv/csv_transaction_file.yaml) task file.
+Check results at `docker/data/custom/report/index.html`.
+
+### UI
+
+```shell
+docker run -d -p 9898:9898 -e DEPLOY_MODE=standalone --name datacaterer datacatering/data-caterer:0.18.0
+```
+
+Open [http://localhost:9898](http://localhost:9898).
+
+[**Full quick start guide**](https://data.catering/latest/get-started/quick-start/)
 
 
 ## Integrations
@@ -123,12 +127,6 @@ Different ways to run Data Caterer based on your use case:
 
 [Can check here for full list of roadmap items.](https://data.catering/latest/use-case/roadmap/)
 
-## Pricing
-
-Data Caterer is set up under a usage pricing model for the latest application version. There are different pricing tiers based on how much you use Data Caterer. This also includes support and requesting features. The current open-source version will be kept for those who want to continue using the open-source version.
-
-[Find out more details here.](https://data.catering/latest/pricing/)
-
 ### Mildly Quick Start
 
 #### Generate and validate data

api/src/main/java/io/github/datacatering/datacaterer/javaapi/api/PlanRun.java

@@ -240,7 +240,7 @@ public ForeignKeyRelation foreignField(String dataSource, String step, String fi
      * @return A ForeignKeyRelation instance.
      */
     public ForeignKeyRelation foreignField(String dataSource, String step, List<String> fields) {
-        return new ForeignKeyRelation(dataSource, step, toScalaList(fields));
+        return new ForeignKeyRelation(dataSource, step, toScalaList(fields), scala.Option.empty(), scala.Option.empty(), scala.Option.empty());
     }
 
     /**
@@ -255,7 +255,8 @@ public ForeignKeyRelation foreignField(ConnectionTaskBuilder<?> connectionTaskBu
         return new ForeignKeyRelation(
                 connectionTaskBuilder.connectionConfigWithTaskBuilder().dataSourceName(),
                 connectionTaskBuilder.getStep().step().name(),
-                toScalaList(List.of(field))
+                toScalaList(List.of(field)),
+                scala.Option.empty(), scala.Option.empty(), scala.Option.empty()
         );
     }
 
@@ -271,7 +272,8 @@ public ForeignKeyRelation foreignField(ConnectionTaskBuilder<?> connectionTaskBu
         return new ForeignKeyRelation(
                 connectionTaskBuilder.connectionConfigWithTaskBuilder().dataSourceName(),
                 connectionTaskBuilder.getStep().step().name(),
-                toScalaList(fields)
+                toScalaList(fields),
+                scala.Option.empty(), scala.Option.empty(), scala.Option.empty()
         );
     }
 
@@ -284,7 +286,7 @@ public ForeignKeyRelation foreignField(ConnectionTaskBuilder<?> connectionTaskBu
      * @return A ForeignKeyRelation instance.
      */
     public ForeignKeyRelation foreignField(ConnectionTaskBuilder<?> connectionTaskBuilder, String step, List<String> fields) {
-        return new ForeignKeyRelation(connectionTaskBuilder.connectionConfigWithTaskBuilder().dataSourceName(), step, toScalaList(fields));
+        return new ForeignKeyRelation(connectionTaskBuilder.connectionConfigWithTaskBuilder().dataSourceName(), step, toScalaList(fields), scala.Option.empty(), scala.Option.empty(), scala.Option.empty());
     }
 
     /**

api/src/main/scala/io/github/datacatering/datacaterer/api/DataCatererConfigurationBuilder.scala

@@ -3,7 +3,7 @@ package io.github.datacatering.datacaterer.api
 import com.softwaremill.quicklens.ModifyPimp
 import io.github.datacatering.datacaterer.api.connection.{BigQueryBuilder, CassandraBuilder, ConnectionTaskBuilder, FileBuilder, HttpBuilder, KafkaBuilder, MySqlBuilder, NoopBuilder, PostgresBuilder, RabbitmqBuilder, SolaceBuilder}
 import io.github.datacatering.datacaterer.api.converter.Converters.toScalaMap
-import io.github.datacatering.datacaterer.api.model.Constants.{BIGQUERY_WRITE_METHOD, _}
+import io.github.datacatering.datacaterer.api.model.Constants._
 import io.github.datacatering.datacaterer.api.model.DataCatererConfiguration
 
 import scala.annotation.varargs