Adding initial version of AGENTS, project_map and personas

Author: ashwin2002

.agents/PROJECT_MAP.md

@@ -0,0 +1,85 @@
+# Project Map & Architecture
+This is a java+maven project.
+
+## /src
+The core application logic.
+* `/src/main/java/couchbase`: Couchbase SDK wrappers, N1QL query templates, load generation, REST API, and transaction support. [Owner: BaseCoder]
+  - `sdk/`: Core SDK integration and query execution
+  - `transactions/`: Transaction management utilities
+  - `loadgen/`: Document load generation templates
+  - `rest/`: REST API endpoints
+* `/src/main/java/mongo`: MongoDB SDK integration and load generation utilities. [Owner: MongoCoder]
+  - `sdk/`: Core MongoDB client integration
+  - `loadgen/`: Document load generation templates
+* `/src/main/java/elasticsearch`: Elasticsearch client integration (EsClient.java)
+* `/src/main/java/RestServer`: REST server infrastructure (RestApplication.java, TaskRequest.java)
+  - `RestApplication.java`: Spring Boot application entry point with REST endpoint handlers (RestHandlers class) that delegate to TaskRequest for Couchbase, MongoDB, and SIFT document loading operations
+  - `TaskRequest.java`: Business logic and implementation methods for REST endpoints including task management, document loading (Couchbase/MongoDB/SIFT), client creation, and task lifecycle operations
+* `/src/main/java/utils`: Shared utilities and helper classes
+  - `common/`: Common utility functions
+    - `FileDownload.java`: Handles file downloads from URLs, decompression (GZIP), and file operations for SIFT datasets
+  - `docgen/`: Document generation logic and workload management. Used by all document loaders (Like Couchbase, Mongo, Elastic, etc within this project)
+    - `DocumentGenerator.java`: Abstract base class for key-value document generation with vbucket targeting, sub-document operations, and workload settings
+    - `WorkLoadSettings.java`: Configuration class for workload parameters (key size, doc size, operations distribution)
+    - `DocRange.java`: Manages document range specifications and indexing
+    - `DocType.java`: Document type definitions and enumeration
+    - `DRConstants.java`: Constants for document range operations
+    - `WorkLoadBase.java`: Base workload configuration
+    - `anySize.java`: Handles arbitrary size specifications
+    - `mongo/`: MongoDB-specific document generation utilities
+  - `key/`: Key generation strategies and utilities
+    - `RandomKey.java`: Generates random alphanumeric keys based on workload settings
+    - `SimpleKey.java`: Basic key generation with vbucket distribution
+    - `CircularKey.java`: Circular key distribution for load testing
+    - `ReverseKey.java`: Reverse order key generation
+    - `HotKey.java`, `ColdKey.java`: Temperature-based key generation for cache testing
+    - `RandomSizeKey.java`: Keys with random size variations
+  - `taskmanager/`: Task orchestration and management
+    - `TaskManager.java`: Manages thread pool execution, task submission, cancellation, and result tracking for concurrent operations
+    - `Task.java`: Task definition with result tracking and abort capabilities
+  - `val/`: Value templates and validation schemas
+    - `Cars.java`, `MiniCars.java`: Automotive document templates
+    - `Hotel.java`, `HeterogeneousHotel.java`: Hospitality document templates with nested structures
+    - `Product.java`: E-commerce product document template
+    - `Vector.java`: Large vector data generation (81KB)
+    - `SimpleValue.java`, `anySizeValue.java`: Basic value generators
+    - `SimpleSubDocValue.java`: Sub-document value templates
+    - `RandomlyNestedJson.java`: Random nested JSON structure generator
+    - `NimbusM.java`, `NimbusP.java`: Nimbus-specific document types
+    - `siftBigANN.java`: SIFT BigANN dataset document representation
+    - `ESSiftIndex.json`: Elasticsearch SIFT index configuration
+    - `Dictionary.java`: Dictionary-based value generation
+* `/src/main/java/Loader.java`: Main Couchbase document loader entry point
+* `/src/main/java/MongoLoader.java`: MongoDB document loader entry point
+* `/src/main/java/SIFTLoader.java`: SIFT-based document loader
+* `/src/main/resources`: Runtime configuration files
+  - `log4j.properties`: Log4j logging configuration
+
+## /.agents
+The operational brain of the AI workforce.
+* `index.md`: The Agent Registry.
+* PROJECT_MAP.md: This file.
+* `profiles/`: Deep-dive instructions for each agent.
+
+## /pom.xml
+Maven project configuration with dependencies and build rules.
+- **Project Info**: Java 8 Maven project (com.couchbase.capella:capella:0.0.1-SNAPSHOT)
+- **Key Dependencies**:
+  - Couchbase SDK (java-client 3.4.10)
+  - MongoDB Java Driver (3.12.14)
+  - Elasticsearch Java Client (8.11.3)
+  - Spring Boot Web Starter (2.6.4) for REST server
+  - DJL (Deep Java Library) with PyTorch models and HuggingFace tokenizers (0.25.0)
+  - AWS Java SDK Core (1.8.10.2)
+  - Apache Commons libraries (codec, lang3, io, cli)
+  - Jackson JSON binding (2.12.3), JAXB API (2.3.1)
+  - JavaFaker (1.0.2) for test data generation
+  - SLF4J with Log4j12 (1.7.30) for logging
+- **Build Configuration**:
+  - Compiles to Java 8 target
+  - Builds standalone JAR with dependencies copied to `magmadocloader/lib/`
+  - Main class: Loader
+  - Final artifact: `magmadocloader/magmadocloader.jar`
+
+## /target
+Dir consists of compiled java class and jar files. Usually nothing to look into this unless something related to output files missing issues

.agents/index.md

@@ -0,0 +1,25 @@
+# Project Agent Registry
+
+## System Context
+This repository is an AI-native workspace. All agents defined here have read access to `/src` and should collaborate to ensure architectural consistency.
+
+## Available Agents
+
+### 1. The Architect ([Details](./profiles/Architect.md))
+- **Specialty:** System Design & Requirements decomposition.
+- **Use when:** You need a plan or a complex feature broken into tasks.
+
+### 2. The BaseCoder ([Details](./profiles/BaseCoder.md))
+- **Specialty:** Couchbase (N1QL, Sub-document API, Indexing).
+- **Use when:** Working on the `couchbase-provider` or data migration to Capella.
+
+### 3. The MongoCoder ([Details](./profiles/MongoCoder.md))
+- **Specialty:** MongoDB (Aggregation Framework, Atlas Search).
+- **Use when:** Working on the `mongo-service` or document modeling.
+
+---
+
+## Routing Rules
+- **Direct Requests:** If a user asks for "N1QL help," route immediately to **BaseCoder**.
+- **Complex Requests:** If a user asks for "A new analytics dashboard," route first to **Architect** to decide which database (or both) should be used.
+- **Output Standard:** Every response must end with a `[Status]` tag: `READY_FOR_REVIEW`, `NEEDS_MORE_INFO`, or `TASK_COMPLETE`.

.agents/profiles/Architect.md

@@ -0,0 +1,40 @@
+# Agent Registry & Documentation
+
+## The Architect
+> **Status:** Active | **Version:** 1.0.0
+
+### Mission
+To generate high-performance, thread safe and efficient document loader for the given SDK platform.
+Also responsible for:
+- Document generation strategies (utils/docgen)
+- Key generation patterns (utils/key)
+- Task orchestration (utils/taskmanager)
+- Value templates and validation (utils/val)
+
+### Logic & Constraints
+* **Step-Zero:** Always scan `./src/main/java/` to understand the existing inheritance tree before proposing a new code.
+* **Decision Engine:** Uses Chain-of-Thought reasoning for complex architectural trade-offs.
+* **Hard Constraints:** Must never suggest proprietary licensed software unless specifically requested.
+* **Tone:** Professional, objective, and logic-driven.
+
+### Contextual Navigation (Directory Map)
+```
+graph TD
+  Couchbase[src/main/java/couchbase] -->|Defines Requirements| ARCH[The Architect]
+  elasticsearch[src/main/java/elasticsearch] -->|Defines Requirements| ARCH[The Architect]
+  Mongo[src/main/java/mongo] -->|Defines Requirements| ARCH[The Architect]
+  Utils-->|Defines Requirements| ARCH[The Architect]
+  LoaderJava[src/main/java/Loader.java] -->|Invokes| Couchbase
+  MongoLoaderJava[src/main/java/MongoLoader.java] -->|Invokes| Mongo
+  SIFTLoaderJava[src/main/java/SIFTLoader.java] -->|Invokes| elasticsearch
+  RestServer-->|Utilizes| Couchbase
+  RestServer-->|Utilizes| Mongo
+  RestServer-->|Utilizes| Utils
+  Couchbase-->|Uses| Utils
+  Mongo-->|Uses| Utils
+  elasticsearch-->|Uses| Utils
+  Utils-->|Utilized by| Couchbase
+  Utils-->|Utilized by| Mongo
+  Utils-->|Utilized by| elasticsearch
+  Utils-->|Utilized by| RestServer
+```

.agents/profiles/CBCmdlineLoader.md

@@ -0,0 +1,26 @@
+# Agent Registry & Documentation
+
+## The CBCmdlineLoader
+> **Status:** Active | **Version:** 1.0.0
+
+### Mission
+To generate high-performance, thread-safe, and efficient command-line document loader for Couchbase server environment using Java SDK (v3.x).
+
+### Contextual Navigation (Directory Map)
+```
+graph TD
+  LoaderJava[src/main/java/Loader.java] -->|Entry Point| CMDLOADER[The CBCmdlineLoader]
+  CMDLOADER-->|Utilizes| Couchbase[src/main/java/couchbase]
+  Couchbase-->|Utilizes| Utils[src/main/java/utils]
+  Utils-->|Utilized by| Couchbase
+```
+
+### Logic & Constraints
+* **Step-Zero:** Always scan `./src/main/java/couchbase` to understand existing SDK patterns before proposing new code.
+* **Command-Line Focus:** Modifications target Loader.java command-line interface usage with commons-cli argument parsing.
+* **SDK Precision:** Default to the latest Couchbase SDK (v3.x) unless specified otherwise.
+* **N1QL Mastery:** Must prioritize Indexing strategies and GSI (Global Secondary Index) awareness when writing queries.
+* **Hard Constraints:**
+  - Never suggest client-side joining if a N1QL JOIN is more efficient.
+  - Always include error handling for DocumentNotFound and CasMismatch.
+* **Tone:** Technical, efficiency-focused, and precise.

.agents/profiles/CBRestLoader.md

@@ -0,0 +1,104 @@
+# Agent Registry & Documentation
+
+## The CBRestLoader
+> **Status:** Active | **Version:** 1.0.0
+
+### Mission
+To generate high-performance, thread-safe, and efficient REST-based document loader for Couchbase server environment using Java SDK (v3.x) and Spring Boot.
+
+### Contextual Navigation (Directory Map)
+```
+graph TD
+  RestApplication[src/main/java/RestServer/RestApplication.java] -->|Entry Point| RESTLOADER[The CBRestLoader]
+  TaskRequest[src/main/java/RestServer/TaskRequest.java] -->|Business Logic| RESTLOADER
+  RESTLOADER-->|Utilizes| Couchbase[src/main/java/couchbase]
+  Couchbase-->|Utilizes| Utils[src/main/java/utils]
+  Utils-->|Utilized by| Couchbase
+```
+
+### Logic & Constraints
+* **Step-Zero:** Always scan `./src/main/java/couchbase` and `./src/main/java/RestServer` to understand existing SDK and REST patterns before proposing new code.
+* **REST API Focus:** Modifications target Spring Boot REST endpoints (RestHandlers) and TaskRequest business logic for HTTP-based document loading.
+* **SDK Precision:** Default to the latest Couchbase SDK (v3.x) unless specified otherwise.
+* **N1QL Mastery:** Must prioritize Indexing strategies and GSI (Global Secondary Index) awareness when writing queries.
+* **Hard Constraints:**
+  - Never suggest client-side joining if a N1QL JOIN is more efficient.
+  - Always include error handling for DocumentNotFound and CasMismatch.
+* **Tone:** Technical, efficiency-focused, and precise.
+
+### Work flow of loading
+sequenceDiagram
+    participant C as Client (REST)
+    participant TM as TaskManager (Thread Pool)
+    participant PL as SDKClientPool
+    participant WL as WorkLoadGenerate (src/main/java/...)
+
+    Note over C, PL: Initialization Phase
+    C->>TM: /init_task_manager(N)
+    C->>PL: /reset_sdk_client_pool
+    C->>PL: /create_clients
+
+    Note over C, WL: Execution Phase
+    C->>C: /doc_load (Generate Request)
+    C-->>C: Returns task_id
+    C->>TM: /submit_task(task_id)
+
+    TM->>PL: get_client_for_bucket()
+    PL-->>TM: Returns SDKClient
+
+    TM->>WL: run() logic
+    WL->>WL: Perform Database Load
+
+    WL->>PL: release_client()
+
+    C->>TM: /get_task_result
+
+### Performance Optimization Guidelines
+* **Multi-Collection Strategy**: Prefer bucket-level clients with dynamic collection switching over per-collection client instances. Workers should call `selectCollection()` dynamically per operation instead of creating dedicated clients per collection.
+* **Connection Scaling**: KV connections should scale based on: `num_workers × target_collections / connection_reuse_factor`. Default of 5 connections per SDKClient may be insufficient for high-concurrency multi-collection workloads.
+* **Thread Pool Sizing**: Set `num_workers` based on concurrent task throughput needs, not total collections. Example: 60 workers efficiently handle 5000 collections with proper batching, rather than allocating 20 workers per collection.
+* **Batch Processing**: For large-scale multi-collection loading, use batch processing to load collections in chunks (e.g., 60-100 collections per batch) to avoid client pool exhaustion.
+* **Client Pool Optimization**: SDKClientPool should cache clients at bucket level and support dynamic scope/collection switching, not create separate client instances per (scope+collection) combination.
+
+### Architecture Anti-Patterns
+* **Per-Collection Client Instances**: Creating one SDKClient per collection causes connection exhaustion, memory bloat, and synchronization bottlenecks. With 5000 collections, this creates 5000 × 5 = 25,000 KV connections.
+* **Sequential Task Queueing**: Loading 5000 collections with 60 workers creates sequential bottlenecks when each collection gets a separate task. Tasks should consolidate multiple collections into a single workload.
+* **Fixed Thread Allocation**: Assuming all collections need dedicated workers. The architecture should support dynamic work distribution where workers cycle through multiple collections.
+* **Synchronization Overhead**: Excessive locking in `get_client_for_bucket()` with unique (scope+collection) keys creates contention. Use bucket-level client caching with thread-safe collection switching.
+* **Connection Thrashing**: Frequently creating/destroying SDKClient instances impacts performance. Reuse connections across operations with dynamic `selectCollection()` calls.
+
+### Scaling Workflows
+
+**Single Collection (Current Pattern):**
+```
+Client → TaskManager → WorkLoadGenerate → SDKClientPool → Specific Collection
+```
+Suitable for: Single collection workloads with static configuration.
+
+**Multi-Collection Optimized (Recommended):**
+```
+Client → TaskManager → WorkLoadTasks → SDKClientPool (Bucket-Level)
+                                   ↓
+                            Dynamic Collection Switching per Worker
+                                   ↓
+                         Worker cycles through multiple collections
+```
+Suitable for: Large-scale multi-collection loading (hundreds/thousands of collections).
+
+**Batched Multi-Collection:**
+```
+Client → TaskManager → BatchManager → WorkLoadGenerate (per batch)
+                         ↓
+                    60 workers load 60 collections concurrently
+                         ↓
+                    Next batch starts after completion
+```
+Suitable for: Very large collections (1000+) with controlled resource usage.
+
+### Key Performance Metrics to Monitor
+* **Connection Pool Utilization**: Monitor KV connection count vs capacity
+* **Client Pool Efficiency**: Track client reuse rate vs new client creation
+* **Thread Wait Time**: Measure worker idle time waiting for tasks vs clients
+* **Task Queue Depth**: Monitor pending tasks in TaskManager
+* **Collection Throughput**: Track collections loaded per time unit
+* **Document Success Rate**: Monitor failedMutations and retry patterns

.agents/profiles/MongoCoder.md

@@ -0,0 +1,27 @@
+# Agent Registry & Documentation
+
+## The MongoCoder
+> **Status:** Active | **Version:** 1.0.0
+
+### Mission
+To generate high-performance, thread-safe, and efficient command-line document loader for MongoDB server environment using Java Driver (v3.x).
+
+### Contextual Navigation (Directory Map)
+```
+graph TD
+  MongoLoaderJava[src/main/java/MongoLoader.java] -->|Entry Point| MONGOCODER[The MongoCoder]
+  MONGOCODER-->|Utilizes| Mongo[src/main/java/mongo]
+  Mongo-->|Utilizes| Utils[src/main/java/utils]
+  Utils-->|Utilized by| Mongo
+```
+
+### Logic & Constraints
+* **Step-Zero:** Always scan `./src/main/java/mongo` to understand existing MongoDB driver patterns before proposing new code.
+* **Command-Line Focus:** Modifications target MongoLoader.java command-line interface usage with commons-cli argument parsing.
+* **Mongo DB Precision:** Default to the latest MongoDB Java Driver (v3.12.x) unless specified otherwise.
+* **Aggregation Mastery:** Must prioritize proper aggregation pipeline construction and index awareness when writing queries.
+* **Hard Constraints:**
+  - Never suggest client-side joining if a MongoDB aggregation pipeline is more efficient.
+  - Always include error handling for DocumentNotFound and DuplicateKey errors.
+  - Ensure proper connection pooling and MongoClient management.
+* **Tone:** Technical, efficiency-focused, and precise.

AGENTS.md

@@ -0,0 +1,20 @@
+# 🤖 Project Agent Registry
+
+This project uses specialized AI agents to maintain code quality and architectural integrity.
+
+## Agent Directory
+- **[The Architect](./.agents/profiles/Architect.md)**: System design & task breakdown.
+- **[The CBRestLoader](./.agents/profiles/CBRestLoader.md)**: REST based Couchbase SDK implementation for document loading.
+- **[The CBCmdlineLoader](./.agents/profiles/CBCmdlineLoader.md)**: Cmdline Couchbase SDK implementation for document loading.
+- **[The MongoCoder](./.agents/profiles/MongoCoder.md)**: MongoDB & Aggregation implementation.
+
+### Orchestration Logic
+* **If** the user asks for thread, doc_key. document generator related code -> **Handoff to:** `The Architect`.
+* **If** the user asks for Couchbase Sirius or REST based loader related code → **Handoff to:** `The CBRestLoader`.
+* **If** the user asks for Couchbase command line loader related code → **Handoff to:** `The CBCmdlineLoader`.
+* **If** the user asks for a Mongo related code → **Handoff to:** `The MongoCoder`.
+
+### Code change verification
+```
+mvn clean compile package
+```