dlm-gitops-stack

A cloud-agnostic, GitOps-driven framework for deploying a complete Data Lifecycle Management (DLM) ecosystem. This repository provides the automated scaffolding to bootstrap Kubernetes clusters across major cloud providers and manage the deployment of Rucio, FTS and associated metadata databases.

Objectives

• Multi-Cloud Flexibility: Standardized deployment patterns for AWS EKS, Azure AKS, and GCP GKE.
• GitOps Workflow: Using ArgoCD as the single source of truth for both infrastructure configuration and application state.
• Automated Data Management: Seamlessly install and configure Rucio (Data Management) and FTS (File Transfer Service).
• External Metadata Integration: Automated connection strings and secret management for external databases (PostgreSQL/Oracle/MySQL) used for metadata persistence.

Project Structure

.
├── bootstrap/                # ArgoCD installation and Root 'App-of-Apps'
│   └── argocd-install/       # Kubernetes manifests for ArgoCD core
├── clusters/                 # Cluster-specific configurations
│   ├── aws-eks/              # EKS specific overrides and add-ons
│   ├── azure-aks/            # AKS specific overrides and add-ons
│   └── gcp-gke/              # GKE specific overrides and add-ons
├── infrastructure/           # Terraform/IaC for managed K8s & DBs
│   ├── modules/              # Reusable cloud modules
│   └── environments/         # Provider-specific TF variables
└── services/                 # Helm charts / Kustomize overlays
    ├── rucio/                # Rucio server, auth, and daemons
    ├── fts/                  # FTS3 server and monitoring
    └── external-db/          # External database endpoints & secrets

Quick Start (In a Nutshell)

1. Provision Infrastructure: Use the /infrastructure directory to spin up your managed Kubernetes cluster and external database instances via Terraform.
2. Bootstrap ArgoCD: Apply the manifests in /bootstrap to your cluster.

   kubectl apply -k bootstrap/argocd-install/

3. Sync the Stack: Point ArgoCD to the /clusters directory corresponding to your cloud provider. ArgoCD will automatically:
   • Deploy Rucio and FTS components.
   • Configure ingress and load balancers.
   • Inject database credentials for metadata management.

Tech Stack

• Orchestration: Kubernetes (EKS, AKS, GKE)
• GitOps: ArgoCD
• Data Management: Rucio
• Transfer Service: FTS3
• Infrastructure: Terraform / Helm / Kustomize