Skip to content

Debasish-87/SLP

BranchesTags

Repository files navigation

Secure LLMOps Platform (SLP)

Secure LLMOps Platform (SLP) is a production-oriented LLMOps and Platform Engineering project designed to automate the deployment, management, and lifecycle of AI applications on AWS using Infrastructure as Code, Kubernetes, GitOps, and CI/CD.

The platform provisions cloud infrastructure using Terraform, builds and scans container images through GitHub Actions, stores them in Amazon Elastic Container Registry (ECR), and automatically deploys applications to Amazon Elastic Kubernetes Service (EKS) using Helm and ArgoCD.

SLP demonstrates how modern organizations deploy and manage AI services in production environments.

Table of Contents

  • Overview
  • Project Objectives
  • Features
  • Architecture
  • End-to-End Workflow
  • Technology Stack
  • Repository Structure
  • Infrastructure
  • Deployment Workflow
  • Monitoring
  • Security
  • Getting Started
  • Terraform
  • Docker
  • GitHub Actions
  • Kubernetes
  • Helm
  • ArgoCD
  • Monitoring
  • Usage
  • Troubleshooting
  • Future Improvements
  • License

Project Overview

SLP is not an AI model.

SLP is an AI Deployment Platform.

Its responsibility is to manage the complete lifecycle of AI applications including:

  • Infrastructure provisioning
  • Containerization
  • Continuous Integration
  • Continuous Deployment
  • GitOps
  • Kubernetes deployment
  • Monitoring
  • Security

The platform currently deploys the Enterprise Intelligence Agent (EIA).

The platform itself is generic and can deploy any AI application packaged as a Docker container.

Project Objectives

The primary goal of this project is to build a production-style LLMOps platform that demonstrates real-world DevOps and Platform Engineering practices.

Objectives include:

  • Infrastructure as Code
  • Automated Kubernetes deployment
  • GitOps workflow
  • Secure container delivery
  • Infrastructure automation
  • Production-ready architecture
  • Monitoring and observability
  • Security scanning

Key Features

Infrastructure

  • AWS VPC
  • Public and Private Subnets
  • Internet Gateway
  • NAT Gateway
  • Route Tables
  • Amazon EKS
  • Amazon ECR
  • IAM Roles

Terraform

  • Modular architecture
  • Remote state support
  • S3 backend
  • DynamoDB state locking
  • Bootstrap infrastructure

Container Platform

  • Docker
  • Python 3.12
  • FastAPI
  • Uvicorn

CI/CD

GitHub Actions automatically performs:

  • Checkout repository
  • Install dependencies
  • Execute tests
  • Build Docker image
  • Security scan
  • Push image to Amazon ECR

GitOps

ArgoCD automatically:

  • Watches Git repository
  • Detects Helm changes
  • Synchronizes Kubernetes cluster
  • Self heals deployments

Kubernetes

Helm automatically creates:

  • Deployment
  • Service
  • ServiceAccount
  • Ingress

Monitoring

Prometheus

Grafana

Security

Trivy

Amazon ECR Image Scanning

IAM

High Level Architecture


                 Developer

                     │

                Git Push

                     │

                     ▼

            GitHub Actions

                     │

      ┌──────────────┴──────────────┐

      │                             │

      ▼                             ▼

 Run Tests                  Docker Build

                                    │

                                    ▼

                            Trivy Security Scan

                                    │

                                    ▼

                             Amazon ECR

                                    │

                                    ▼

                           Update Helm Chart

                                    │

                                    ▼

                               Git Commit

                                    │

                                    ▼

                                 ArgoCD

                                    │

                                    ▼

                              Amazon EKS

                                    │

                                    ▼

                        Enterprise AI Agent

                                    │

                                    ▼

                         Prometheus Metrics

                                    │

                                    ▼

                              Grafana Dashboard

End-to-End Workflow

The complete deployment workflow is shown below.


Developer

↓

Git Push

↓

GitHub Actions

↓

Python Dependencies

↓

Unit Tests

↓

Docker Build

↓

Security Scan

↓

Push Docker Image

↓

Amazon ECR

↓

Update Helm Values

↓

Git Commit

↓

ArgoCD Detects Change

↓

Automatic Synchronization

↓

Amazon EKS

↓

Running AI Application

↓

Prometheus

↓

Grafana

Technology Stack

Cloud

  • Amazon Web Services

Infrastructure

  • Terraform

Containerization

  • Docker

Container Registry

  • Amazon Elastic Container Registry

Container Orchestration

  • Kubernetes

  • Amazon Elastic Kubernetes Service

Package Management

  • Helm

GitOps

  • ArgoCD

Continuous Integration

  • GitHub Actions

Monitoring

  • Prometheus

  • Grafana

Security

  • Trivy

Programming Language

  • Python

Framework

  • FastAPI

Repository Structure


SLP

├── terraform/

│   ├── bootstrap/

│   ├── environments/

│   └── modules/

├── docker/

├── helm/

│   └── eia/

├── argocd/

├── monitoring/

│   ├── prometheus/

│   └── grafana/

├── security/

│   ├── trivy/

│   └── external-secrets/

├── scripts/

├── .github/

│   └── workflows/

├── Makefile

└── README.md

Infrastructure Components

The platform provisions the following AWS infrastructure.

  • Virtual Private Cloud
  • Public Subnets
  • Private Subnets
  • Internet Gateway
  • NAT Gateway
  • Route Tables
  • Elastic Kubernetes Service
  • Elastic Container Registry
  • IAM Roles
  • Terraform Remote Backend
  • DynamoDB State Locking

Deployment Strategy

The platform follows GitOps.

No manual deployment is required.

The deployment pipeline is completely automated.

Developer only pushes code.

Everything else is performed automatically.


Developer

↓

Git Push

↓

CI

↓

Docker

↓

ECR

↓

Helm

↓

ArgoCD

↓

Kubernetes

Why Secure LLMOps Platform?

Modern AI systems require significantly more than model development.

Organizations require:

  • Infrastructure automation
  • Secure deployment
  • Version control
  • Continuous deployment
  • Rollback capability
  • Monitoring
  • Security
  • Scalability

SLP provides these capabilities using modern cloud-native technologies.

Current Project Status

Completed Components

  • Terraform
  • VPC
  • IAM
  • Amazon ECR
  • Amazon EKS
  • Docker
  • GitHub Actions
  • Helm
  • ArgoCD
  • Prometheus
  • Grafana
  • Trivy

Upcoming Enhancements

  • External Secrets
  • Multi-environment deployment
  • Blue/Green Deployment
  • Canary Releases
  • Autoscaling
  • OpenTelemetry
  • Loki
  • Alertmanager

Prerequisites

Before deploying the Secure LLMOps Platform, ensure the following tools are installed.

Operating System

Ubuntu 22.04 LTS (Recommended)

The platform can also run on:

  • Ubuntu
  • Debian
  • macOS
  • Windows (WSL2)

Required Software

Terraform >= 1.8

AWS CLI v2

kubectl

Docker

Helm

Git

Python 3.12

AWS Services

The project uses the following AWS services.

  • Amazon VPC
  • Amazon EKS
  • Amazon ECR
  • Amazon IAM
  • Amazon S3
  • Amazon DynamoDB

Installation

Clone the repository.

git clone https://github.com/Debasish-87/SLP.git

cd SLP

AWS Authentication

Configure AWS CLI.

aws configure

Provide

AWS Access Key ID

AWS Secret Access Key

Region

Output Format

Verify

aws sts get-caller-identity

GitHub Secrets

The CI/CD pipeline requires the following repository secrets.

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_REGION

These secrets are used by GitHub Actions to authenticate with AWS and push container images into Amazon ECR.

Terraform Bootstrap

Terraform state is stored remotely.

Bootstrap creates

  • Amazon S3 Bucket
  • DynamoDB Lock Table

Move into bootstrap directory.

cd terraform/bootstrap

Initialize Terraform.

terraform init

Create backend resources.

terraform apply

Example

Terraform

↓

Amazon S3

↓

Terraform State

↓

DynamoDB

↓

State Locking

Infrastructure Deployment

Navigate to the development environment.

cd terraform/environments/dev

Initialize Terraform.

terraform init

Validate configuration.

terraform validate

Review execution plan.

terraform plan

Provision infrastructure.

terraform apply

Terraform provisions

  • VPC
  • Public Subnets
  • Private Subnets
  • Internet Gateway
  • NAT Gateway
  • Route Tables
  • IAM
  • Amazon ECR
  • Amazon EKS

Terraform Project Structure

terraform

├── bootstrap

├── environments

│   ├── dev

│   ├── staging

│   └── prod

└── modules

    ├── vpc

    ├── iam

    ├── ecr

    └── eks

Configure kubectl

Retrieve cluster credentials.

aws eks update-kubeconfig \
--region ap-south-1 \
--name slp-dev

Verify cluster.

kubectl get nodes

Expected output

NAME

STATUS

ROLES

AGE

VERSION

Docker

Build application image.

docker build \
-f docker/Dockerfile \
-t eia:latest .

Run locally.

docker run -p 8000:8000 eia:latest

Verify

http://localhost:8000

Amazon ECR

Authenticate Docker.

aws ecr get-login-password \
| docker login \
--username AWS \
--password-stdin \
<ACCOUNT_ID>.dkr.ecr.ap-south-1.amazonaws.com

Push image.

docker push \
<ACCOUNT_ID>.dkr.ecr.ap-south-1.amazonaws.com/eia:latest

GitHub Actions

Every push to the main branch automatically triggers the CI/CD pipeline.

Pipeline stages

Checkout Repository

↓

Install Dependencies

↓

Execute Tests

↓

Build Docker Image

↓

Trivy Security Scan

↓

Push Image

↓

Deploy Workflow

Kubernetes

Verify cluster.

kubectl get nodes

kubectl get pods -A

Verify namespaces.

kubectl get ns

Helm

Validate chart.

helm lint helm/eia

Render manifests.

helm template eia helm/eia

Install chart.

helm install eia helm/eia

Upgrade release.

helm upgrade eia helm/eia

Uninstall.

helm uninstall eia

Helm Resources

The Helm chart creates

  • Deployment

  • Service

  • ServiceAccount

  • Ingress

ArgoCD Installation

Install namespace.

kubectl create namespace argocd

Install ArgoCD.

kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify.

kubectl get pods -n argocd

ArgoCD Login

Forward the service.

kubectl port-forward svc/argocd-server \
-n argocd \
8080:443

Open

https://localhost:8080

Retrieve admin password.

kubectl -n argocd \
get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" \
| base64 -d

Username

admin

GitOps Workflow

Developer

↓

Git Push

↓

GitHub Actions

↓

Docker Build

↓

Amazon ECR

↓

Update Helm Values

↓

Git Commit

↓

ArgoCD

↓

Automatic Deployment

↓

Amazon EKS

Deployment Verification

Verify Deployment.

kubectl get deployments

Verify Pods.

kubectl get pods

Verify Services.

kubectl get svc

Verify Ingress.

kubectl get ingress

Bootstrap Script

The platform provides a bootstrap script.

make bootstrap

or

./scripts/bootstrap.sh

This command performs

  • Terraform Initialization

  • Infrastructure Provisioning

  • Kubernetes Configuration

  • ArgoCD Installation

  • Monitoring Installation

Monitoring

SLP provides monitoring using Prometheus and Grafana.

Monitoring enables engineers to observe cluster health, application health, resource utilization, and Kubernetes workloads.

Prometheus

Prometheus collects metrics from Kubernetes resources.

Collected metrics include:

  • Node CPU Usage
  • Node Memory Usage
  • Pod CPU Usage
  • Pod Memory Usage
  • Deployment Status
  • Replica Status
  • Container Restart Count
  • Kubernetes API Metrics

Install Prometheus

./monitoring/prometheus/install.sh

Verify installation.

kubectl get pods -n monitoring

ServiceMonitor

The platform includes a ServiceMonitor resource.

This automatically instructs Prometheus to scrape metrics from the deployed application.

Example flow

Application

↓

Service

↓

ServiceMonitor

↓

Prometheus

↓

Metrics Database

Grafana

Grafana is responsible for visualizing metrics collected by Prometheus.

Current deployment provides:

  • Persistent Storage
  • Dedicated Namespace
  • Configurable Resources

Install Grafana

./monitoring/grafana/install.sh

Verify installation.

kubectl get pods -n monitoring

Access Grafana

Forward service.

kubectl port-forward svc/grafana \
3000:80 \
-n monitoring

Open

http://localhost:3000

Default Credentials

Username

admin

Password

admin123

Production deployments should store credentials using Kubernetes Secrets or AWS Secrets Manager.

Metrics Pipeline

Application

↓

Service

↓

Prometheus

↓

Time Series Database

↓

Grafana Dashboard

Security

Security scanning is integrated into the CI pipeline.

Every Docker image is scanned before deployment.

Trivy

Trivy performs vulnerability scanning.

Scan includes

  • Operating System packages

  • Python packages

  • Known CVEs

  • High Severity Vulnerabilities

  • Critical Vulnerabilities

Security Pipeline

Docker Build

↓

Trivy Scan

↓

Pass

↓

Push Amazon ECR

If vulnerabilities exceed the configured threshold, the build fails automatically.

Amazon ECR Security

Amazon ECR also performs image scanning.

This provides an additional layer of protection beyond Trivy.

IAM

The platform uses dedicated IAM roles.

Cluster Role

Worker Node Role

AmazonEKSClusterPolicy

AmazonEKSWorkerNodePolicy

AmazonEKS_CNI_Policy

AmazonEC2ContainerRegistryReadOnly

Least-privilege permissions should be used in production.

Makefile

SLP includes a Makefile to simplify daily operations.

Available commands

make bootstrap

Bootstrap the complete platform.

make build

Build Docker image.

make deploy

Deploy application.

make monitor

Install monitoring stack.

make destroy

Destroy infrastructure.

make lint

Validate Helm chart.

make template

Render Kubernetes manifests.

Scripts

The scripts directory automates common platform tasks.

scripts/

bootstrap.sh

build.sh

deploy.sh

destroy.sh

bootstrap.sh

Responsibilities

  • Terraform Initialization
  • Infrastructure Provisioning
  • Kubernetes Configuration
  • ArgoCD Installation
  • Monitoring Installation

build.sh

Responsibilities

  • Build Docker Image

deploy.sh

Responsibilities

  • Deploy Helm Chart
  • Verify Kubernetes Deployment

destroy.sh

Responsibilities

  • Remove Monitoring
  • Remove ArgoCD
  • Destroy AWS Infrastructure

Deployment Workflow

The complete operational workflow is shown below.

Developer

↓

Git Push

↓

GitHub Actions

↓

Install Dependencies

↓

Run Tests

↓

Docker Build

↓

Security Scan

↓

Amazon ECR

↓

Update Helm Values

↓

Git Commit

↓

ArgoCD

↓

Helm Release

↓

Deployment

↓

Running Pods

↓

Prometheus

↓

Grafana

Kubernetes Resources

The platform deploys the following resources.

Deployment

Service

ServiceAccount

Ingress

ReplicaSet

Pods

Scaling

Scaling is handled through Amazon EKS Node Groups.

Application replicas are managed using Kubernetes Deployments.

Example

kubectl scale deployment eia \
--replicas=5

Rollout

Check rollout.

kubectl rollout status deployment/eia

Restart rollout.

kubectl rollout restart deployment/eia

Rollback.

kubectl rollout undo deployment/eia

Useful Commands

List Nodes

kubectl get nodes

List Pods

kubectl get pods

List Services

kubectl get svc

List Deployments

kubectl get deploy

List Namespaces

kubectl get ns

Logging

Application logs.

kubectl logs <pod-name>

Stream logs.

kubectl logs -f <pod-name>

Health Checks

Deployment uses

  • Startup Probe

  • Readiness Probe

  • Liveness Probe

These ensure that unhealthy containers are restarted automatically.

Current Capabilities

Infrastructure Provisioning

Containerization

Continuous Integration

Continuous Deployment

GitOps

Container Registry

Kubernetes Orchestration

Monitoring

Security Scanning

Infrastructure Automation

Troubleshooting

This section covers common issues that may occur during deployment and their resolutions.

Terraform Initialization Failed

Error

Backend configuration changed

Resolution

terraform init -reconfigure

AWS Authentication Failed

Error

Unable to locate credentials

Resolution

Verify AWS credentials.

aws configure

or verify GitHub repository secrets.

Required secrets:

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_REGION

Docker Build Failed

Resolution

Verify Docker installation.

docker version

Verify Docker daemon.

docker ps

Kubernetes Cluster Unreachable

Verify kubeconfig.

kubectl config current-context

Update kubeconfig.

aws eks update-kubeconfig \
--region ap-south-1 \
--name slp-dev

Helm Installation Failed

Validate chart.

helm lint helm/eia

Render manifests.

helm template eia helm/eia

ArgoCD Not Synchronizing

Verify application.

kubectl get applications -n argocd

Force synchronization.

argocd app sync eia

Prometheus Not Collecting Metrics

Verify ServiceMonitor.

kubectl get servicemonitor -A

Verify Prometheus Pods.

kubectl get pods -n monitoring

Grafana Not Accessible

Check service.

kubectl get svc -n monitoring

Forward service.

kubectl port-forward svc/grafana \
3000:80 \
-n monitoring

Best Practices

The following practices are recommended for production deployments.

Infrastructure

  • Store Terraform state remotely.
  • Enable state locking.
  • Use separate AWS accounts for production.
  • Enable AWS CloudTrail.
  • Use least-privilege IAM policies.

Kubernetes

  • Use dedicated namespaces.
  • Configure Network Policies.
  • Define Resource Requests and Limits.
  • Configure Pod Disruption Budgets.
  • Enable Horizontal Pod Autoscaler.

CI/CD

  • Protect the main branch.
  • Require pull request reviews.
  • Execute automated testing.
  • Perform container image scanning.
  • Use immutable image tags.

GitOps

  • Never deploy using kubectl apply in production.
  • Treat Git as the single source of truth.
  • Review deployment changes through pull requests.
  • Enable automatic synchronization only after validation.

Monitoring

Monitor at minimum:

  • Node CPU
  • Node Memory
  • Pod CPU
  • Pod Memory
  • Pod Restarts
  • Deployment Availability
  • Cluster Health

Security

  • Rotate credentials regularly.
  • Use AWS Secrets Manager.
  • Enable image scanning.
  • Keep dependencies updated.
  • Apply security patches promptly.

Production Considerations

The current implementation demonstrates a production-oriented architecture.

Additional enterprise capabilities that can be introduced include:

  • Blue/Green Deployments
  • Canary Releases
  • Horizontal Pod Autoscaler
  • Vertical Pod Autoscaler
  • Cluster Autoscaler
  • AWS Load Balancer Controller
  • External DNS
  • Cert Manager
  • OpenTelemetry
  • Loki
  • Tempo
  • Alertmanager
  • Karpenter
  • Velero Backup
  • External Secrets Operator

Future Roadmap

Planned enhancements include:

Platform

  • Multi-Environment Deployment
  • Multi-Cluster Support
  • Multi-Region Deployment

Security

  • External Secrets
  • AWS Secrets Manager
  • HashiCorp Vault
  • OPA Gatekeeper
  • Kyverno Policies

Observability

  • OpenTelemetry
  • Loki
  • Tempo
  • Jaeger
  • Distributed Tracing

Kubernetes

  • Autoscaling
  • Canary Deployment
  • Blue/Green Deployment
  • Progressive Delivery

CI/CD

  • Release Versioning
  • Automated Rollback
  • Semantic Versioning
  • Release Notes Generation

Contributing

Contributions are welcome.

Recommended workflow:

  1. Fork the repository.
  2. Create a feature branch.
  3. Commit changes.
  4. Push the branch.
  5. Open a Pull Request.

Ensure all tests pass before submitting changes.

License

This project is released under the MIT License.

Refer to the LICENSE file for complete licensing information.

Project Summary

Secure LLMOps Platform demonstrates the implementation of a modern cloud-native deployment platform for AI applications.

The project combines Infrastructure as Code, Kubernetes, GitOps, Continuous Integration, Continuous Deployment, Monitoring, and Security into a single deployment platform.

Primary capabilities include:

  • AWS Infrastructure Provisioning
  • Terraform Automation
  • Docker Containerization
  • Amazon ECR
  • Amazon EKS
  • GitHub Actions
  • Helm
  • ArgoCD
  • GitOps
  • Prometheus
  • Grafana
  • Trivy Security Scanning

The platform is designed to automate the deployment lifecycle of AI services while following modern DevOps and Platform Engineering practices.

Skills Demonstrated

This project demonstrates practical experience with:

Cloud

  • Amazon Web Services

Infrastructure

  • Terraform

Containers

  • Docker

Container Registry

  • Amazon Elastic Container Registry

Container Orchestration

  • Kubernetes
  • Amazon EKS

CI/CD

  • GitHub Actions

GitOps

  • ArgoCD

Package Management

  • Helm

Monitoring

  • Prometheus
  • Grafana

Security

  • Trivy

Platform Engineering

  • Infrastructure Automation
  • Deployment Automation
  • Kubernetes Operations
  • GitOps Workflows
  • Cloud Infrastructure Management

Author

Debasish

Secure LLMOps Platform (SLP)

Production-Style LLMOps Platform for Deploying and Managing AI Applications on AWS.

About

Production-ready LLMOps platform for deploying and managing AI applications on AWS using Terraform, Kubernetes, Helm, ArgoCD, GitHub Actions, Prometheus, and Grafana.

Topics

docker kubernetes aws devops cloud ai terraform helm grafana prometheus infrastructure-as-code ecr gitops platform-engineering eks argocd github-actions fastapi trivy llmops

Resources

Readme

License

View license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages