feat: Add Data Lakehouse Architecture scope document

Core DW Infrastructure/docs/zabbix/01-getting-started.md

@@ -0,0 +1,97 @@
+---
+title: Zabbix Getting Started for Redback
+description: Introduction and setup guide for Zabbix in the Redback environment
+---
+
+
+1. Purpose of Zabbix in the Redback Platform
+Zabbix is used within the Redback Data Warehouse environment as the central monitoring and observability platform. Its primary goals are to:
+•	Monitor infrastructure health (VMs, containers, databases)
+•	Detect outages, failures, and resource exhaustion early
+•	Provide alerting and visibility for operational teams
+•	Serve as a learning and handover platform for future capstone trimesters
+Zabbix complements existing logging tools by focusing on metrics, availability, and alerts, rather than raw logs.
+
+2. High-Level Architecture
+Zabbix in Redback follows a containerized architecture, deployed using Docker Compose.
+Core Components
+
+•	Zabbix Server
+The Zabbix Server is the center of the entire Zabbix monitoring system. It is responsible for:
+	Data collection gathers metrics from agents, proxies, and remote checks.
+	Trigger evaluation analyzes incoming data against rules (triggers) to detect anomalies or issues.
+	Event and alert generation create actionable events and send notifications when thresholds are exceeded.
+	Central repository stores operational, statistical, and configuration data in the database for long-term use. 
+All major monitoring decisions are made here: when a metric exceeds a threshold, when to create an event, and when to notify users. The server can also perform agent-less checks (e.g., simple network checks like HTTP or ICMP) and handles data from all agents and proxies. 
+A key part of the server’s operation is how it interacts with the database; configuration and metrics are stored there, and the server periodically reads database tables to update its cache of active items. This is why changes made in the web interface (e.g., adding a host or template) may take a short time (e.g., up to a couple of minutes) to reflect in active monitoring.
+
+•	Zabbix Database (PostgreSQL in Redback )
+The Database Storage component is where all Zabbix configuration, monitoring data, triggers, history, and trends are stored. Zabbix supports multiple databases, including PostgreSQL, MySQL, Oracle, and others. In the Redback environment, PostgreSQL is used. 
+This component is critical because:
+	The server reads and writes data here constantly.
+	Configuration defined in the web interface (like hosts, templates, triggers) is stored here.
+	Historical metrics (for graphs, reports, and trend analysis) are preserved here.
+Without a working database connection, Zabbix cannot initialize, as it has nowhere to store collected metrics or read configuration, which was the root cause of the initial setup issues when the database could not write due to disk exhaustion
+
+•	Zabbix Web
+The Web Interface is the user-facing part of Zabbix, built on PHP and typically served by a web server such as Nginx or Apache. It provides:
+	Browser-based access to monitoring data and dashboards
+	Configuration tools for hosts, templates, triggers, actions, and user management
+	Reporting, graphing, and visualization of collected data
+The frontend interacts with the database to read and write configuration changes, and to display metrics stored by the server. It is separated from the server process but depends on the same database.
+
+
+•	Zabbix Agents
+The Zabbix Agent is lightweight software installed on individual monitored systems (e.g., VMs, servers). Its main role is to:
+•	Collect detailed local metrics: CPU, memory, disk usage, processes, services, etc.
+•	Report these metrics to the Zabbix Server (either passively or actively)
+	Passive mode: server polls agent when needed
+	Active mode: agent pushes data to the server at configured intervals 
+Agents use native system calls for efficiency and can detect low-level system conditions (like a full disk or failed process) and help trigger alerts. Without agents, you are limited to agent-less monitoring (e.g., SNMP, simple network checks), which is less detailed.
+
+•	Zabbix Proxy (not currently deployed in Redback but useful for scale)
+A Zabbix Proxy is an optional component used to collect and buffer monitoring data from remote sites or large environments, then send it in batches to the Zabbix server. Proxies help to:
+Reduce load on the central server
+Enable monitoring of remote or isolated networks
+Collect data locally and forward it efficiently 
+While not required for a basic setup, proxies are useful in distributed infrastructures where direct server connectivity may be limited or resource intensive.
+
+3. Logical Flow of Data
+    1.	Zabbix Agents and SNMP devices send metrics to the Zabbix Server
+    2.	Zabbix Server writes data into PostgreSQL
+    3.	Triggers evaluate incoming data
+    4.	Events are generated if thresholds are crossed
+    5.	Notifications are sent via configured media types
+    6.	Data is visualised in dashboards or external tools (e.g. Grafana)
+
+4. Access Assumptions
+This documentation assumes the following:
+Infrastructure Access
+•	Access to the GPU VM (or equivalent Redback infrastructure VM)
+•	Docker and Docker Compose installed
+•	Permission to start/stop containers
+Networking
+•	Containers communicate over a shared Docker network
+•	Zabbix Server can resolve:
+o	PostgreSQL container hostname
+o	Agent hostnames
+•	Required ports are open internally (e.g. 5432, 10051, 10050)
+Credentials
+•	Zabbix admin credentials available
+•	PostgreSQL user and database credentials provided via .env
+
+5. Initial Access to Zabbix
+•	Identify the Zabbix Web container port mapping
+•	Access via browser:
+http://<VM-IP>:<ZABBIX_WEB_PORT>
+•	Log in using the admin credentials defined during setup
+
+
+6. Where to Go Next
+After initial access, continue with the following guides:
+•	How to Onboard a New Host
+•	User Groups and Permissions
+•	Setting up MFA
+•	Monitoring Templates (Agent & SNMP)
+•	Troubleshooting Common Zabbix/Postgres Issues
+

Core DW Infrastructure/docs/zabbix/02-troubleshooting.md

@@ -0,0 +1,69 @@
+---
+title: Troubleshooting Zabbix & PostgreSQL
+description: Common failure scenarios and safe remediation steps
+---
+
+Troubleshooting Common Zabbix/Postgres Issues
+This document covers the most common failure scenarios encountered in the Redback Zabbix monitoring environment, particularly when running Zabbix and PostgreSQL in containers. It focuses on symptoms, root causes, and safe remediation steps, with an emphasis on what must NOT be deleted.
+
+1. Disk Full Symptoms (Most Common Failure)
+One of the most frequent causes of Zabbix failure in Redback is disk exhaustion on the VM.
+Common Symptoms
+•	Zabbix Server stuck on:
+**** PostgreSQL server is not available. Waiting 5 seconds...
+•	PostgreSQL container is running but unusable
+•	Database initialization never completes
+•	Containers report healthy but functionality is broken
+
+Safe Diagnosis Steps
+df -h
+docker system df
+docker volume ls
+docker volume inspect <volume_name>
+Check container sizes:
+docker ps -s
+
+
+2. PostgreSQL “Container Up but Unusable” Behaviour
+PostgreSQL may appear healthy while internally failing due to disk issues.
+Indicators
+•	docker ps shows Postgres as Up (healthy)
+•	psql connects but:
+o	No databases are listed
+o	No roles are listed
+•	Errors related to WAL or file writes in logs
+
+3. Zabbix Database Initialization Failures
+Zabbix requires full write access to the PostgreSQL data directory.
+What Happens When It Fails
+•	Zabbix cannot create a schema
+•	Zabbix server retries indefinitely
+•	Web UI never completes startup
+Root Causes
+•	Database container not ready before Zabbix starts
+•	Incorrect DB credentials
+•	Database not initialized properly
+Diagnosis
+docker logs zabbix-server
+docker logs postgres
+Check environment variables:
+•	POSTGRES_USER
+•	POSTGRES_PASSWORD
+•	POSTGRES_DB
+4. Safe Remediation Steps 
+Recommended Actions
+•	Identify largest containers:
+•	docker system df
+•	Inspect container log directories
+•	Truncate log files only after confirmation
+•	Implement log rotation inside affected containers
+•	Restart PostgreSQL after freeing disk space
+•	Restart Zabbix Server and Web containers
+
+5. Validation Checklist After Fix
+•	PostgreSQL accepts writes
+•	Zabbix Server initializes database
+•	Zabbix Web loads successfully
+•	Hosts and dashboards appear
+•	No repeated DB connection errors in logs
+