Merge pull request #133 from Sakeeb91/feature/docker-engine

Add Docker init and runtime support for engine

Author: ziagham

ISSUE-110-IMPLEMENTATION-PLAN.md

@@ -0,0 +1,105 @@
+# Implementation Plan – Issue #110 (Dockerize FlowSynx Engine)
+
+**Scope & Answers Incorporated**
+- **Component scope:** Engine only for now. Console support is deferred; add later once config needs are clear.
+- **Registry & tags:** Public images on Docker Hub (`docker.io/flowsynx/flowsynx`). Tags follow semantic versioning (e.g., `1.2.4-linux-amd64`, `1.2.4-windows-ltsc2022-amd64`).
+- **Ports:** Engine listens on **6262** (Console would be 6264 when added).
+- **Config/storage:** Persist install mode and Docker metadata alongside `flowctl` in an `appsettings.json` file (same directory as the CLI).
+- **Persistence (recommended):** Default to a host bind mount `~/.flowsynx/data -> /app/data` so workflows, logs/telemetry, and plugins survive container restarts. Allow overrides for power users (named volumes or custom paths).
+- **Testing expectation:** Unit tests only for the Docker pathway.
+
+---
+
+## Goals
+- Add Docker-based initialization and run support for the FlowSynx engine via an explicit flag.
+- Keep binary flow as the default and fully functional.
+- Provide friendly UX when Docker is absent or not running and offer fallback guidance to binary mode.
+
+---
+
+## Architecture & Key Decisions
+- **Docker client approach:** Use the existing process wrapper to shell out to the Docker CLI (simpler dependency story). Consider Docker.DotNet later if needed.
+- **Configuration:** `appsettings.json` (sibling to `flowctl`) stores deployment mode and Docker artifact metadata.
+- **Persistence:** Single host bind by default: mount `~/.flowsynx/data` to `/app/data`. Inside `/app/data`, keep subfolders for state/db, logs/telemetry, and plugins. This keeps user data portable and inspectable. CLI flags allow overriding to a named volume or a different host path.
+- **Mode selection:** Binary remains default. Docker requires `--docker` (or `--use-docker`); we record the last successful mode in config to provide actionable hints.
+
+---
+
+## Workplan
+
+### 1) Docker Service Abstraction
+- Add `IDockerService` in Core and a CLI-based implementation in Infrastructure that can:
+  - Check availability of Docker daemon.
+  - Pull images with progress feedback.
+  - Create/run/stop/remove containers.
+  - Inspect container status and fetch logs (tail).
+- Keep surface area focused on engine needs (no console operations yet).
+
+### 2) Configuration & State
+- Add/load `appsettings.json` beside the `flowctl` binary to track:
+  - `deploymentMode`: `binary` or `docker`.
+  - Docker details: image name, tag, container name/id, mapped host port, and host mount path used.
+  - Last successful run mode to shape UX hints.
+- Provide a small config service to read/update this file safely.
+
+### 3) Init Command (`flowctl init --docker`)
+- Add `--docker` flag (explicit opt-in).
+- Options:
+  - `--flowsynx-version` (tag), defaults to latest tag lookup if omitted.
+  - `--container-name` (default: `flowsynx-engine`).
+  - `--port` (host) default: 6262.
+  - `--mount` (hostPath:containerPath) with default bind `~/.flowsynx/data:/app/data`.
+  - `--platform` auto-detected to choose the right arch tag (linux-amd64, windows-ltsc2022-amd64).
+- Flow:
+  1. Check Docker availability; emit friendly guidance if missing/stopped and suggest binary fallback.
+  2. Resolve tag (use provided tag or fetch latest), pick platform-specific image.
+  3. Pull image.
+  4. Create container with port mapping 6262 and bind mount `~/.flowsynx/data:/app/data` (unless overridden).
+  5. Start container and wait for the engine to report healthy/up.
+  6. Persist Docker metadata + mode in `appsettings.json`.
+  7. Output connection info and how to run/stop.
+
+### 4) Run Command (`flowctl run --docker`)
+- Detect mode from `appsettings.json`; require `--docker` to enter Docker path.
+- Behavior:
+  - If container exists, start it (background by default).
+  - If missing, recreate using stored image/tag/port/mount.
+  - `--background` keeps container detached; without it, attach logs and handle Ctrl+C gracefully.
+- Friendly errors when Docker is unavailable; suggest binary run when appropriate.
+
+### 5) Stop/Remove Helpers (Scoped to Engine)
+- `flowctl stop --docker`: stop the engine container if running.
+- `flowctl uninstall --docker`: remove container; prompt before removing the host data directory if requested (default is to leave `~/.flowsynx/data` intact).
+
+---
+
+## Persistence Details (Best Course)
+- **Default:** Bind mount `~/.flowsynx/data` to `/app/data` to persist workflow state/db, logs/telemetry, and plugins in one place that users can inspect and back up.
+- **Why:** Easy to reason about, works across platforms, no hidden Docker volumes, and aligns with existing FlowSynx data expectations.
+- **Overrides:** Allow users to:
+  - Supply a different host path via `--mount hostPath:/app/data`.
+  - Opt into a named volume via `--mount flowsynx-data:/app/data` if they prefer Docker-managed storage.
+- Document permissions considerations for Linux/macOS when binding host paths.
+
+---
+
+## Testing
+- Unit tests for Docker service (availability, pull/create/run/stop flows mocked).
+- Unit tests for init/run option parsing and config persistence.
+- No integration tests required for this scope.
+
+---
+
+## Risks & Mitigations
+- Docker not installed or daemon down → clear error + link to install; suggest binary fallback.
+- Port 6262 conflict → detect before run/create and allow `--port` override.
+- Permission issues on host binds → document chmod/chown guidance; allow named volume override.
+- Tag/platform mismatch → auto-detect OS/arch to choose the correct tag suffix, surface a clear error if unsupported.
+
+---
+
+## Future (Out of Scope Here)
+- Add Console container support (port 6264) once its config story is defined.
+- Broader Docker commands (logs/status/exec) and integration tests if needed.
+- Registry auth/PAT handling if images ever become private.
+

MERGE-NOTE-PR-133.md

@@ -0,0 +1,22 @@
+# Merge Conflict Note – PR #133 (feature/docker-engine vs master)
+
+There is a single conflict when bringing `origin/master` into `feature/docker-engine`:
+
+- **File:** `src/FlowCtl/Commands/Run/RunCommandOptionsHandler.cs`
+- **Cause:** `feature/docker-engine` adds the Docker run path and settings persistence; `master` added XML docs/cleanup around `GetArgumentStr` (and nearby helpers). Git could not auto-merge those overlapping edits.
+
+## How to resolve
+
+1. Keep the Docker logic from `feature/docker-engine`:
+   - The `RunDockerAsync` method with Docker pull/run/start and settings persistence.
+   - The platform/tag resolution helpers and the Docker mode hinting.
+2. Keep the upstream doc/comment/formatting improvements from `master`:
+   - The XML documentation and tightened argument construction around `GetArgumentStr` (and related small cleanups).
+3. Resulting file should have:
+   - The full Docker pathway (pull/create/start, attach logs when not background).
+   - The `GetArgumentStr` with upstream doc comment.
+   - No conflict markers.
+
+## Notes
+- Do the merge in a clean worktree to avoid touching unrelated local changes (e.g., ConsoleLogger removals, JsonSerializer tweaks).
+- After resolving, rerun `dotnet test` and push the updated branch before re-running the PR checks.

README.md

@@ -148,10 +148,32 @@ This will output version information in JSON format, similar to the example belo
 
 > This command is useful for confirming compatibility and ensuring you're using the intended versions.
 
+### Initialize FlowSynx in Docker Mode
+Docker mode runs only the FlowSynx engine inside a container and persists data to your host. Docker must be installed and running.
+
+```
+flowctl init --docker --container-name flowsynx-engine --port 6262 --mount ~/.flowsynx/data --container-path /app/data
+```
+
+- Images are pulled from Docker Hub (`flowsynx/flowsynx`) using platform-specific tags (`linux-amd64` or `windows-ltsc2022-amd64`).
+- Data (workflows, logs, plugins) is stored on the host path you provide (default `~/.flowsynx/data`).
+- Install mode and container details are stored in `appsettings.json` next to the `flowctl` binary.
+
+To start the engine in Docker mode:
+```
+flowctl run --docker --background
+```
+
+To stop or remove the container:
+```
+flowctl stop --docker
+flowctl uninstall --docker [--remove-data]
+```
+
 ### Uninstalling FlowSynx (Standalone Mode)
 To uninstall FlowSynx in standalone mode, run the following command:
 ```
 flowctl uninstall
 ```
 This will remove the FlowSynx engine binary, and the default installation directory that was created during flowctl init.
-> ⚠️ This operation is irreversible and will delete all local FlowSynx files associated with the standalone setup.
+> ⚠️ This operation is irreversible and will delete all local FlowSynx files associated with the standalone setup.

src/FlowCtl.Core/Models/Configuration/AppSettings.cs

@@ -0,0 +1,29 @@
+namespace FlowCtl.Core.Models.Configuration;
+
+public class AppSettings
+{
+    public DeploymentMode DeploymentMode { get; set; } = DeploymentMode.Binary;
+    public DockerSettings Docker { get; set; } = new();
+    public BinarySettings Binary { get; set; } = new();
+}
+
+public class DockerSettings
+{
+    public string ImageName { get; set; } = string.Empty;
+    public string Tag { get; set; } = string.Empty;
+    public string ContainerName { get; set; } = string.Empty;
+    public int Port { get; set; } = 6262;
+    public string HostDataPath { get; set; } = string.Empty;
+    public string ContainerDataPath { get; set; } = "/app/data";
+}
+
+public class BinarySettings
+{
+    public string? Version { get; set; }
+}
+
+public enum DeploymentMode
+{
+    Binary,
+    Docker
+}

src/FlowCtl.Core/Models/Docker/DockerCommandResult.cs

@@ -0,0 +1,10 @@
+namespace FlowCtl.Core.Models.Docker;
+
+public class DockerCommandResult
+{
+    public int ExitCode { get; set; }
+    public string Output { get; set; } = string.Empty;
+    public string Error { get; set; } = string.Empty;
+
+    public bool Success => ExitCode == 0;
+}

src/FlowCtl.Core/Models/Docker/DockerRunOptions.cs

@@ -0,0 +1,13 @@
+namespace FlowCtl.Core.Models.Docker;
+
+public class DockerRunOptions
+{
+    public string ImageName { get; set; } = string.Empty;
+    public string Tag { get; set; } = string.Empty;
+    public string ContainerName { get; set; } = string.Empty;
+    public int HostPort { get; set; }
+    public int ContainerPort { get; set; } = 6262;
+    public string HostDataPath { get; set; } = string.Empty;
+    public string ContainerDataPath { get; set; } = string.Empty;
+    public bool Detached { get; set; } = true;
+}

src/FlowCtl.Core/Services/Configuration/IAppSettingsService.cs

@@ -0,0 +1,21 @@
+using FlowCtl.Core.Models.Configuration;
+
+namespace FlowCtl.Core.Services.Configuration;
+
+public interface IAppSettingsService
+{
+    /// <summary>
+    /// Reads application settings from disk or returns defaults when the file is missing or invalid.
+    /// </summary>
+    Task<AppSettings> LoadAsync(CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Persists application settings to disk.
+    /// </summary>
+    Task SaveAsync(AppSettings settings, CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Returns the expected file path for the settings file.
+    /// </summary>
+    string GetSettingsPath();
+}

src/FlowCtl.Core/Services/Docker/IDockerService.cs

@@ -0,0 +1,16 @@
+using FlowCtl.Core.Models.Docker;
+
+namespace FlowCtl.Core.Services.Docker;
+
+public interface IDockerService
+{
+    Task<bool> IsDockerAvailableAsync(CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> PullImageAsync(string imageName, string tag, CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> RunContainerAsync(DockerRunOptions options, CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> StartContainerAsync(string containerName, CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> StopContainerAsync(string containerName, CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> RemoveContainerAsync(string containerName, bool force, CancellationToken cancellationToken = default);
+    Task<bool> ContainerExistsAsync(string containerName, CancellationToken cancellationToken = default);
+    Task<bool> IsContainerRunningAsync(string containerName, CancellationToken cancellationToken = default);
+    Task<DockerCommandResult> TailLogsAsync(string containerName, CancellationToken cancellationToken = default);
+}

src/FlowCtl.Infrastructure/Extensions/ServiceCollectionExtensions.cs

@@ -1,8 +1,12 @@
-using FlowCtl.Core.Services.Authentication;
+using FlowCtl.Core.Services.Authentication;
+using FlowCtl.Core.Services.Configuration;
+using FlowCtl.Core.Services.Docker;
 using FlowCtl.Core.Services.Extractor;
 using FlowCtl.Core.Services.Github;
 using FlowCtl.Core.Services.ProcessHost;
 using FlowCtl.Infrastructure.Services.Authentication;
+using FlowCtl.Infrastructure.Services.Configuration;
+using FlowCtl.Infrastructure.Services.Docker;
 using FlowCtl.Infrastructure.Services.Extractor;
 using FlowCtl.Infrastructure.Services.Github;
 using FlowCtl.Infrastructure.Services.ProcessHost;
@@ -23,8 +27,10 @@ public static IServiceCollection AddInfrastructure(this IServiceCollection servi
                 .AddScoped<IProcessHandler, ProcessHandler>()
                 .AddScoped<IArchiveExtractor, ArchiveExtractor>()
                 .AddScoped<IDataProtectorWrapper, DataProtectorWrapper>()
-                .AddScoped<IAuthenticationManager, AuthenticationManager>();
+                .AddScoped<IAuthenticationManager, AuthenticationManager>()
+                .AddScoped<IAppSettingsService, AppSettingsService>()
+                .AddScoped<IDockerService, DockerService>();
 
         return services;
     }
-}
+}

src/FlowCtl.Infrastructure/Services/Configuration/AppSettingsService.cs

@@ -0,0 +1,88 @@
+using FlowCtl.Core.Models.Configuration;
+using FlowCtl.Core.Serialization;
+using FlowCtl.Core.Services.Configuration;
+using FlowCtl.Core.Services.Location;
+
+namespace FlowCtl.Infrastructure.Services.Configuration;
+
+public class AppSettingsService : IAppSettingsService
+{
+    private const string SettingsFileName = "appsettings.json";
+
+    private readonly ILocation _location;
+    private readonly IJsonSerializer _serializer;
+    private readonly IJsonDeserializer _deserializer;
+
+    public AppSettingsService(ILocation location, IJsonSerializer serializer, IJsonDeserializer deserializer)
+    {
+        _location = location ?? throw new ArgumentNullException(nameof(location));
+        _serializer = serializer ?? throw new ArgumentNullException(nameof(serializer));
+        _deserializer = deserializer ?? throw new ArgumentNullException(nameof(deserializer));
+    }
+
+    public async Task<AppSettings> LoadAsync(CancellationToken cancellationToken = default)
+    {
+        var settingsPath = GetSettingsPath();
+        if (!File.Exists(settingsPath))
+            return CreateDefaultSettings();
+
+        try
+        {
+            var content = await File.ReadAllTextAsync(settingsPath, cancellationToken);
+            if (string.IsNullOrWhiteSpace(content))
+                return CreateDefaultSettings();
+
+            var settings = _deserializer.Deserialize<AppSettings>(content);
+            Normalize(settings);
+            return settings;
+        }
+        catch
+        {
+            return CreateDefaultSettings();
+        }
+    }
+
+    public async Task SaveAsync(AppSettings settings, CancellationToken cancellationToken = default)
+    {
+        if (settings == null)
+            throw new ArgumentNullException(nameof(settings));
+
+        var settingsPath = GetSettingsPath();
+        var settingsDirectory = Path.GetDirectoryName(settingsPath);
+        if (!string.IsNullOrEmpty(settingsDirectory))
+            Directory.CreateDirectory(settingsDirectory);
+
+        var formatted = _serializer.Serialize(settings, new JsonSerializationConfiguration { Indented = true });
+        await File.WriteAllTextAsync(settingsPath, formatted, cancellationToken);
+    }
+
+    public string GetSettingsPath()
+    {
+        return Path.Combine(_location.RootLocation, SettingsFileName);
+    }
+
+    private AppSettings CreateDefaultSettings()
+    {
+        var defaultHostPath = Path.Combine(_location.DefaultFlowSynxDirectoryName, "data");
+        return new AppSettings
+        {
+            DeploymentMode = DeploymentMode.Binary,
+            Docker = new DockerSettings
+            {
+                ImageName = "flowsynx/flowsynx",
+                Tag = string.Empty,
+                ContainerName = "flowsynx-engine",
+                Port = 6262,
+                HostDataPath = defaultHostPath,
+                ContainerDataPath = "/app/data"
+            },
+            Binary = new BinarySettings()
+        };
+    }
+
+    private static void Normalize(AppSettings settings)
+    {
+        settings.Docker ??= new DockerSettings();
+        settings.Binary ??= new BinarySettings();
+    }
+}