Implement configurable session management in HAPPE

Author: frostdev-ops

TASKS.md

@@ -189,3 +189,138 @@ This task list breaks down the implementation steps outlined in `PLANNING.md`.
 - [ ] **Configuration:** Refine configuration loading (e.g., use environment variables, config files).
 - [ ] **Logging:** Improve logging messages for clarity and debugging.
 - [ ] **Version Control:** Commit changes frequently with clear messages.
+
+## Phase 9: Enhance IDA with Broker LLM (Direct Integration)
+
+This phase implements the plan to integrate a "broker" LLM directly within IDA
+to refine memory retrieval results before sending them to HAPPE.
+
+### Sub-Phase 1: Prerequisites and Setup
+
+- [x] **Broker LLM Configuration Loading:**
+    - [x] Verify `MemoryBrokerConfig` fields (`provider`, `api_key`, `model_name`, `base_url`) are loaded correctly into `IdaConfig`.
+    - [x] Add loaded `IdaConfig` to shared `DaemonState` struct.
+    - [x] Update `DaemonState` instantiation in `ida/src/bin/ida-daemon.rs`.
+    - [x] Ensure `handle_message` in `ida/src/ipc_server.rs` can access the `IdaConfig`.
+- [x] **Ensure MCP Access Reuse in IDA:**
+    - [x] Identify where `MemoryStore` is initialized in `ida/src/bin/ida-daemon.rs`.
+    - [x] Store the `Arc<dyn McpHostInterface + Send + Sync>` used for `MemoryStore` in the `DaemonState`.
+    - [x] Update `DaemonState` definition and instantiation.
+- [x] **Integrate Broker LLM Client into IDA:**
+    - [x] **Configuration (`core/src/config.rs`):**
+        - [x] Modify `MemoryBrokerConfig` struct (add `provider`, `base_url`, etc.).
+        - [x] Update corresponding `IdaConfig` in `ida/src/config.rs` and its `From` impl.
+        - [x] Update default config generation (`install` or `tools`) for `[ida.memory_broker]`.
+    - [x] **Implementation (IDA Crate):**
+        - [x] Add dependencies (`reqwest`, `serde`, `serde_json`, `async-trait`) to `ida/Cargo.toml`.
+        - [x] Create `ida/src/llm_clients.rs`.
+        - [x] Define `trait LLMClient { async fn generate(&self, prompt: &str) -> Result<String>; }`.
+        - [x] Implement the trait for supported providers (e.g., `GeminiClient`, `OllamaClient`).
+        - [x] Create factory function `create_llm_client(config: &CoreMemoryBrokerConfig) -> Result<Option<Arc<dyn LLMClient>>>`.
+        - [x] Add `Option<Arc<dyn LLMClient>>` field to `DaemonState`.
+        - [x] Initialize the client in `ida/src/bin/ida-daemon.rs` using the factory.
+
+### Sub-Phase 2: Core Logic Implementation in IDA
+
+- [x] **Modify `ida::memory_mcp_client::retrieve_memories`:**
+    - [x] Change signature to accept `broker_llm_client: &Option<Arc<dyn LLMClient>>` and `conversation_context: Option<String>`.
+    - [x] After semantic search, if results exist and client exists:
+        - [x] Construct broker prompt (query, context, candidates, instructions).
+        - [x] Call `client.generate(&broker_prompt).await`.
+        - [x] Parse response (e.g., comma-separated keys).
+        - [x] Filter semantic search results based on broker response.
+        - [x] Implement fallback logic on broker error.
+    - [x] Update call site in `ida/src/ipc_server.rs` (`handle_message`) to pass the client and context.
+
+### Sub-Phase 3: Integrating Conversation Context (IPC Changes)
+
+- [x] **Modify IPC Message (`ipc/src/internal_messages.rs`):**
+    - [x] Add `conversation_context: Option<String>` to `GetMemoriesRequest`.
+- [x] **Update HAPPE Client (`happe/src/ida_client.rs`):**
+    - [x] Update `get_memories` signature and request construction.
+- [x] **Update HAPPE Orchestrator (`happe/src/coordinator.rs` or similar):**
+    - [x] Gather context and pass it to `ida_client::get_memories`.
+- [x] **Update IDA Server (`ida/src/ipc_server.rs`):**
+    - [x] Extract context and pass it to `memory_mcp_client::retrieve_memories`.
+
+### Sub-Phase 4: Testing and Refinement
+
+- [ ] **Testing:** Unit, Integration, End-to-End tests.
+- [ ] **Refinement:** Prompt engineering, latency analysis, fallback logic, configuration.
+
+## Phase 10: Implement Configurable Session Management in HAPPE
+
+This phase adds a stateful session management system to HAPPE, allowing it to maintain conversation history across multiple requests within a session. This is crucial for providing proper context to `IDA` for memory retrieval.
+
+### Sub-Phase 1: Define Core Session Store Trait & In-Memory Adapter
+
+- [x] **Module Setup:**
+    - [x] Create directory `happe/src/session/`.
+    - [x] Create directory `happe/src/session/adapters/`.
+    - [x] Create file `happe/src/session/mod.rs`.
+    - [x] Create file `happe/src/session/store.rs`.
+    - [x] Create file `happe/src/session/adapters/in_memory.rs`.
+    - [x] Declare `session` module in `happe/src/lib.rs` (`pub mod session;`).
+- [x] **Define `SessionStore` Trait (`happe/src/session/store.rs`):**
+    - [x] Define `#[async_trait] pub trait SessionStore: Send + Sync`.
+    - [x] Add methods for session management:
+        - [x] `create_session`
+        - [x] `get_session`
+        - [x] `save_session`
+        - [x] `delete_session`
+        - [x] `cleanup_expired_sessions`
+- [x] **Implement `InMemorySessionStore` (`happe/src/session/adapters/in_memory.rs`):**
+    - [x] Define `struct InMemorySessionStore`.
+    - [x] Add thread-safe storage using `Arc<RwLock<HashMap<String, Session>>>`.
+    - [x] Implement `SessionStore` trait.
+    - [x] Add expiration and cleanup logic.
+- [x] **Dependencies (`happe/Cargo.toml`):**
+    - [x] Add `async-trait = "0.1"`.
+    - [x] Add `uuid = { version = "1", features = ["v4"] }` (for session ID generation).
+    - [x] Ensure `tokio` features include `sync`.
+
+### Sub-Phase 2: Integrate Session Store into HAPPE State
+
+- [x] **State (`happe/src/http_server.rs`, `happe/src/ipc_server.rs`):**
+    - [x] Add `session_store: SessionStoreRef` to `AppState` struct (`http_server.rs`).
+    - [x] Add `session_store: SessionStoreRef` to `IpcServerState` struct (`ipc_server.rs`).
+    - [x] Initialize session store in HTTP and IPC servers.
+    - [x] Add session cleanup task to the IPC server.
+
+### Sub-Phase 3: Modify Coordinator & IPC Handler
+
+- [x] **Coordinator (`happe/src/coordinator.rs`):**
+    - [x] Modify `process_query` signature to accept `session: &Session`.
+    - [x] Add helper function `get_conversation_history` to extract history from the session.
+    - [x] Add helper function `update_session_history` to store turns in the session.
+- [x] **IPC Handler (`happe/src/ipc_server.rs`):**
+    - [x] Modify `handle_connection`:
+        - [x] Get or create session for the request.
+        - [x] Pass session to `coordinator::process_query`.
+        - [x] Update session with new conversation turn.
+        - [x] Save session back to the store.
+- [x] **HTTP Handler (`happe/src/http_server.rs`):**
+    - [x] Modify `handle_query`:
+        - [x] Extract session ID from request or create a new one.
+        - [x] Get or create session.
+        - [x] Pass session to `coordinator::process_query`.
+        - [x] Update and save session.
+
+### Sub-Phase 4: Update IPC Request & Client (`@cli`)
+
+- [x] **IPC Request (`ipc/src/happe_request/types.rs` or `mod.rs`):**
+    - [x] Add `session_id: Option<String>` field to `HappeQueryRequest` struct.
+    - [x] Add `session_id: Option<String>` field to `HappeQueryResponse` struct.
+- [x] **CLI (`@cli` Crate):**
+    - [x] Identify where the main interaction loop/HAPPE client logic resides (e.g., `cli/src/app.rs` or `cli/src/happe_client.rs`).
+    - [x] On CLI startup, generate a persistent `session_id` for the duration of the run (e.g., `let session_id = uuid::Uuid::new_v4().to_string();`).
+    - [x] Modify the code that creates and sends `HappeQueryRequest` to include this `session_id`.
+
+### Sub-Phase 5: Testing & Refinement
+
+- [x] **Unit Tests:**
+    - [x] Add tests for the `Session` struct in `store.rs`.
+    - [x] Add tests for `InMemorySessionStore`.
+- [ ] **Integration Tests:** Test IPC handler with session history.
+- [ ] **End-to-End Tests (`@cli` -> `happe-daemon`):** Verify context is maintained across multiple turns within a single CLI run.
+- [ ] **Refinement:** Assess performance, history pruning, error handling, security of session IDs.

cli/src/app.rs

@@ -11,6 +11,7 @@ use crate::output::print_happe_response;
 /// Runs a single query mode, sending one prompt to the HAPPE daemon and displaying the response
 pub async fn run_single_query(prompt: String, happe_client: &HappeClient) -> Result<()> {
     info!("Running single query: {}", prompt);
+    info!("Using session ID: {}", happe_client.session_id());
 
     // Display a spinner while waiting for response
     let spinner = ProgressBar::new_spinner();
@@ -27,13 +28,18 @@ pub async fn run_single_query(prompt: String, happe_client: &HappeClient) -> Res
     match happe_client.send_query(prompt).await {
         Ok(response) => {
             spinner.finish_and_clear();
-            
+
             if let Some(error) = response.error {
                 error!("HAPPE error: {}", error);
                 println!("Error: {}", error);
                 return Ok(());
             }
-            
+
+            // Log session ID if one was returned
+            if let Some(session_id) = &response.session_id {
+                debug!("Response session ID: {}", session_id);
+            }
+
             print_happe_response(&response.response);
         }
         Err(e) => {
@@ -49,6 +55,7 @@ pub async fn run_single_query(prompt: String, happe_client: &HappeClient) -> Res
 /// Runs an interactive chat session with the HAPPE daemon
 pub async fn run_interactive_chat(happe_client: &HappeClient) -> Result<()> {
     println!("Starting interactive chat session with HAPPE daemon.");
+    println!("Session ID: {}", happe_client.session_id().blue());
     println!("Type 'exit' or 'quit' to end the session.");
     println!();
 
@@ -89,13 +96,19 @@ pub async fn run_interactive_chat(happe_client: &HappeClient) -> Result<()> {
         match happe_client.send_query(input.to_string()).await {
             Ok(response) => {
                 spinner.finish_and_clear();
-                
+
                 if let Some(error) = response.error {
                     error!("HAPPE error: {}", error);
                     println!("Error: {}", error);
                     continue;
                 }
-                
+
+                // Log session ID if one was returned
+                if let Some(session_id) = &response.session_id {
+                    debug!("Response session ID: {}", session_id);
+                    // We don't need to update our session ID as the client will track it
+                }
+
                 print_happe_response(&response.response);
             }
             Err(e) => {