feat: Add master training trigger endpoint

[rhuret]

Summary

This PR adds a dedicated master endpoint to trigger training on demand:

• POST /api/training/trigger
• force=false schedules training only when the master is idle
• force=true cancels the active training run and queues exactly one replacement run

The implementation keeps the existing fit_period-driven background ticker and
reuses the current master training pipeline instead of introducing a separate
execution path.

Why

Today, training is mainly driven by the periodic fit_period loop or indirectly
through data import flows. That is not precise enough for operational workflows
where ingestion and training must stay synchronized.

In our case, ingestion and training are orchestrated by separate steps. Once an
external ingestion job finishes, we need a deterministic way to trigger a fresh
training cycle immediately instead of waiting for the next scheduled tick.

This reduces the gap between:

• data becoming available after ingestion
• models and cached recommendations being refreshed by training

Without an explicit trigger, ingestion/training synchronization depends on timing
and cron alignment, which increases the risk of serving stale recommendations
after a successful data import.

Changes

• add POST /api/training/trigger on gorse-master
• add explicit in-memory training state to track:
  • whether a run is currently in progress
  • whether a replacement run is already queued
• make forced retraining cancel the active run cooperatively and reschedule one replacement run
• return JSON responses for both success and error cases
• stop canceled runs promptly on context.Canceled
• avoid logging expected cancellation as cache/training errors during force=true

Behavior

Idle master

POST /api/training/trigger

• returns 202 Accepted
• schedules one immediate run

POST /api/training/trigger?force=true

• returns 202 Accepted
• schedules one immediate run

Training already running

POST /api/training/trigger

• returns 200 OK
• does not cancel the current run
• does not enqueue an extra rerun

POST /api/training/trigger?force=true

• returns 202 Accepted
• cancels the active run
• queues exactly one replacement run

Validation

Tested with:

go test ./master

Covered by tests for:

• 405 Method Not Allowed
• 401 Unauthorized
• invalid force values
• default force=false
• non-forced no-op while training is running
• forced cancel-and-reschedule behavior
• canceled-context helper behavior in cache update checks

[codecov]

Codecov Report

❌ Patch coverage is 47.01493% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.54%. Comparing base (b185641) to head (604ba46).

Files with missing lines	Patch %	Lines
master/tasks.go	6.66%	56 Missing ⚠️
master/master.go	60.86%	8 Missing and 1 partial ⚠️
master/rest.go	88.23%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1201      +/-   ##
==========================================
- Coverage   72.72%   72.54%   -0.19%     
==========================================
  Files          88       88              
  Lines       16258    16390     +132     
==========================================
+ Hits        11824    11890      +66     
- Misses       3233     3299      +66     
  Partials     1201     1201              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.