Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Failure rate sparkline per queue — a mini 12-bar chart in the Queues table shows the percentage of jobs that failed (vs. completed) in each of the last 12 hours; bars are red, sized proportionally to the failure rate (0–100 %), and include a tooltip with the hour label and exact percentage; empty hours render as a faint border-colored bar; queues with no activity in the last 12 hours show "—"
- Queue depth trend — a "Queue Depth — Last 12 Hours" card on the dashboard shows estimated queue depth at 12 hourly snapshots; depth at time T is the count of jobs created before T that had not yet finished by T; bars are purple (distinct from the blue throughput and red failure rate charts); the card header shows current depth; empty state shown when no active jobs exist in the window
- Slow job detection — a configurable `slow_job_threshold` setting (default nil = disabled) flags claimed jobs that have been running longer than the threshold; when set, the Running tab gains a "Running For" column showing each job's elapsed time, slow jobs are highlighted with an orange row background and a "slow" badge, and a "Slow Jobs" warning card appears on the dashboard linking to the Running tab

## [0.8.0] - 2026-05-20

Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ SolidQueueWeb surfaces all of this in a browser UI available at any route you ch
- **Dark mode** — ☽/☀ toggle in the header; preference persists to `localStorage` and defaults to the OS `prefers-color-scheme` on first visit; zero extra dependencies — implemented via CSS custom properties and a small Stimulus controller
- **Dashboard quick actions** — "Retry All Failed" and "Discard All Blocked" cards appear on the dashboard only when the respective count is non-zero; one-click bulk operations with confirm dialogs, keeping the dashboard clean when everything is healthy
- **CSV export** — "Export CSV" button on the jobs, failed jobs, and history pages downloads all records matching the current filters; columns are tailored per view
- **Slow job detection** — when `slow_job_threshold` is configured, claimed jobs running longer than the threshold are flagged with an orange row, a "slow" badge, and a "Running For" duration column on the Running tab; a "Slow Jobs" warning card appears on the dashboard with a link to the Running tab

## Screenshots

Expand Down Expand Up @@ -96,6 +97,7 @@ SolidQueueWeb.configure do |config|
config.dashboard_refresh_interval = 10_000 # dashboard auto-refresh in ms (default: 5_000)
config.default_refresh_interval = 30_000 # jobs/processes/history auto-refresh in ms (default: 10_000)
config.search_results_limit = 10 # max results per status in global search (default: 25)
config.slow_job_threshold = 5.minutes # flag claimed jobs running longer than this (default: nil = disabled)
end

SolidQueueWeb.authenticate do
Expand All @@ -111,9 +113,6 @@ No authentication is enforced by default. When the `authenticate` block returns

Planned features, roughly ordered by priority:

**Observability**
- Slow job detection — flag jobs exceeding a configurable duration threshold

**Operations**
- Scheduled job management — reschedule a job to run immediately, or push its `scheduled_at` forward
- Bulk retry with delay — retry all failed jobs with a configurable stagger to avoid thundering herd
Expand Down
9 changes: 8 additions & 1 deletion app/assets/stylesheets/solid_queue_web/_04_table.css
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,16 @@ td {

tr:last-child td { border-bottom: none; }
tbody tr:hover { background: var(--bg); }
.sqd-table-link,
.sqd-table-link:hover,
.sqd-table-link:visited { text-decoration: none; color: var(--primary); }

.sqd-empty {
text-align: center;
padding: 3rem 1rem;
color: var(--muted);
}
}

.sqd-row--slow { background: rgba(253, 126, 20, 0.07); }
.sqd-row--slow:hover { background: rgba(253, 126, 20, 0.13); }
.sqd-slow-duration { color: var(--warning); font-weight: 600; }
1 change: 1 addition & 0 deletions app/assets/stylesheets/solid_queue_web/_05_badges.css
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
.sqd-badge--supervisor { background: #e0d7f5; color: #4a2c8a; }
.sqd-badge--worker { background: #d1e7dd; color: #0f5132; }
.sqd-badge--dispatcher { background: #cff4fc; color: #055160; }
.sqd-badge--slow { background: #ffe8cc; color: #7c3d00; }

.sqd-process-meta { font-size: 12px; color: var(--muted); }
.sqd-process-meta span + span::before { content: " · "; }
Expand Down
5 changes: 4 additions & 1 deletion app/services/solid_queue_web/dashboard_stats.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module SolidQueueWeb
class DashboardStats
attr_reader :counts, :throughput, :sparkline, :depth_sparkline
attr_reader :counts, :throughput, :sparkline, :depth_sparkline, :slow_jobs_count

def initialize
@now = Time.current
Expand Down Expand Up @@ -32,6 +32,9 @@ def compute
finished_times.count { |t| t >= from && t < to }
end

threshold = SolidQueueWeb.slow_job_threshold
@slow_jobs_count = threshold ? SolidQueue::ClaimedExecution.where("created_at <= ?", threshold.ago).count : 0

job_timestamps = SolidQueue::Job
.where("created_at >= ? OR finished_at IS NULL", 72.hours.ago)
.pluck(:created_at, :finished_at)
Expand Down
14 changes: 14 additions & 0 deletions app/views/solid_queue_web/dashboard/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,20 @@
</div>
<% end %>

<% if SolidQueueWeb.slow_job_threshold && @stats.slow_jobs_count > 0 %>
<div class="sqd-card">
<div class="sqd-card__header">
<span class="sqd-card__title">Slow Jobs</span>
</div>
<div style="padding: 1rem; display: flex; flex-direction: column; gap: 0.5rem;">
<p style="color: var(--warning); font-size: 13px;">
<%= pluralize(@stats.slow_jobs_count, "job") %> running longer than <%= distance_of_time_in_words(SolidQueueWeb.slow_job_threshold) %>.
</p>
<%= link_to "Review →", jobs_path(status: "claimed"), class: "sqd-btn sqd-btn--muted" %>
</div>
</div>
<% end %>

<% if @stats.counts[:blocked] > 0 %>
<div class="sqd-card">
<div class="sqd-card__header">
Expand Down
2 changes: 1 addition & 1 deletion app/views/solid_queue_web/failed_jobs/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@
data-action="change->selection#toggle"
aria-label="Select job <%= job.class_name %>">
</td>
<td><%= link_to job.class_name, job_path(job) %></td>
<td><%= link_to job.class_name, job_path(job), class: "sqd-table-link" %></td>
<td>
<%= link_to job.queue_name, failed_jobs_path(queue: job.queue_name, q: @search, period: @period),
class: "sqd-mono", style: "color: inherit;" %>
Expand Down
2 changes: 1 addition & 1 deletion app/views/solid_queue_web/history/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
<tbody>
<% @jobs.each do |job| %>
<tr>
<td><%= link_to job.class_name, job_path(job) %></td>
<td><%= link_to job.class_name, job_path(job), class: "sqd-table-link" %></td>
<td>
<%= link_to job.queue_name, history_path(queue: job.queue_name, q: @search, period: @period),
class: "sqd-mono", style: "color: inherit;" %>
Expand Down
19 changes: 16 additions & 3 deletions app/views/solid_queue_web/jobs/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@
</td>
<td>
<span class="sqd-badge sqd-badge--<%= @status %>"><%= @status %></span>
<%= link_to job.class_name, job_path(job), style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
<%= link_to job.class_name, job_path(job), class: "sqd-table-link", style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
</td>
<td>
<%= link_to job.queue_name, queue_jobs_path(queue_name: job.queue_name, status: @status),
Expand Down Expand Up @@ -116,6 +116,7 @@
<% if @jobs.empty? %>
<div class="sqd-empty">No <%= @status %> jobs.</div>
<% else %>
<% slow_threshold = @status == "claimed" ? SolidQueueWeb.slow_job_threshold : nil %>
<table>
<thead>
<tr>
Expand All @@ -124,15 +125,22 @@
<th scope="col">Priority</th>
<th scope="col">Scheduled At</th>
<th scope="col">Enqueued At</th>
<% if @status == "claimed" %>
<th scope="col">Running For</th>
<% end %>
</tr>
</thead>
<tbody>
<% @jobs.each do |execution| %>
<% job = execution.job %>
<tr id="execution_<%= execution.id %>">
<% slow = slow_threshold && execution.created_at <= slow_threshold.ago %>
<tr id="execution_<%= execution.id %>"<%= slow ? ' class="sqd-row--slow"'.html_safe : "" %>>
<td>
<span class="sqd-badge sqd-badge--<%= @status %>"><%= @status %></span>
<%= link_to job.class_name, job_path(job), style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
<% if slow %>
<span class="sqd-badge sqd-badge--slow">slow</span>
<% end %>
<%= link_to job.class_name, job_path(job), class: "sqd-table-link", style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
</td>
<td>
<%= link_to job.queue_name, queue_jobs_path(queue_name: job.queue_name, status: @status),
Expand All @@ -143,6 +151,11 @@
<%= job.scheduled_at ? job.scheduled_at.strftime("%Y-%m-%d %H:%M:%S") : "—" %>
</td>
<td class="sqd-mono"><%= job.created_at.strftime("%Y-%m-%d %H:%M:%S") %></td>
<% if @status == "claimed" %>
<td class="sqd-mono<%= slow ? " sqd-slow-duration" : "" %>">
<%= time_ago_in_words(execution.created_at) %>
</td>
<% end %>
</tr>
<% end %>
</tbody>
Expand Down
2 changes: 1 addition & 1 deletion app/views/solid_queue_web/queues/jobs/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
<tr id="execution_<%= execution.id %>">
<td>
<span class="sqd-badge sqd-badge--<%= @status %>"><%= @status %></span>
<%= link_to job.class_name, job_path(job), style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
<%= link_to job.class_name, job_path(job), class: "sqd-table-link", style: "margin-left: 0.5rem;", data: { turbo_frame: "_top" } %>
</td>
<td><%= job.priority %></td>
<td class="sqd-mono">
Expand Down
2 changes: 1 addition & 1 deletion app/views/solid_queue_web/search/index.html.erb
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
<% data[:executions].each do |execution| %>
<% job = execution.job %>
<tr>
<td><%= link_to job.class_name, job_path(job) %></td>
<td><%= link_to job.class_name, job_path(job), class: "sqd-table-link" %></td>
<td class="sqd-mono"><%= job.queue_name %></td>
<td class="sqd-mono"><%= job.created_at.strftime("%Y-%m-%d %H:%M:%S") %></td>
</tr>
Expand Down
7 changes: 6 additions & 1 deletion lib/solid_queue_web.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

module SolidQueueWeb
class << self
attr_writer :page_size, :dashboard_refresh_interval, :default_refresh_interval, :search_results_limit
attr_writer :page_size, :dashboard_refresh_interval, :default_refresh_interval, :search_results_limit,
:slow_job_threshold

def page_size
@page_size || 25
Expand All @@ -22,6 +23,10 @@ def search_results_limit
@search_results_limit || 25
end

def slow_job_threshold
@slow_job_threshold
end

def configure
yield self
end
Expand Down
5 changes: 5 additions & 0 deletions spec/dummy/config/initializers/solid_queue_web.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
if Rails.env.development?
SolidQueueWeb.configure do |config|
config.slow_job_threshold = 5.minutes
end
end
21 changes: 19 additions & 2 deletions spec/dummy/db/seeds.rb
Original file line number Diff line number Diff line change
Expand Up @@ -71,13 +71,30 @@
arguments: { batch: i + 1 }.to_json,
priority: 0,
active_job_id: SecureRandom.uuid,
created_at: rand(1..10).minutes.ago,
updated_at: rand(1..10).minutes.ago
created_at: rand(1..4).minutes.ago,
updated_at: rand(1..4).minutes.ago
)
job.ready_execution&.destroy
SolidQueue::ClaimedExecution.create!(job: job, process: workers.sample, created_at: job.created_at)
end

puts "Seeding slow claimed jobs (for slow job detection)..."
# These exceed a 5-minute threshold — visible as orange rows when slow_job_threshold is configured
slow_durations = [8.minutes, 20.minutes, 45.minutes]
slow_durations.each_with_index do |duration, i|
job = SolidQueue::Job.create!(
queue_name: queues.sample,
class_name: job_classes.sample,
arguments: { slow_job: true, idx: i }.to_json,
priority: 0,
active_job_id: SecureRandom.uuid,
created_at: duration.ago,
updated_at: duration.ago
)
job.ready_execution&.destroy
SolidQueue::ClaimedExecution.create!(job: job, process: workers.sample, created_at: duration.ago)
end

puts "Seeding failed jobs..."
errors = [
{ class: "RuntimeError", message: "undefined method `foo' for nil:NilClass" },
Expand Down
32 changes: 32 additions & 0 deletions spec/requests/solid_queue_web/dashboard_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,38 @@
end
end

describe "slow jobs card" do
after { SolidQueueWeb.slow_job_threshold = nil }

it "shows the slow jobs card when threshold is set and jobs exceed it" do
process = SolidQueue::Process.create!(
kind: "Worker", pid: 99_998, hostname: "test-host",
name: "worker-slow-test", last_heartbeat_at: Time.current
)
job = SolidQueue::Job.create!(
queue_name: "default", class_name: "SlowJob",
arguments: {}, active_job_id: SecureRandom.uuid
)
execution = SolidQueue::ClaimedExecution.create!(job: job, process: process)
execution.update_columns(created_at: 10.minutes.ago)
SolidQueueWeb.slow_job_threshold = 1.second

get "/jobs"
expect(response.body).to include("Slow Jobs")
end

it "does not show the slow jobs card when threshold is not configured" do
get "/jobs"
expect(response.body).not_to include("Slow Jobs")
end

it "does not show the slow jobs card when no jobs exceed the threshold" do
SolidQueueWeb.slow_job_threshold = 1.hour
get "/jobs"
expect(response.body).not_to include("Slow Jobs")
end
end

describe "authentication" do
after { SolidQueueWeb.instance_variable_set(:@authenticate, nil) }

Expand Down
53 changes: 53 additions & 0 deletions spec/requests/solid_queue_web/jobs_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -269,4 +269,57 @@
expect(response.body).to include("Could not discard jobs")
end
end

describe "slow job detection (claimed tab)" do
let!(:worker_process) do
SolidQueue::Process.create!(
kind: "Worker", pid: 99_999, hostname: "test-host",
name: "worker-test", last_heartbeat_at: Time.current
)
end

let!(:claimed_job) do
SolidQueue::Job.create!(
queue_name: "default", class_name: "SlowJob",
arguments: {}, active_job_id: SecureRandom.uuid
)
end

let!(:claimed_execution) do
SolidQueue::ClaimedExecution.create!(job: claimed_job, process: worker_process)
end

after { SolidQueueWeb.slow_job_threshold = nil }

it "shows the Running For column on the claimed tab" do
get "/jobs/list", params: { status: "claimed" }
expect(response.body).to include("Running For")
end

it "does not show the Running For column on other tabs" do
get "/jobs/list", params: { status: "ready" }
expect(response.body).not_to include("Running For")
end

it "highlights slow jobs when threshold is configured and exceeded" do
SolidQueueWeb.slow_job_threshold = 1.second
claimed_execution.update_columns(created_at: 10.minutes.ago)

get "/jobs/list", params: { status: "claimed" }
expect(response.body).to include('class="sqd-row--slow"')
expect(response.body).to include("sqd-badge--slow")
end

it "does not highlight jobs within the threshold" do
SolidQueueWeb.slow_job_threshold = 1.hour
get "/jobs/list", params: { status: "claimed" }
expect(response.body).not_to include('class="sqd-row--slow"')
end

it "does not highlight any row when threshold is not configured" do
claimed_execution.update_columns(created_at: 10.hours.ago)
get "/jobs/list", params: { status: "claimed" }
expect(response.body).not_to include('class="sqd-row--slow"')
end
end
end
Loading