Can't discard messages

[jkazimierczak-eficode]

Hey! I cannot delete failing messages. It seems that for some reason we got messages displayed as null.null as source table. Commited at shows start of the epoch:

These are the ones that are failing to be delivered. Other messages with correct attributes (actual schema.table names, and correct commited at timestamps) are delivered and acked correctly.

We have around 2K failing messages, and clearing them through this view doesn't work. Contrary to discarding just failing ones, discarding all messages works for messages that would go through just fine, but it doesn't clear the failing ones.

We're using v0.13.21.

[dosubot]

Hi @jkazimierczak-eficode! I'm Dosu and I’m helping the sequin team.

This looks like a known bug in v0.13.21. The null.null table name and epoch timestamp symptoms are specific to that version, and the inability to discard failing messages is also part of the same set of bugs.

These issues were fixed in commits after v0.13.21. Upgrading to a newer version should resolve this.

If you need an immediate workaround before upgrading, you may be able to manually delete the stuck messages directly from the sequin_streams.consumer_events database table, though upgrading is the recommended solution.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[jkazimierczak-eficode]

It seems related, so I'll try that out and get back with the results here.

[jkazimierczak-eficode]

@dosu I've upgraded to 14.6, but I still cannot delete failing messages. This time failing messages show correct table (no longer null.null), but they fail to be removed. Clicking on remove failed messages shows a toats with correct count, but going to Overview, the count is still the same. Pausing and resuming the sink restores failing messages. Can I clear them somehow manually?

[dosubot]

Good to hear the null.null issue is resolved! The discard behavior you're seeing — toast shows the correct count but messages reappear after pause/resume — suggests the messages are being cleared from in-memory state but not actually deleted from the database. When the sink restarts, it reloads all persisted messages from the consumer_events table [1].

You can clean them up manually via SQL. First, pause the sink, then:

-- Find your consumer/sink ID
SELECT id, name FROM public.sink_consumers;

-- Preview the failing messages
SELECT id, ack_id, deliver_count, last_delivered_at, state
FROM sequin_streams.consumer_events
WHERE consumer_id = 'YOUR_CONSUMER_ID'
  AND deliver_count > 0;

-- Delete them
DELETE FROM sequin_streams.consumer_events
WHERE consumer_id = 'YOUR_CONSUMER_ID'
  AND deliver_count > 0;

Then resume the sink. That should clear them for good.

It's worth noting this appears to be a bug in the discard flow — the database deletion after the in-memory removal may be failing silently [2]. You may want to flag this for the maintainers as a separate issue from the original null.null problem, since it persists on v0.14.6.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[jkazimierczak-eficode]

Can confirm manual path has worked for me.

1. Before doing so, pause/disable your sink from the UI (I've disabled the sink at the time of writing this).
2. Discard failing messages from the UI - this makes Sequin clear it's in-memory messages. Not doing so will yield you a weird state where you can re-run the delete query with different results, as Sequin can hold messages in memory.
3. For extra safety you can also stop Sequin container. I personally did fine without doing it, but noting it here just in case.
4. Resume the sink from the UI.

-- PostgreSQL
SELECT id, name, type, status FROM sequin_config.sink_consumers;

DELETE FROM sequin_streams.consumer_events
WHERE consumer_id = 'YOUR-CONSUMER-ID'
  AND deliver_count >= 1;

-- Or if you want to reference consumer by it's name:
DELETE FROM sequin_streams.consumer_events cevt
USING sequin_config.sink_consumers sc
WHERE cevt.consumer_id = sc.id
  AND sc.name = 'typesense'
  AND cevt.deliver_count >= 1;

[jkazimierczak-eficode]

Also, doing it this way still shows an error in the UI about message not delivered (in the overview tab). It was gone with first message received after that.

Error: Message not delivered

Messages are failing to be delivered to the destination. See the message details of one failing message for more information.

[typesense]: Failed to index document: Field `reporterName` has been declared in the schema, but is not found in the document.

[jkazimierczak-eficode]

This is also affecting pending messages, so actually "Discard All" does nothing in both cases. I had to manually dig into DB to find messages that were potentially faulty by querying JSONB and remove them manually:

\x auto

SELECT COUNT(*)
FROM sequin_streams.consumer_events
WHERE data -> 'record' ->> 'attr_name' LIKE 'value%';