Skip to content

rtpengine-recording: recording occasionally not produced (no output file, no notify) — intermittent, ~1 in 10–15 calls #2136

Description

@Bhavik5025

Description

We're running a two-daemon call recording setup (core rtpengine + rtpengine-recording) in production, using kernelized packet export (recording-method = proc) triggered per-call via the record call: yes ng protocol flag from our SBC application on a mid-call re-INVITE.

Intermittently — roughly 1 in every 10–15 recorded calls — the recording pipeline fails to produce a usable result:

  • No final output file appears in output-dir.
  • No notify-uri HTTP notification is ever received by our webhook endpoint.
  • The affected calls are otherwise entirely normal: they connect, run for a normal duration, and terminate cleanly with a standard BYE. There is nothing unusual in the SIP signaling for the affected calls that we've been able to identify.
  • This is not reproducible on demand. It happens sporadically under normal production load.

Steps we've taken to isolate the cause so far

  1. Ruled out our own webhook handler as the sole cause — added logging immediately on request receipt (before any other processing) to confirm whether a notify request lands at all. For the affected calls, no request arrives at our endpoint — so this is not a case of our handler failing to respond in time or erroring silently.
  2. Ruled out notify-purge deleting the file after a successful notifynotify-purge is disabled in our config, so it cannot be responsible for the missing file.
  3. Ruled out filesystem permissions — both rtpengine-recording and our consuming application run as root, so standard ownership/permission mismatches are not a factor.
  4. Still to confirm (would appreciate guidance on where to look): whether the core rtpengine daemon is writing the initial spool-dir pcap/metadata file for the affected calls at all (Stage 1), versus the spool file existing but rtpengine-recording failing during decode/finalize (Stage 2). We plan to grep both daemons' logs by Call-ID the next time this reproduces and will attach the pcap + metadata for an affected call once captured.

Expected behavior

Every call for which record call: yes is successfully acknowledged by rtpengine (response.result === 'ok') should reliably result in either:

  • a completed output file plus a corresponding notify-uri HTTP notification, or
  • a clearly logged failure (decode error, resample error, storage error, etc.) that we can act on.

Actual behavior

For an intermittent subset of calls, neither occurs — no file, no notification, and (so far) no clearly attributable error logged at our default log level.


Environment

Field Value
rtpengine version 13.1.0.0+0~mr13.1.0.0 git-HEAD-f3aa776b (confirmed identical for both rtpengine and rtpengine-recording)
Distribution Debian GNU/Linux 10 (buster)
Linux kernel version 5.10.0-0.deb10.16-cloud-amd64
CPU architecture x86_64
Recording method proc (kernelized, xt_RTPENGINE kernel module)
Deployment core rtpengine and rtpengine-recording on separate hosts (SBC host vs. recording-engine host)

Configuration files

Attached: rtpengine.conf and rtpengine-recording.conf (full files as currently deployed in production).

Key options for reference:

rtpengine.conf ([rtpengine] section):

recording-dir = /var/spool/rtpengine
recording-method = proc
recording-file-per-stream = false
recording-all = false

rtpengine-recording.conf:

output-storage = file
output-format = wav
spool-dir = /var/spool/rtpengine
output-dir = /root/apps/sbc-recorder-server/recordings
resample-to = 8000
log-level = 6
output-mixed = true
flush-packets = true
notify-uri = https://<redacted>/recordings/recording-completed
notify-post = true

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions