Description
We're running a two-daemon call recording setup (core rtpengine + rtpengine-recording) in production, using kernelized packet export (recording-method = proc) triggered per-call via the record call: yes ng protocol flag from our SBC application on a mid-call re-INVITE.
Intermittently — roughly 1 in every 10–15 recorded calls — the recording pipeline fails to produce a usable result:
- No final output file appears in
output-dir.
- No
notify-uri HTTP notification is ever received by our webhook endpoint.
- The affected calls are otherwise entirely normal: they connect, run for a normal duration, and terminate cleanly with a standard BYE. There is nothing unusual in the SIP signaling for the affected calls that we've been able to identify.
- This is not reproducible on demand. It happens sporadically under normal production load.
Steps we've taken to isolate the cause so far
- Ruled out our own webhook handler as the sole cause — added logging immediately on request receipt (before any other processing) to confirm whether a notify request lands at all. For the affected calls, no request arrives at our endpoint — so this is not a case of our handler failing to respond in time or erroring silently.
- Ruled out
notify-purge deleting the file after a successful notify — notify-purge is disabled in our config, so it cannot be responsible for the missing file.
- Ruled out filesystem permissions — both
rtpengine-recording and our consuming application run as root, so standard ownership/permission mismatches are not a factor.
- Still to confirm (would appreciate guidance on where to look): whether the core
rtpengine daemon is writing the initial spool-dir pcap/metadata file for the affected calls at all (Stage 1), versus the spool file existing but rtpengine-recording failing during decode/finalize (Stage 2). We plan to grep both daemons' logs by Call-ID the next time this reproduces and will attach the pcap + metadata for an affected call once captured.
Expected behavior
Every call for which record call: yes is successfully acknowledged by rtpengine (response.result === 'ok') should reliably result in either:
- a completed output file plus a corresponding
notify-uri HTTP notification, or
- a clearly logged failure (decode error, resample error, storage error, etc.) that we can act on.
Actual behavior
For an intermittent subset of calls, neither occurs — no file, no notification, and (so far) no clearly attributable error logged at our default log level.
Environment
| Field |
Value |
| rtpengine version |
13.1.0.0+0~mr13.1.0.0 git-HEAD-f3aa776b (confirmed identical for both rtpengine and rtpengine-recording) |
| Distribution |
Debian GNU/Linux 10 (buster) |
| Linux kernel version |
5.10.0-0.deb10.16-cloud-amd64 |
| CPU architecture |
x86_64 |
| Recording method |
proc (kernelized, xt_RTPENGINE kernel module) |
| Deployment |
core rtpengine and rtpengine-recording on separate hosts (SBC host vs. recording-engine host) |
Configuration files
Attached: rtpengine.conf and rtpengine-recording.conf (full files as currently deployed in production).
Key options for reference:
rtpengine.conf ([rtpengine] section):
recording-dir = /var/spool/rtpengine
recording-method = proc
recording-file-per-stream = false
recording-all = false
rtpengine-recording.conf:
output-storage = file
output-format = wav
spool-dir = /var/spool/rtpengine
output-dir = /root/apps/sbc-recorder-server/recordings
resample-to = 8000
log-level = 6
output-mixed = true
flush-packets = true
notify-uri = https://<redacted>/recordings/recording-completed
notify-post = true
Description
We're running a two-daemon call recording setup (core
rtpengine+rtpengine-recording) in production, using kernelized packet export (recording-method = proc) triggered per-call via therecord call: yesngprotocol flag from our SBC application on a mid-call re-INVITE.Intermittently — roughly 1 in every 10–15 recorded calls — the recording pipeline fails to produce a usable result:
output-dir.notify-uriHTTP notification is ever received by our webhook endpoint.Steps we've taken to isolate the cause so far
notify-purgedeleting the file after a successful notify —notify-purgeis disabled in our config, so it cannot be responsible for the missing file.rtpengine-recordingand our consuming application run asroot, so standard ownership/permission mismatches are not a factor.rtpenginedaemon is writing the initial spool-dir pcap/metadata file for the affected calls at all (Stage 1), versus the spool file existing butrtpengine-recordingfailing during decode/finalize (Stage 2). We plan to grep both daemons' logs by Call-ID the next time this reproduces and will attach the pcap + metadata for an affected call once captured.Expected behavior
Every call for which
record call: yesis successfully acknowledged by rtpengine (response.result === 'ok') should reliably result in either:notify-uriHTTP notification, orActual behavior
For an intermittent subset of calls, neither occurs — no file, no notification, and (so far) no clearly attributable error logged at our default log level.
Environment
Configuration files
Attached:
rtpengine.confandrtpengine-recording.conf(full files as currently deployed in production).Key options for reference:
rtpengine.conf([rtpengine]section):rtpengine-recording.conf: