Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.3.0
What happened and how to reproduce it?
The dag_processing.processes metric description states: "the delta is negative when, since the last metric was sent, processes have completed." However, _collect_results() , the normal graceful completion path, emits no stats.decr, so the delta is never negative under normal operation. A decr is only emitted for abnormal exits , which are : stop (file deleted), timeout, and terminate (shutdown). This means the metric does not behave as documented.
Current behaviour
stats.incr("dag_processing.processes", tags={"action": "start", ...}) is called
in _start_new_processes() when a child DagFileProcessorProcess is spawned.
A matching stats.decr(...) is called in three abnormal exit paths:
terminate_orphan_processes() action: "stop" (file deleted mid-parse)_kill_timed_out_processors() action: "timeout" (exceeded processor_timeout)terminate() action: "terminate" (manager shutting down)
However, _collect_results() , the normal graceful exit path called every loop
iteration pops the finished processor from self._processors and calls
processor.close() with no corresponding decr:
This means under normal operation (parsers completing successfully), the metric value drifts upward indefinitely and never accurately reflects the number of currently running processes
What you think should happen instead?
A stats.decr("dag_processing.processes", tags={"action": "finish", ...}) should be emitted when a process completes gracefully, symmetric with the incr in _start_new_processes().
Operating System
No response
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.3.0
What happened and how to reproduce it?
The dag_processing.processes metric description states: "the delta is negative when, since the last metric was sent, processes have completed." However,
_collect_results(), the normal graceful completion path, emits no stats.decr, so the delta is never negative under normal operation. A decr is only emitted for abnormal exits , which are : stop (file deleted), timeout, and terminate (shutdown). This means the metric does not behave as documented.Current behaviour
stats.incr("dag_processing.processes", tags={"action": "start", ...})is calledin
_start_new_processes()when a childDagFileProcessorProcessis spawned.A matching
stats.decr(...)is called in three abnormal exit paths:terminate_orphan_processes()action:"stop"(file deleted mid-parse)_kill_timed_out_processors()action:"timeout"(exceededprocessor_timeout)terminate()action:"terminate"(manager shutting down)However,
_collect_results(), the normal graceful exit path called every loopiteration pops the finished processor from
self._processorsand callsprocessor.close()with no correspondingdecr:This means under normal operation (parsers completing successfully), the metric value drifts upward indefinitely and never accurately reflects the number of currently running processes
What you think should happen instead?
A
stats.decr("dag_processing.processes", tags={"action": "finish", ...})should be emitted when a process completes gracefully, symmetric with the incr in_start_new_processes().Operating System
No response
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct