set up opentelemetry metrics for new infra (and now)

[syphar]

datadog is the target system for logs & metrics in the new infra. We're already running the datadog agent locally on the server to ship our logs, so we can also start using it for metrics. I already activated the OTLP endpoint, and also enabled the process & nginx monitoring. I'll be able to setup & test in more detail when I have the necessary permissions.

This way we can have both systems running side by side for some time, and compare.

In our codebase I chose not the "native" datadog libraries, but opentelemetry as standard solution. Hopefully this will lead to us never having to refactor any more, only choose new exporters.

After I validated everything and created the necessary dashboards in datadog, we can drop the whole prometheus logic & integration.

Some detail notes:

• I chose a simple approach where I dropped custom histogram bucket sizes, wanting to see first how they look like in datadog.
• I didn't migrate the whole recent_* metrics, I'm thinking about either dropping them completely (after the initial learning they brought, I didn't see much additional value), or perhaps using datadog log-based metrics for them, where datadog generates metrics by parsing logs.
• I tried to move from a glocal god-object for metrics to each main component having its own metrics.
• the thread- and file-descriptor-count metrics should be replacable by using the datadog process monitor. I configured this and can evaluate when I have the increased access I need.
• In my mind, the metrics that I would relate to the builder should be in one struct, but are used in two places. This is a pending refactor, but not now.

[GuillaumeGomez]

Does it need to be setup when running locally? Also: can we access these new logs locally? If so, please add it into the README or the docs.

[syphar]

Does it need to be setup when running locally? Also: can we access these new logs locally? If so, please add it into the README or the docs.

It's optionally, and it's not logs but metrics. So no local change is needed.

If you want to debug the metrics, see what is pushed to the exporter, I can add a couple of lines how to configure it for debugging.

( it's why I added the new opentelemetry container)