metrics.md

Firecracker Metrics Configuration

For the metrics capability, Firecracker uses a single Metrics system. This system can be configured either by: a) sending a PUT API Request to the /metrics path: or b) using the --metrics-path CLI option.

Note the metrics configuration is not part of the guest configuration and is not restored from a snapshot.

Prerequisites

In order to configure the Metrics, first you have to create the resource that will be used for storing the metrics:

# Create the required named pipe:
mkfifo metrics.fifo

# The Metrics system also works with usual files:
touch metrics.file

Configuring the system via CLI

When launching Firecracker, use the CLI option to set the metrics file.

./firecracker --metrics-path metrics.fifo

Configuring the system via API

You can configure the Metrics system by sending the following API command:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/metrics" \
    -H "accept: application/json" \
    -H "Content-Type: application/json" \
    -d "{
             \"metrics_path\": \"metrics.fifo\"
    }"

Details about this configuration can be found in the swagger definition.

The metrics are written to the metrics_path in JSON format.

Optional metrics fields

Firecracker can emit additional top-level fields on every metrics line. Each is opt-in and off by default, so the default output is unchanged. They are configured alongside metrics_path, through the PUT /metrics API request or the metrics block of a configuration file; the --metrics-path CLI option configures only the path. Like the rest of the metrics configuration, they are set once before boot and fixed for the lifetime of the microVM.

id

Set emit_id to true to emit the microVM instance id (the value passed to --id, defaulting to anonymous-instance) under a top-level id field. This lets lines collected from multiple microVMs into one destination be attributed to their source.

properties

Set properties to a map of operator-defined key-value pairs to emit them under a top-level properties field.

Examples

Configure the fields via the API by adding them to the request body:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/metrics" \
    -H "accept: application/json" \
    -H "Content-Type: application/json" \
    -d "{
             \"metrics_path\": \"metrics.fifo\",
             \"emit_id\": true,
             \"properties\": {
                 \"customer_id\": \"1234\",
                 \"bundle_id\": \"fn-abc\"
             }
    }"

The same fields can be provided in the metrics block of a configuration file passed via --config-file:

{
    "metrics": {
        "metrics_path": "metrics.fifo",
        "emit_id": true,
        "properties": {
            "customer_id": "1234",
            "bundle_id": "fn-abc"
        }
    }
}

With neither field configured, a line carries only the default keys:

{"utc_timestamp_ms": 1739000000000, "api_server": {"...": 0}}

With both fields configured as above, the same line also carries the instance id and the properties:

{"utc_timestamp_ms": 1739000000000, "id": "my-instance", "properties": {"bundle_id": "fn-abc", "customer_id": "1234"}, "api_server": {"...": 0}}

Flushing the metrics

The metrics get flushed in two ways:

• without user intervention every 60 seconds;
• upon user demand, by issuing a FlushMetrics request. You can find how to use this request in the actions API.

If the path provided is a named pipe, you can use the script below to read from it:

metrics=metrics.fifo

while true
do
    if read line <$metrics; then
        echo $line
    fi
done

echo "Reader exiting"

Otherwise, if the path points to a normal file, you can simply do:

cat metrics.file

Metrics emitted by Firecracker

The metrics emitted by Firecracker are in JSON format. Below are the keys present in each metrics json object emitted by Firecracker:

"api_server"
"balloon"
"block"
"deprecated_api"
"entropy"
"get_api_requests"
"i8042"
"latencies_us"
"logger"
"mmds"
"net"
"patch_api_requests"
"put_api_requests"
"rtc"
"seccomp"
"signals"
"uart"
"vcpu"
"vhost_user_block"
"vmm"
"vsock"

Below table explains where Firecracker metrics are defined :

Metrics key	Device	Additional comments
balloon	BalloonDeviceMetrics	Represent metrics for the Balloon device.
block	BlockDeviceMetrics	Represent aggregate metrics for Virtio Block device.
block_{block_drive_id}	BlockDeviceMetrics	Represent Virtio Block device metrics for the endpoint "/drives/{drive_id}" e.g. "block_rootfs": represent metrics for the endpoint "/drives/rootfs"
i8042	I8042DeviceMetrics	Represent Metrics specific to the i8042 device.
net	NetDeviceMetrics	Represent aggregate metrics for Virtio Net device.
net_{iface_id}	NetDeviceMetrics	Represent Virtio Net device metrics for the endpoint "/network-interfaces/{iface_id}" e.g. net_eth0 represent metrics for the endpoint "/network-interfaces/eth0"
rtc	RTCDeviceMetrics	Represent Metrics specific to the RTC device. Note: this is emitted only on aarch64.
uart	SerialDeviceMetrics	Represent Metrics specific to the serial device.
vhost_user_{dev}_{dev_id}	VhostUserDeviceMetrics	Represent Vhost-user device metrics for the device dev and device id dev_id. e.g. "vhost_user_block_rootfs": represent metrics for vhost-user block device having the endpoint "/drives/rootfs"
vsock	VsockDeviceMetrics	Represent Metrics specific to the vsock device.
entropy	EntropyDeviceMetrics	Represent Metrics specific to the entropy device.
"api_server" "deprecated_api" "get_api_requests" "latencies_us" "logger" "mmds" "patch_api_requests" "put_api_requests" "seccomp" "signals" "vcpu" "vmm"	metrics.rs	Rest of the metrics are defined in the same file metrics.rs.

Note: Firecracker emits all the above metrics regardless of the presense of that component i.e. even if vsock device is not attached to the Microvm, Firecracker will still emit the Vsock metrics with key as vsock and value of all metrics defined in VsockDeviceMetrics as 0.

Units for Firecracker metrics:

Units for Firecracker metrics are embedded in their name.
Below pseudo code should be to extract units from Firecracker metrics name:
Note: An example of full_key for below logic is "vcpu.exit_io_in_agg.min_us"

    if substring "_bytes" or "_bytes_count" is present in any subkey of full_key
        Unit is "Bytes"
    else substring "_ms" is present in any subkey of full_key
        Unit is "Milliseconds"
    else substring "_us" is present in any subkey of full_key
        Unit is "Microseconds"
    else
        Unit is "Count"