diff --git a/CHANGELOG.md b/CHANGELOG.md index df8598e..19b7f33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,14 +5,19 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [Unreleased] TBD -## [1.1.4] +### Added +- Add new `LogTarget` +- Introduce new `formatter:` options: `:json`, `:logfmt`, and `:noop` +- Introduce new `transform:` options: `:cloud_watch`, `:l2met`, and `:noop` + +## [1.1.4] 2024-05-29 ### Changed - Updated gems in the lockfile -## [1.1.3] +## [1.1.3] 2024-05-13 ### Changed - Updated gems in the lockfile @@ -23,14 +28,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Dropped - Check for support for 'ubuntu-18.04' -## [1.1.2] +## [1.1.2] 2022-12-28 - Add Puma 6 compatibility -## [1.1.1] + +## [1.1.1] 2022-06-22 Public release. -## [1.1.0] +## [1.1.0] 2022-06-22 Out of beta testing, reading for usage. Following is a recap from Alpha & Beta releases. @@ -42,8 +48,7 @@ Out of beta testing, reading for usage. Following is a recap from Alpha & Beta r - `config.socket_parser` option to allow custom parser implementation as needed - Datadog widgets examples under `docs/examples.md` -## [1.1.0 Beta] - +## [1.1.0 Beta] ??? ### Added Different ways to parse `Socket::Option`. Mainly due to the fact that `#inspect` can't @@ -56,8 +61,7 @@ struct, so it should more or less stay stable. You can configure it by passing in `config.socket_parser = :inspect` or `config.socket_parser = ->(opt) { your implementation }`. -## [1.1.0 Alpha] - +## [1.1.0 Alpha] ??? ### Added Socket telemetry, and to be more precise new metric: `sockets.backlog`. If enabled it will @@ -66,32 +70,32 @@ be acknowledged by Puma). It will be exposed under `sockets-backlog` metric. You can enable and test it via `config.sockets_telemetry!` option. -## [1.0.0] - 2021-09-08 +## [1.0.0] 2021-09-08 ### Added -- Release to Github Packages -- Explicitly flush datadog metrics after publishing them +- Release to GitHub Packages +- Explicitly flush Datadog metrics after publishing them - Middleware for measuring and tracking request queue time ### Changed -- Replace `statsd.batch` with direct calls, as it aggregates metrics interally by default now. +- Replace `statsd.batch` with direct calls, as it aggregates metrics internally by default now. Also `#batch` method is deprecated and will be removed in version 6 of Datadog Statsd client. -## [0.3.1] - 2021-03-26 +## [0.3.1] 2021-03-26 ### Changed - IO target replaces dots in telemetry keys with dashes for better integration with AWS CloudWatch -## [0.3.0] - 2020-12-21 +## [0.3.0] 2020-12-21 ### Added - Datadog Target integration tests ### Fixed - Datadog Target -## [0.2.0] - 2020-12-21 +## [0.2.0] 2020-12-21 ### Fixed - Removed debugging information -## [0.1.0] - 2020-12-18 +## [0.1.0] 2020-12-18 ### Added - Core Plugin - Telemetry generation diff --git a/README.md b/README.md index d0b6f66..06ff085 100644 --- a/README.md +++ b/README.md @@ -41,22 +41,61 @@ Puma::Plugin::Telemetry.configure do |config| end ``` -### Basic +### Basic IO Target -Output telemetry as JSON to `STDOUT` +A basic I/O target will emit telemetry data to `STDOUT`, formatted in JSON. ```ruby - config.add_target :io +config.add_target :io ``` +#### Options + +This target has configurable `formatter:` and `transform:` options. +The `formatter:` options are + +* `:json` _(default)_ - Print the logs in JSON. +* `:logfmt` - Print the logs in key/value pairs, as per `logfmt`. +* `:noop` - A NOOP formatter which returns the telemetry `Hash` unaltered, passing it directly to the `io:` instance. + +The `transform:` options are + +* `:cloud_watch` _(default)_ - Transforms telemetry keys, replacing dots with dashes to support AWS CloudWatch Log Metrics filters. +* `:logfmt` - Transforms telemetry keys, prepending `sample#` for [L2Met][l2met] consumption. +* `:noop` - A NOOP transform which returns the telemetry `Hash` unaltered. + +### Log target + +Emitting to `STDOUT` via the basic `IOTarget` can work for getting telemetry into logs, we also provide an explicit `LogTarget`. +This target will defaults to emitting telemetry at the `INFO` log level via a [standard library `::Logger`][logger] instance. +That default logger will print to `STDOUT` in [the `logfmt` format][logfmt]. + +```ruby +config.add_target :log +``` + +You can pass an explicit `logger:` option if you wanted to, for example, use the same logger as Rails. + +```ruby +config.add_target :log, logger: Rails.logger +``` + +This target also has configurable `formatter:` and `transform:` options. +The [possible options are the same as for the `IOTarget`](#options), but the defaults are different. +The `LogTarget` defaults to `formatter: :logfmt`, and `transform: :noop` + +[l2met]: https://github.com/ryandotsmith/l2met?tab=readme-ov-file#l2met "l2met - Logs to Metrics" +[logfmt]: https://brandur.org/logfmt "logfmt - Structured log format" +[logger]: https://rubyapi.org/o/logger "Ruby's Logger, from the stdlib" + ### Datadog StatsD target -Given gem provides built in target for Datadog StatsD client, that uses batch operation to publish metrics. +A target for the Datadog StatsD client, that uses batch operation to publish metrics. -**NOTE** Be sure to have `dogstatsd` gem installed. +**NOTE** Be sure to have the `dogstatsd` gem installed. ```ruby - config.add_target :dogstatsd, client: Datadog::Statsd.new +config.add_target :dogstatsd, client: Datadog::Statsd.new ``` You can provide all the tags, namespaces, and other configuration options as always to `Datadog::Statsd.new` method. @@ -75,6 +114,7 @@ Puma::Plugin::Telemetry.configure do |config| config.socket_parser = :inspect config.add_target :io, formatter: :json, io: StringIO.new config.add_target :dogstatsd, client: Datadog::Statsd.new(tags: { env: ENV["RAILS_ENV"] }) + config.add_target :log, logger: Rails.logger, formatter: :logfmt, transform: :l2met) end ``` @@ -85,8 +125,8 @@ Target is a simple object that implements `call` methods that accepts `telemetry Just be mindful that if the API takes long to call, it will slow down frequency with which telemetry will get reported. ```ruby - # Example logfmt to stdout target - config.add_target proc { |telemetry| puts telemetry.map { |k, v| "#{k}=#{v.inspect}" }.join(" ") } + # Example key/value log to `STDOUT` target + config.add_target ->(telemetry) { puts telemetry.map { |k, v| "#{k}=#{v.inspect}" }.join(" ") } ``` ## Extra middleware diff --git a/lib/puma/plugin/telemetry.rb b/lib/puma/plugin/telemetry.rb index 235ecb2..31dd8b2 100644 --- a/lib/puma/plugin/telemetry.rb +++ b/lib/puma/plugin/telemetry.rb @@ -7,6 +7,7 @@ require 'puma/plugin/telemetry/data' require 'puma/plugin/telemetry/targets/datadog_statsd_target' require 'puma/plugin/telemetry/targets/io_target' +require 'puma/plugin/telemetry/targets/log_target' require 'puma/plugin/telemetry/config' module Puma diff --git a/lib/puma/plugin/telemetry/config.rb b/lib/puma/plugin/telemetry/config.rb index 1f26718..b066148 100644 --- a/lib/puma/plugin/telemetry/config.rb +++ b/lib/puma/plugin/telemetry/config.rb @@ -32,7 +32,8 @@ class Config TARGETS = { dogstatsd: Telemetry::Targets::DatadogStatsdTarget, - io: Telemetry::Targets::IOTarget + io: Telemetry::Targets::IOTarget, + log: Telemetry::Targets::LogTarget }.freeze # Whenever telemetry should run with puma diff --git a/lib/puma/plugin/telemetry/formatters/json_formatter.rb b/lib/puma/plugin/telemetry/formatters/json_formatter.rb new file mode 100644 index 0000000..a299408 --- /dev/null +++ b/lib/puma/plugin/telemetry/formatters/json_formatter.rb @@ -0,0 +1,19 @@ +# frozen_string_literal: true + +require 'json' + +module Puma + class Plugin + module Telemetry + module Formatters + # JSON formatter, expects `call` method accepting telemetry hash + # + class JSONFormatter + def self.call(telemetry) + ::JSON.dump(telemetry) + end + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/formatters/logfmt_formatter.rb b/lib/puma/plugin/telemetry/formatters/logfmt_formatter.rb new file mode 100644 index 0000000..36c1e03 --- /dev/null +++ b/lib/puma/plugin/telemetry/formatters/logfmt_formatter.rb @@ -0,0 +1,17 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Formatters + # Logfmt formatter, expects `call` method accepting telemetry hash + # + class LogfmtFormatter + def self.call(telemetry) + telemetry.map { |k, v| "#{String(k)}=#{v.inspect}" }.join(' ') + end + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/formatters/noop_formatter.rb b/lib/puma/plugin/telemetry/formatters/noop_formatter.rb new file mode 100644 index 0000000..11bd0e3 --- /dev/null +++ b/lib/puma/plugin/telemetry/formatters/noop_formatter.rb @@ -0,0 +1,16 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Formatters + # A NOOP formatter - it returns the telemetry Hash it was given + class NoopFormatter + def self.call(telemetry) + telemetry + end + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/targets/base_formatting_target.rb b/lib/puma/plugin/telemetry/targets/base_formatting_target.rb new file mode 100644 index 0000000..329fac5 --- /dev/null +++ b/lib/puma/plugin/telemetry/targets/base_formatting_target.rb @@ -0,0 +1,47 @@ +# frozen_string_literal: true + +require_relative '../formatters/json_formatter' +require_relative '../formatters/logfmt_formatter' +require_relative '../formatters/noop_formatter' +require_relative '../transforms/cloud_watch_transform' +require_relative '../transforms/l2met_transform' +require_relative '../transforms/noop_transform' + +module Puma + class Plugin + module Telemetry + module Targets + # A base class for other Targets concerned with formatting telemetry + # + class BaseFormattingTarget + def initialize(formatter: :json, transform: :cloud_watch) + @formatter = FORMATTERS.fetch(formatter) { formatter } + @transform = TRANSFORMS.fetch(transform) { transform } + end + + def call(_telemetry) + raise "#{__method__} must be implemented by #{self.class.name}" + end + + private + + attr_reader :formatter, :transform + + FORMATTERS = { + json: Formatters::JSONFormatter, + logfmt: Formatters::LogfmtFormatter, + noop: Formatters::NoopFormatter + }.freeze + private_constant :FORMATTERS + + TRANSFORMS = { + cloud_watch: Transforms::CloudWatchTranform, + l2met: Transforms::L2metTransform, + noop: Transforms::NoopTransform + }.freeze + private_constant :TRANSFORMS + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/targets/io_target.rb b/lib/puma/plugin/telemetry/targets/io_target.rb index 3272393..760f630 100644 --- a/lib/puma/plugin/telemetry/targets/io_target.rb +++ b/lib/puma/plugin/telemetry/targets/io_target.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require 'json' +require_relative 'base_formatting_target' module Puma class Plugin @@ -8,33 +8,19 @@ module Telemetry module Targets # Simple IO Target, publishing metrics to STDOUT or logs # - class IOTarget - # JSON formatter for IO, expects `call` method accepting telemetry hash - # - class JSONFormatter - # NOTE: Replace dots with dashes for better support of AWS CloudWatch - # Log Metric filters, as they don't support dots in key names. - def self.call(telemetry) - log = telemetry.transform_keys { |k| k.tr('.', '-') } - - log['name'] = 'Puma::Plugin::Telemetry' - log['message'] = 'Publish telemetry' - - ::JSON.dump(log) - end - end - - def initialize(io: $stdout, formatter: :json) + class IOTarget < BaseFormattingTarget + def initialize(io: $stdout, formatter: :json, transform: :cloud_watch) + super(formatter: formatter, transform: transform) @io = io - @formatter = case formatter - when :json then JSONFormatter - else formatter - end end def call(telemetry) - @io.puts(@formatter.call(telemetry)) + io.puts(formatter.call(transform.call(telemetry))) end + + private + + attr_reader :io end end end diff --git a/lib/puma/plugin/telemetry/targets/log_target.rb b/lib/puma/plugin/telemetry/targets/log_target.rb new file mode 100644 index 0000000..ea501f6 --- /dev/null +++ b/lib/puma/plugin/telemetry/targets/log_target.rb @@ -0,0 +1,30 @@ +# frozen_string_literal: true + +require 'logger' +require_relative 'base_formatting_target' + +module Puma + class Plugin + module Telemetry + module Targets + # Simple Log Target, publishing metrics to a Ruby ::Logger at stdout + # at the INFO log level + # + class LogTarget < BaseFormattingTarget + def initialize(logger: ::Logger.new($stdout), formatter: :logfmt, transform: :noop) + super(formatter: formatter, transform: transform) + @logger = logger + end + + def call(telemetry) + logger.info(formatter.call(transform.call(telemetry))) + end + + private + + attr_reader :logger + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/transforms/cloud_watch_transform.rb b/lib/puma/plugin/telemetry/transforms/cloud_watch_transform.rb new file mode 100644 index 0000000..0c44788 --- /dev/null +++ b/lib/puma/plugin/telemetry/transforms/cloud_watch_transform.rb @@ -0,0 +1,23 @@ +# frozen_string_literal: true + +require 'json' + +module Puma + class Plugin + module Telemetry + module Transforms + # Replace dots with dashes for better support of AWS CloudWatch Log + # Metric filters, as they don't support dots in key names. + # Expects `call` method accepting telemetry Hash + class CloudWatchTranform + def self.call(telemetry) + telemetry.transform_keys { |k| String(k).tr('.', '-') }.tap do |data| + data['name'] = 'Puma::Plugin::Telemetry' + data['message'] = 'Publish telemetry' + end + end + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/transforms/l2met_transform.rb b/lib/puma/plugin/telemetry/transforms/l2met_transform.rb new file mode 100644 index 0000000..48abcb3 --- /dev/null +++ b/lib/puma/plugin/telemetry/transforms/l2met_transform.rb @@ -0,0 +1,46 @@ +# frozen_string_literal: true + +require 'English' +require 'pathname' + +module Puma + class Plugin + module Telemetry + module Transforms + # L2Met (Logs to Metrics) transform that makes all keys a `sample#` in the L2Met format. + class L2metTransform + def self.call(telemetry) + new.call(telemetry) + end + + def initialize(host_env: ENV, program_name: $PROGRAM_NAME, socket: Socket) + @host_env = host_env + @program_name = program_name + @socket = socket + end + + def call(telemetry) + telemetry.transform_keys { |k| "sample##{k}" }.tap do |data| + data['name'] ||= 'Puma::Plugin::Telemetry' + data['source'] ||= source + end + end + + private + + attr_reader :host_env, :program_name, :socket + + def source + @source ||= host_env['L2MET_SOURCE'] || + host_env['DYNO'] || # For Heroku + host_with_exe_name # Last-ditch effort + end + + def host_with_exe_name + "#{socket.gethostname}/#{Pathname(program_name).basename}" + end + end + end + end + end +end diff --git a/lib/puma/plugin/telemetry/transforms/noop_transform.rb b/lib/puma/plugin/telemetry/transforms/noop_transform.rb new file mode 100644 index 0000000..30c8690 --- /dev/null +++ b/lib/puma/plugin/telemetry/transforms/noop_transform.rb @@ -0,0 +1,18 @@ +# frozen_string_literal: true + +require 'json' + +module Puma + class Plugin + module Telemetry + module Transforms + # A NOOP Transform - it returns the telemetry Hash it was given + class NoopTransform + def self.call(telemetry) + telemetry + end + end + end + end + end +end diff --git a/spec/fixtures/sockets.rb b/spec/fixtures/sockets.rb index b6cfbf0..7261cc0 100644 --- a/spec/fixtures/sockets.rb +++ b/spec/fixtures/sockets.rb @@ -21,7 +21,7 @@ Puma::Plugin::Telemetry.configure do |config| # Simple `key=value` formatter - config.add_target :io, formatter: ->(t) { t.map { |r| r.join('=') }.join(' ') } + config.add_target(:io, formatter: :logfmt, transform: :noop) config.frequency = 1 config.enabled = true diff --git a/spec/puma/plugin/telemetry/formatters/json_formatter_spec.rb b/spec/puma/plugin/telemetry/formatters/json_formatter_spec.rb new file mode 100644 index 0000000..34ef629 --- /dev/null +++ b/spec/puma/plugin/telemetry/formatters/json_formatter_spec.rb @@ -0,0 +1,27 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Formatters + RSpec.describe JSONFormatter do + subject(:formatter) { described_class } + + it 'formats the telemetry as a JSON string' do + string = formatter.call('foo' => 'bar') + + data = ::JSON.parse(string) + expect(data.fetch('foo')).to eq('bar') + end + + it 'handles symbol keys' do + string = formatter.call(foo: 'bar') + + data = ::JSON.parse(string) + expect(data.fetch('foo')).to eq('bar') + end + end + end + end + end +end diff --git a/spec/puma/plugin/telemetry/formatters/logfmt_formatter_spec.rb b/spec/puma/plugin/telemetry/formatters/logfmt_formatter_spec.rb new file mode 100644 index 0000000..1d8be56 --- /dev/null +++ b/spec/puma/plugin/telemetry/formatters/logfmt_formatter_spec.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Formatters + RSpec.describe LogfmtFormatter do + subject(:formatter) { described_class } + + it 'formats the telemetry in key/value pairs' do + string = formatter.call('foo' => 'bar', 'count' => 2) + + expect(string).to include('foo="bar"').and(include('count=2')) + end + + it 'handles symbol keys' do + string = formatter.call(foo: 'bar') + + expect(string).to include('foo="bar"') + end + end + end + end + end +end diff --git a/spec/puma/plugin/telemetry/targets/io_target_spec.rb b/spec/puma/plugin/telemetry/targets/io_target_spec.rb new file mode 100644 index 0000000..8f21abd --- /dev/null +++ b/spec/puma/plugin/telemetry/targets/io_target_spec.rb @@ -0,0 +1,22 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Targets + RSpec.describe IOTarget do + subject(:target) { described_class.new(io: io, formatter: logfmt) } + let(:io) { StringIO.new } + let(:telemetry) { { foo: 'bar' } } + let(:logfmt) { ->(telemetry) { telemetry.map { |k, v| "#{k}=#{v}" }.join(' ') } } + + it 'puts to the io object' do + target.call(telemetry) + + expect(io.string).to include('foo=bar') + end + end + end + end + end +end diff --git a/spec/puma/plugin/telemetry/targets/log_target_spec.rb b/spec/puma/plugin/telemetry/targets/log_target_spec.rb new file mode 100644 index 0000000..59ee59f --- /dev/null +++ b/spec/puma/plugin/telemetry/targets/log_target_spec.rb @@ -0,0 +1,23 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Targets + RSpec.describe LogTarget do + subject(:target) { described_class.new(logger: logger, formatter: logfmt) } + let(:logger) { ::Logger.new(io) } + let(:io) { StringIO.new } + let(:telemetry) { { foo: 'bar' } } + let(:logfmt) { ->(telemetry) { telemetry.map { |k, v| "#{k}=#{v}" }.join(' ') } } + + it 'logs the telemetry at the INFO level' do + target.call(telemetry) + + expect(io.string).to include('INFO').and(include('foo=bar')) + end + end + end + end + end +end diff --git a/spec/puma/plugin/telemetry/transforms/cloud_watch_transform_spec.rb b/spec/puma/plugin/telemetry/transforms/cloud_watch_transform_spec.rb new file mode 100644 index 0000000..a102e0f --- /dev/null +++ b/spec/puma/plugin/telemetry/transforms/cloud_watch_transform_spec.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Transforms + RSpec.describe CloudWatchTranform do + subject(:transform) { described_class } + + it 'replaces dots with dashes in keys' do + data = transform.call('the.foo' => 'the.bar') + + expect(data.fetch('the-foo')).to eq('the.bar') + end + + it 'handles symbol keys' do + data = transform.call(foo: 'bar') + + expect(data.fetch('foo')).to eq('bar') + end + end + end + end + end +end diff --git a/spec/puma/plugin/telemetry/transforms/l2met_transform_spec.rb b/spec/puma/plugin/telemetry/transforms/l2met_transform_spec.rb new file mode 100644 index 0000000..786f2ad --- /dev/null +++ b/spec/puma/plugin/telemetry/transforms/l2met_transform_spec.rb @@ -0,0 +1,50 @@ +# frozen_string_literal: true + +module Puma + class Plugin + module Telemetry + module Transforms + RSpec.describe L2metTransform do + subject(:transform) { described_class.new(host_env: fake_env, socket: fake_socket) } + + let(:fake_env) { {} } + let(:fake_socket) { class_double('Socket', gethostname: 'GIBSON') } + + it 'transforms the telemetry key to an L2Met sample' do + data = transform.call('widgets.size' => 2, 'speed' => 10.5) + + expect(data).to include('sample#widgets.size' => 2, 'sample#speed' => 10.5) + end + + it 'handles symbol keys' do + data = transform.call('queue.depth': 2) + + expect(data).to include('sample#queue.depth' => 2) + end + + it 'adds source from L2MET_SOURCE in ENV' do + fake_env['L2MET_SOURCE'] = 'some-machine' + + data = transform.call('queue.depth' => 2) + + expect(data).to include('source' => 'some-machine') + end + + it 'adds source from DYNO in ENV' do + fake_env['DYNO'] = 'web.10' + + data = transform.call('queue.depth' => 2) + + expect(data).to include('source' => 'web.10') + end + + it 'adds source from $PROGRAM_NAME when L2MET_SOURCE nor DYNO are in ENV' do + data = transform.call('queue.depth' => 2) + + expect(data).to include('source' => 'GIBSON/rspec') + end + end + end + end + end +end