|
| 1 | +# Metadata |
| 2 | + |
| 3 | +[Logger metadata](`Logger#module-metadata`) is structured data attached to log |
| 4 | +messages. Metadata can be passed to `Logger` with the message: |
| 5 | + |
| 6 | +```elixir |
| 7 | +Logger.info("Hello", user_id: 123) |
| 8 | +``` |
| 9 | + |
| 10 | +or set beforehand with `Logger.metadata/1`: |
| 11 | + |
| 12 | +```elixir |
| 13 | +Logger.metadata(user_id: 123) |
| 14 | +``` |
| 15 | + |
| 16 | +> #### Not Just Keyword List {: .tip} |
| 17 | +> |
| 18 | +> Big Logger don't want you to know, but maps totally work as metadata too: |
| 19 | +> |
| 20 | +> ```elixir |
| 21 | +> iex> Logger.metadata(%{hello: "world"}) |
| 22 | +> :ok |
| 23 | +> iex> Logger.metadata() |
| 24 | +> [hello: "world"] |
| 25 | +> ``` |
| 26 | +
|
| 27 | +Metadata is tricky to use correctly because it behaves very differently in |
| 28 | +development versus production environments. In development, the default console |
| 29 | +logger outputs minimal metadata, and even when configured to show more, console |
| 30 | +space is limited, so developers are pushed to embed important data directly in |
| 31 | +log messages. Production logging solutions, however, very much prefer structured |
| 32 | +metadata for filtering and searching, paired with static log messages that |
| 33 | +enable effective fingerprinting and grouping of similar events. |
| 34 | +
|
| 35 | +This guide focuses on the latter approach: using static log messages paired with |
| 36 | +rich metadata: |
| 37 | +
|
| 38 | +```elixir |
| 39 | +Logger.error("Unexpected API response", status_code: 422, user_id: 123) |
| 40 | +``` |
| 41 | +
|
| 42 | +When working with metadata, logging libraries typically grapple with two key |
| 43 | +challenges: serialization and scrubbing. |
| 44 | + |
| 45 | +## Serialization |
| 46 | + |
| 47 | +Metadata can hold Elixir terms of any type, but to send them somewhere and |
| 48 | +display them to users, they must be serialized. Unfortunately, there's no |
| 49 | +universally good way to handle this! Elixir's default |
| 50 | +`Logger.Formatter#module-metadata` supports only a handful of types. The |
| 51 | +de-facto expectation, however, is that specialized logging libraries can handle |
| 52 | +any term and display it reasonably well. Consequently, every logging library |
| 53 | +implements a step where it makes the hard decisions about what to do with |
| 54 | +tuples, structs, and other complex data types. This process is sometimes called |
| 55 | +encoding or sanitization. |
| 56 | + |
| 57 | +One solution that works well and can be easily integrated into your project is |
| 58 | +the |
| 59 | +[`LoggerJSON.Formatters.RedactorEncoder.encode/2`](https://hexdocs.pm/logger_json/LoggerJSON.Formatter.RedactorEncoder.html#encode/2) |
| 60 | +function. It accepts any Elixir term and makes it JSON-serializable: |
| 61 | + |
| 62 | +```elixir |
| 63 | +iex> LoggerJSON.Formatter.RedactorEncoder.encode(%{tuple: {:ok, "foo"}, pid: self()}, []) |
| 64 | +%{pid: "#PID<0.219.0>", tuple: [:ok, "foo"]} |
| 65 | +``` |
| 66 | + |
| 67 | +## Scrubbing |
| 68 | + |
| 69 | +Scrubbing is the process of removing sensitive fields from metadata. Data like |
| 70 | +passwords, API keys, or credit card numbers should never be sent to your logging |
| 71 | +service unnecessarily. While many logging services implement scrubbing on the |
| 72 | +receiving end, some libraries handle this on the client side as well. |
| 73 | + |
| 74 | +The challenge with scrubbing is that it must be configurable. Applications store |
| 75 | +diverse types of secrets, and no set of default rules can catch them all. |
| 76 | +Fortunately, the same solution used for serialization works here too. |
| 77 | +[`LoggerJSON.Formatters.RedactorEncoder.encode/2`](https://hexdocs.pm/logger_json/LoggerJSON.Formatter.RedactorEncoder.html#encode/2) |
| 78 | +accepts a list of "redactors" that will be called to scrub potentially sensitive |
| 79 | +data. It includes a powerful |
| 80 | +[`LoggerJSON.Redactors.RedactKeys`](https://hexdocs.pm/logger_json/LoggerJSON.Redactors.RedactKeys.html) |
| 81 | +redactor that redacts all values stored under specified keys: |
| 82 | + |
| 83 | +```elixir |
| 84 | +iex> LoggerJSON.Formatter.RedactorEncoder.encode( |
| 85 | + %{user: "Marion", password: "SCP-3125"}, |
| 86 | + [{LoggerJSON.Redactors.RedactKeys, ["password"]}] |
| 87 | +) |
| 88 | +%{user: "Marion", password: "[REDACTED]"} |
| 89 | +``` |
| 90 | + |
| 91 | +## Conclusion |
| 92 | + |
| 93 | +While [`LoggerJSON`](https://hex.pm/packages/logger_json)'s primary goal isn't to solve our metadata struggles, it |
| 94 | +comes with a set of tools that can be very handy. Even if you don't want to |
| 95 | +depend on it directly, it can provide you with a good starting point for your |
| 96 | +own solution. And `LoggerHandlerKit.Act.metadata_serialization/1` can help you |
| 97 | +with test cases! |
0 commit comments