Skip to content

Commit 5962fc4

Browse files
committed
Add Metadata guide
1 parent 2789943 commit 5962fc4

2 files changed

Lines changed: 99 additions & 1 deletion

File tree

guides/metadata.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Metadata
2+
3+
[Logger metadata](`Logger#module-metadata`) is structured data attached to log
4+
messages. Metadata can be passed to `Logger` with the message:
5+
6+
```elixir
7+
Logger.info("Hello", user_id: 123)
8+
```
9+
10+
or set beforehand with `Logger.metadata/1`:
11+
12+
```elixir
13+
Logger.metadata(user_id: 123)
14+
```
15+
16+
> #### Not Just Keyword List {: .tip}
17+
>
18+
> Big Logger don't want you to know, but maps totally work as metadata too:
19+
>
20+
> ```elixir
21+
> iex> Logger.metadata(%{hello: "world"})
22+
> :ok
23+
> iex> Logger.metadata()
24+
> [hello: "world"]
25+
> ```
26+
27+
Metadata is tricky to use correctly because it behaves very differently in
28+
development versus production environments. In development, the default console
29+
logger outputs minimal metadata, and even when configured to show more, console
30+
space is limited, so developers are pushed to embed important data directly in
31+
log messages. Production logging solutions, however, very much prefer structured
32+
metadata for filtering and searching, paired with static log messages that
33+
enable effective fingerprinting and grouping of similar events.
34+
35+
This guide focuses on the latter approach: using static log messages paired with
36+
rich metadata:
37+
38+
```elixir
39+
Logger.error("Unexpected API response", status_code: 422, user_id: 123)
40+
```
41+
42+
When working with metadata, logging libraries typically grapple with two key
43+
challenges: serialization and scrubbing.
44+
45+
## Serialization
46+
47+
Metadata can hold Elixir terms of any type, but to send them somewhere and
48+
display them to users, they must be serialized. Unfortunately, there's no
49+
universally good way to handle this! Elixir's default
50+
`Logger.Formatter#module-metadata` supports only a handful of types. The
51+
de-facto expectation, however, is that specialized logging libraries can handle
52+
any term and display it reasonably well. Consequently, every logging library
53+
implements a step where it makes the hard decisions about what to do with
54+
tuples, structs, and other complex data types. This process is sometimes called
55+
encoding or sanitization.
56+
57+
One solution that works well and can be easily integrated into your project is
58+
the
59+
[`LoggerJSON.Formatters.RedactorEncoder.encode/2`](https://hexdocs.pm/logger_json/LoggerJSON.Formatter.RedactorEncoder.html#encode/2)
60+
function. It accepts any Elixir term and makes it JSON-serializable:
61+
62+
```elixir
63+
iex> LoggerJSON.Formatter.RedactorEncoder.encode(%{tuple: {:ok, "foo"}, pid: self()}, [])
64+
%{pid: "#PID<0.219.0>", tuple: [:ok, "foo"]}
65+
```
66+
67+
## Scrubbing
68+
69+
Scrubbing is the process of removing sensitive fields from metadata. Data like
70+
passwords, API keys, or credit card numbers should never be sent to your logging
71+
service unnecessarily. While many logging services implement scrubbing on the
72+
receiving end, some libraries handle this on the client side as well.
73+
74+
The challenge with scrubbing is that it must be configurable. Applications store
75+
diverse types of secrets, and no set of default rules can catch them all.
76+
Fortunately, the same solution used for serialization works here too.
77+
[`LoggerJSON.Formatters.RedactorEncoder.encode/2`](https://hexdocs.pm/logger_json/LoggerJSON.Formatter.RedactorEncoder.html#encode/2)
78+
accepts a list of "redactors" that will be called to scrub potentially sensitive
79+
data. It includes a powerful
80+
[`LoggerJSON.Redactors.RedactKeys`](https://hexdocs.pm/logger_json/LoggerJSON.Redactors.RedactKeys.html)
81+
redactor that redacts all values stored under specified keys:
82+
83+
```elixir
84+
iex> LoggerJSON.Formatter.RedactorEncoder.encode(
85+
%{user: "Marion", password: "SCP-3125"},
86+
[{LoggerJSON.Redactors.RedactKeys, ["password"]}]
87+
)
88+
%{user: "Marion", password: "[REDACTED]"}
89+
```
90+
91+
## Conclusion
92+
93+
While [`LoggerJSON`](https://hex.pm/packages/logger_json)'s primary goal isn't to solve our metadata struggles, it
94+
comes with a set of tools that can be very handy. Even if you don't want to
95+
depend on it directly, it can provide you with a good starting point for your
96+
own solution. And `LoggerHandlerKit.Act.metadata_serialization/1` can help you
97+
with test cases!

mix.exs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,8 @@ defmodule LoggerHandlerKit.MixProject do
4646
extras: [
4747
"README.md",
4848
"guides/translation.md",
49-
"guides/unhandled.md"
49+
"guides/unhandled.md",
50+
"guides/metadata.md"
5051
],
5152
groups_for_modules: [
5253
Helpers: [

0 commit comments

Comments
 (0)