Replies: 4 comments 9 replies
-
|
Thanks for preparing this document, @weiqingy. However, this doc is more like a coding plan, which contains lots of implementation details like how to modify each file and how to implement the desired behavior. I think what we need now is a more high-level design, focusing on the definition of concepts, user-facing interfaces and expected behaviors. IMO, the following questions should be clearly answered, some of which seem missing from the current doc (or swamped in the massive implementation details).
Additionally, I have a few more comments concerning the design:
|
Beta Was this translation helpful? Give feedback.
-
|
If max-length is a total size cap, the question becomes: how do we allocate the budget across fields? Proportional allocation, equal split, largest-first greedy - each has trade-offs, and the result is hard for users to predict. The config looks simple (one number), but the behavior is opaque. I'm rethinking about this - per-field thresholds seems actually the cleaner model. Each threshold independently controls one truncation strategy, so the behavior is predictable, e.g.,: These compose by walking the object graph - array trimming reduces element count, then string truncation caps leaf string values within the retained structure, and depth collapsing handles deeply nested objects. For example, given a ChatRequestEvent with 50 messages and a 10K-char response:
Truncated fields use the wrapper format you suggested for parsability: Each threshold has clear, independent behavior - no opaque budget allocation. |
Beta Was this translation helpful? Give feedback.
-
|
I think you're right. The proposed 3 options have likely covered most cases that may cause large event size and are worth truncating. And it's definitely easier to implement and to understand. I initially thought that having only one option might be easier for the users to tune and to get a predictable event size. But after a second thought, I think maybe an absolute predictable event size is not necessary. In cases where people run into huge event logs, it's likely caused by one or multiple of long list, long string and deep hierarchy issues. So the proposed options should be good enough to solve users' problem. |
Beta Was this translation helpful? Give feedback.
-
|
@xintongsong I've updated the design doc to reflect our discussion above. Please take a look and let me know if there are any remaining concerns. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
One more minor comments. I think we should add some infix between Other than that, LGTM. I think we can move to the implementation. |
Beta Was this translation helpful? Give feedback.
-
|
@xintongsong Good catch - you're right that This cleanly separates per-type settings from built-in options: Will move forward with implementation. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @weiqingy, thanks for this design. Regarding the matters I care about—such as how users configure log levels, what configuration options are available, the schema of log records, and the interplay between log levels and event filters—I think everything is now clear. As for the migration to language-independent events, I think my expectations align with this design. We will use type strings to distinguish events instead of the current FQCNs. This will affect the keys used when configuring log levels, key matching (comparing type strings rather than inspecting FQCNs), and how hierarchy inheritance is adapted. I think this design has been considered thoroughly enough. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the review, @wenjin272 ! |
Beta Was this translation helpful? Give feedback.
-
|
@alnzng As the original author of the current event logging mechanism, would you like to also take a look at this design? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
GitHub issue: #541
Motivation
The event log captures every event flowing through an agent for debugging, auditing, and observability. Today there is no mechanism to control the verbosity of logged events. This makes it impossible to:
This design introduces per-event-type configurable log levels so that operators can independently control the verbosity of each event type.
Log Levels
Three log levels, ordered from least to most verbose:
OFFSTANDARDVERBOSEThe default level for all event types is
STANDARDwith truncation active. This meansSTANDARDandVERBOSEhave distinct behaviors out of the box.Why Default to STANDARD with Active Truncation
We chose active truncation because:
STANDARDandVERBOSEmean different things from day one. No configuration required to see the distinction.VERBOSE.event-log.level: VERBOSEto restore the previous behavior, or setting specific event types toVERBOSEfor full detail where needed.Configuration
Config Key Pattern
Per-event-type settings use the pattern:
The
typeinfix separates per-event-type settings from built-in config options (likeevent-log.standard.*). Without it, a custom event type prefixed withstandard(e.g.,standard.MyEvent) would create ambiguous keys. The event type appears in the middle, and the property name appears at the tail. This structure groups all settings for a given event type together and allows future per-type properties (e.g., routing events to different logger destinations) without restructuring the key namespace. This follows standard hierarchical logger configuration conventions.Future extensibility example:
Event Type Names in Config Keys
Config keys use the fully-qualified class name of the event type. This avoids ambiguity when different packages define event classes with the same simple name.
Hierarchical Inheritance
Log level resolution follows hierarchy inheritance. The dot-separated event type name defines a natural hierarchy. When an event type has no exact config match, the level is inherited from the nearest configured ancestor.
The root config key
event-log.levelserves as the global default — no specialdefaultkeyword is needed.Resolution order (most specific wins):
event-log.type.org.apache.flink.agents.api.event.ChatRequestEvent.levelevent-log.type.org.apache.flink.agents.api.event.levelevent-log.type.org.apache.flink.agents.api.levelevent-log.levelSTANDARD(ifevent-log.levelis not configured)Example: Given these event types:
And this config:
Resolution:
...api.event.ChatRequestEventVERBOSE...api.event.ToolRequestEventOFF...api.event...api.InputEventSTANDARD...apikey → inherits from root...api.OutputEventSTANDARDComplete Config Key Reference
event-log.levelOFF/STANDARD/VERBOSE)STANDARDevent-log.type.<EVENT_TYPE>.levelOFF/STANDARD/VERBOSE)event-log.standard.max-string-length2000STANDARDlevel. Strings exceeding this limit are truncated.0disables string truncation. Has no effect atVERBOSElevel.event-log.standard.max-array-elements20STANDARDlevel. Arrays exceeding this limit are trimmed.0disables array trimming. Has no effect atVERBOSElevel.event-log.standard.max-depth5STANDARDlevel. Objects nested beyond this depth are collapsed.0disables depth collapsing. Has no effect atVERBOSElevel.Configuration Examples
Config file:
The config key uses whatever type string appears in the event log's
eventTypefield. For Java events, that's the Java FQCN (e.g.,org.apache.flink.agents.api.event.ChatRequestEvent). For Python events, that's the Python module path (e.g.,flink_agents.api.events.event.OutputEvent). The hierarchy inheritance works the same way for both — it walks up the dot-separated segments.Known limitations of the current model:
OutputEventand PythonOutputEventare the same concept, but they have different type strings (org.apache.flink.agents.api.OutputEventvsflink_agents.api.events.event.OutputEvent). There is no single config key that covers both.event-log.type.org.apache.flink.agents.api.event.level: OFFsilences all Java events in that package, but equivalent Python events are unaffected.org.apache..., Python withflink_agents.... The only shared ancestor is the rootevent-log.level, which is too broad for targeted control.These limitations are acceptable for the initial release because most jobs today are either pure Java or pure Python. See Migration to Language-Independent Events for how these limitations are resolved when events become language-independent.
Override at job submission time:
Truncation Strategy (STANDARD Level)
At
STANDARDlevel, events may be truncated according to three independently configurable per-field thresholds. Each threshold controls one truncation strategy with clear, predictable behavior. Truncation never applies atVERBOSElevel.Per-Field Thresholds
The three thresholds compose by walking the serialized event's object graph:
max-string-length(default:2000) — Any leaf string value exceeding this length is truncated. This most commonly affects LLM response text, tool call arguments, and tool response bodies.max-array-elements(default:20) — Arrays with more elements than this limit are trimmed to retain only the first N elements.max-depth(default:5) — Object structures nested beyond this depth are collapsed.The thresholds compose by walking the object graph: array trimming reduces element count, then string truncation caps leaf string values within the retained structure, and depth collapsing handles deeply nested objects.
Example: Given a
ChatRequestEventwith 50 messages and a 10K-char response:max-array-elements: 20trimsmessagesfrom 50 to 20 elementsmax-string-length: 2000truncates thecontentfield inside each retained message (e.g., a 5000-char content → 2000 chars), and truncates the 10K-charresponseto 2000 charsmodel,role,requestIdare untouchedEach threshold has clear, independent behavior — no opaque budget allocation across fields.
Setting any threshold to
0disables that specific truncation strategy. Setting all three to0makesSTANDARDbehave identically toVERBOSE(except for the metadata label).Truncation Wrapper Format
Truncated fields are wrapped in metadata objects to keep the output as valid, parseable JSON. This enables downstream tooling to detect and reason about truncation programmatically.
Wrapper formats by type:
{"truncatedString": "retained content...", "omittedChars": N}{"truncatedList": [retained elements...], "omittedElements": N}{"truncatedObject": {retained fields...}, "omittedFields": N}This means truncated fields change type (e.g., a
stringfield becomes anobject). Consumers parsing event logs atSTANDARDlevel should check for wrapper objects. Consumers needing a stable schema should useVERBOSElevel, which never truncates.What Does NOT Get Truncated
Structural and identifying fields are always preserved in full:
eventType,id,attributes,timestampEvent Log Record Schema
This section describes the JSON schema of each record written to the event log file. Two new top-level fields (
logLevel,eventType) are added. Users and downstream tools that parse event log files should be aware of these additions.Records include top-level
logLevelandeventTypefields:{ "timestamp": "2024-01-15T10:30:00Z", "logLevel": "VERBOSE", "eventType": "org.apache.flink.agents.api.event.ChatRequestEvent", "event": { "eventType": "org.apache.flink.agents.api.event.ChatRequestEvent", "id": "...", "attributes": {}, "model": "gpt-4", "messages": [ {"role": "system", "content": "You are a helpful assistant..."}, {"role": "user", "content": "Analyze this document..."} ] } }At
STANDARDlevel with truncation applied:{ "timestamp": "2024-01-15T10:30:00Z", "logLevel": "STANDARD", "eventType": "org.apache.flink.agents.api.event.ChatRequestEvent", "event": { "eventType": "org.apache.flink.agents.api.event.ChatRequestEvent", "id": "...", "attributes": {}, "model": "gpt-4", "messages": { "truncatedList": [ {"role": "system", "content": "You are a helpful assistant..."}, {"role": "user", "content": {"truncatedString": "Analyze this doc...", "omittedChars": 1000}}, {"role": "assistant", "content": {"truncatedString": "Based on my...", "omittedChars": 3000}} ], "omittedElements": 30 } } }The
eventTypefield is emitted at the top level (alongsidetimestamp) for convenient downstream filtering without needing to parse into theeventobject.Old records without
logLevelor top-leveleventTypeare deserialized correctly, defaulting toVERBOSE(since they were written before log levels existed and contain full untruncated content).EventFilter Removal
The existing
EventFiltermechanism (EventFilter.java) is removed as part of this design. The log level system fully subsumesEventFilter's functionality:ACCEPT_ALL(default)event-log.level: STANDARD(default)REJECT_ALLevent-log.level: OFFbyEventType(ChatRequestEvent.class)event-log.level: OFF, thenevent-log.type.<ChatRequestEvent>.level: STANDARDLog levels provide additional capabilities that
EventFiltercannot:-Dflags (no Java code required)The only capability lost is custom
accept(Event, EventContext)logic that filters by event content or context (e.g., filtering by key or attribute values). This is acceptable because:EventFilterlogicEventFilterunless they implement their own event logger, which is unlikely in practiceFiles affected:
api/src/main/java/org/apache/flink/agents/api/EventFilter.javaapi/src/main/java/org/apache/flink/agents/api/logger/EventLoggerConfig.java— removeeventFilterfield andBuilder.eventFilter()methodruntime/src/main/java/org/apache/flink/agents/runtime/eventlog/FileEventLogger.java— removeeventFilter.accept()check inappend()runtime/src/test/java/org/apache/flink/agents/runtime/eventlog/FileEventLoggerTest.java— remove EventFilter-related testsValidation
No upfront validation of configured event type names is performed. If a configured type string doesn't match any event at runtime, it simply has no effect — the hierarchy fallback resolves the level from the nearest ancestor or the root default. This matches how standard logging frameworks like Log4j handle unknown logger names.
Rationale: Not all event types are known at initialization time. Users may define custom events that are only instantiated at runtime via
ctx.sendEvent(). Strict upfront validation would produce false warnings for valid custom event types. The hierarchy fallback is safe — a misconfigured type name silently inherits from its parent, and events still get logged at the appropriate level.Observability
When truncation is active, a counter metric
eventLogTruncatedEventsis incremented each time an event is truncated. This helps operators decide whether to adjust truncation thresholds or switch specific event types toVERBOSE.Backward Compatibility
STANDARDwith active truncation. This is a behavior change from today — events atSTANDARDlevel may be truncated. To restore previous behavior, setevent-log.level: VERBOSE.logLevelor top-leveleventTypefields deserialize correctly, defaulting toVERBOSE(old records contain full untruncated content).EventFilteris removed (see EventFilter Removal). This is acceptable for 0.x pre-release.Migration to Language-Independent Events
There is ongoing discussion about changing events to language-independent JSON objects to simplify custom event definitions, especially for cross-language use cases where users currently need to define the same event type in both Java and Python.
Current Model (This Design)
Config keys use the event's type string as-is — Java FQCNs for Java events, Python module paths for Python events:
This has known limitations in mixed-language jobs (see Configuration Examples), but is acceptable for the initial release because most jobs today are either pure Java or pure Python.
Future Model (Language-Independent Events)
If events become plain JSON with a user-chosen type string (e.g.,
"ChatRequestEvent","OutputEvent"), the config keys simplify and the cross-language limitations disappear:Migration Plan
When language-independent events are adopted:
eventTypefield in log records would change from FQCNs/module paths to plain type strings. Config keys follow automatically since they are based on theeventTypevalue.event-log.levelstill serves as the global default. If the community adopts a naming convention with dots (e.g.,chat.request,tool.response), hierarchy inheritance continues to work.Design Decision
This design targets the current model (Java FQCNs + Python module paths) for the initial implementation. The
event-log.type.<TYPE>.levelconfig key pattern and hierarchy inheritance mechanism are compatible with both the current and future models — only the type strings that users write in config files would change during migration.Beta Was this translation helpful? Give feedback.
All reactions