Skip to content

Commit 5be7e79

Browse files
authored
Refactor MAL codegen into focused classes, fix CI issues (#13763)
**1. Refactor MALClassGenerator (1484 → 5 focused classes)** Split by responsibility with clear comments showing which MAL expressions each class handles: | Class | Lines | Responsibility | |---|---|---| | MALClassGenerator | ~350 | Public API + orchestration | | MALExprCodegen | ~310 | Expression → Java source (variable-per-expression model) | | MALMethodChainCodegen | ~340 | Method chain + extension + argument codegen | | MALMetadataExtractor | ~370 | Static AST → metadata extraction | | MALBytecodeHelper | ~290 | Javassist class naming, debug, bytecode attrs | **2. Variable-per-expression codegen (replaces shared `sf` variable)** Each metric expression gets its own named variable (e.g. `_metric1`, `_metric2`). Chain calls reassign to the same variable. Binary operations combine variables directly. No save/restore dance. Extension `::` calls work everywhere including sub-expressions. Before: ```java SampleFamily sf; sf = ((SF) samples.getOrDefault("metric1", EMPTY)); sf = sf.sum(...); SampleFamily _t0 = sf; // save sf = ((SF) samples.getOrDefault("metric2", EMPTY)); sf = sf.avg(...); sf = _t0.plus(sf); // restore + combine return sf; ``` After: ```java SampleFamily _metric1 = ((SF) samples.getOrDefault("metric1", EMPTY)); _metric1 = _metric1.sum(...); SampleFamily _metric2 = ((SF) samples.getOrDefault("metric2", EMPTY)); _metric2 = _metric2.avg(...); _metric1 = _metric1.plus(_metric2); return _metric1; ``` **3. Unified codegen path** Removed the inline expression codegen path. Complex expression arguments are pre-computed to temp variables via the statement-based path. Single codegen strategy eliminates duplication. **4. Developer extension docs** New docs under Contributing Guides > Project Extensions: - `docs/en/guides/mal-extension.md` — MAL `namespace::method()` SPI for developers - `docs/en/guides/lal-extension.md` — LAL custom input/output types for developers - Renamed `source-extension.md` title to "OAL Extension" - Clear boundary: user docs (`concepts-and-designs/`) for script syntax, developer docs (`guides/`) for SPI implementation **5. Other improvements** - Extract `generateFilterTestBody()` shared by `compileFilter()` and `generateFilterSource()` - Fail fast on unknown inline expression types - Fix upload-artifact v4 conflict (`overwrite: true`) - Fix Node.js 20 deprecation (`FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true`, checkout v4→v6)
1 parent 0e46059 commit 5be7e79

16 files changed

Lines changed: 1837 additions & 1286 deletions

File tree

.github/workflows/skywalking.yaml

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ concurrency:
2626
cancel-in-progress: true
2727

2828
env:
29+
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
2930
SW_AGENT_JDK_VERSION: 8
3031
MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120
3132
SEGMENT_DOWNLOAD_TIMEOUT_MINS: 5 # Cache restore timeout
@@ -37,7 +38,7 @@ jobs:
3738
runs-on: ubuntu-latest
3839
timeout-minutes: 10
3940
steps:
40-
- uses: actions/checkout@v4
41+
- uses: actions/checkout@v6
4142
with:
4243
submodules: true
4344
persist-credentials: false
@@ -50,7 +51,7 @@ jobs:
5051
runs-on: ubuntu-latest
5152
timeout-minutes: 10
5253
steps:
53-
- uses: actions/checkout@v4
54+
- uses: actions/checkout@v6
5455
with:
5556
submodules: true
5657
persist-credentials: false
@@ -66,7 +67,7 @@ jobs:
6667
runs-on: ubuntu-latest
6768
timeout-minutes: 30
6869
steps:
69-
- uses: actions/checkout@v4
70+
- uses: actions/checkout@v6
7071
with:
7172
submodules: true
7273
persist-credentials: false
@@ -115,7 +116,7 @@ jobs:
115116
pom: ${{ steps.filter.outputs.pom }}
116117
ui: ${{ steps.filter.outputs.ui }}
117118
steps:
118-
- uses: actions/checkout@v4 # required for push event
119+
- uses: actions/checkout@v6 # required for push event
119120
with:
120121
fetch-depth: 0
121122
submodules: true
@@ -166,7 +167,7 @@ jobs:
166167
runs-on: ubuntu-latest
167168
timeout-minutes: 30
168169
steps:
169-
- uses: actions/checkout@v4
170+
- uses: actions/checkout@v6
170171
with:
171172
submodules: true
172173
persist-credentials: false
@@ -201,7 +202,7 @@ jobs:
201202
matrix:
202203
java-version: [11, 17, 25]
203204
steps:
204-
- uses: actions/checkout@v4
205+
- uses: actions/checkout@v6
205206
with:
206207
submodules: true
207208
persist-credentials: false
@@ -248,7 +249,7 @@ jobs:
248249
- os: ubuntu-latest
249250
java-version: 25
250251
steps:
251-
- uses: actions/checkout@v4
252+
- uses: actions/checkout@v6
252253
with:
253254
submodules: true
254255
persist-credentials: false
@@ -277,7 +278,7 @@ jobs:
277278
matrix:
278279
java-version: [11, 17, 21, 25]
279280
steps:
280-
- uses: actions/checkout@v4
281+
- uses: actions/checkout@v6
281282
with:
282283
submodules: true
283284
persist-credentials: false
@@ -312,7 +313,7 @@ jobs:
312313
- name: ElasticSearch / OpenSearch
313314
class: org.apache.skywalking.library.elasticsearch.ElasticSearchIT
314315
steps:
315-
- uses: actions/checkout@v4
316+
- uses: actions/checkout@v6
316317
with:
317318
submodules: true
318319
persist-credentials: false
@@ -713,7 +714,7 @@ jobs:
713714
- name: Pprof MySQL
714715
config: test/e2e-v2/cases/profiling/pprof/mysql/e2e.yaml
715716
steps:
716-
- uses: actions/checkout@v4
717+
- uses: actions/checkout@v6
717718
with:
718719
submodules: true
719720
persist-credentials: false
@@ -771,6 +772,7 @@ jobs:
771772
with:
772773
name: test-logs-${{ matrix.test.name }}
773774
path: "${{ env.SW_INFRA_E2E_LOG_DIR }}"
775+
overwrite: true
774776

775777
e2e-test-istio:
776778
if: |
@@ -805,7 +807,7 @@ jobs:
805807
kubernetes: 28
806808

807809
steps:
808-
- uses: actions/checkout@v4
810+
- uses: actions/checkout@v6
809811
with:
810812
submodules: true
811813
persist-credentials: false
@@ -847,6 +849,7 @@ jobs:
847849
with:
848850
name: test-logs-${{ matrix.test.name }}
849851
path: "${{ env.SW_INFRA_E2E_LOG_DIR }}"
852+
overwrite: true
850853

851854
e2e-test-istio-ambient:
852855
if: |
@@ -876,7 +879,7 @@ jobs:
876879
- istio: 1.29.0
877880
kubernetes: 28
878881
steps:
879-
- uses: actions/checkout@v4
882+
- uses: actions/checkout@v6
880883
with:
881884
submodules: true
882885
persist-credentials: false
@@ -918,6 +921,7 @@ jobs:
918921
with:
919922
name: test-istio-ambient-logs-${{ matrix.versions.istio }}-${{ matrix.versions.kubernetes }}-${{ matrix.analyzer }}
920923
path: "${{ env.SW_INFRA_E2E_LOG_DIR }}"
924+
overwrite: true
921925

922926
e2e-test-java-versions:
923927
if: |
@@ -932,7 +936,7 @@ jobs:
932936
matrix:
933937
java-version: [11, 17, 25]
934938
steps:
935-
- uses: actions/checkout@v4
939+
- uses: actions/checkout@v6
936940
with:
937941
submodules: true
938942
persist-credentials: false
@@ -986,7 +990,7 @@ jobs:
986990
- name: BanyanDB Stages
987991
config: test/e2e-v2/cases/storage/banyandb/stages/e2e.yaml
988992
steps:
989-
- uses: actions/checkout@v4
993+
- uses: actions/checkout@v6
990994
with:
991995
submodules: true
992996
persist-credentials: false
@@ -1073,6 +1077,7 @@ jobs:
10731077
with:
10741078
name: test-logs-${{ matrix.test.name }}
10751079
path: "${{ env.SW_INFRA_E2E_LOG_DIR }}"
1080+
overwrite: true
10761081

10771082
required:
10781083
if: always()

docs/en/concepts-and-designs/lal.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -611,8 +611,15 @@ See bundled LAL scripts for complete examples:
611611

612612
### Extending: custom output types
613613

614-
You can create custom output types to transform logs into any entity type for your own use cases. There are two approaches:
614+
For developers who need to create custom output types (implementing `LALOutputBuilder`, extending
615+
`AbstractLog`, registering `LALSourceTypeProvider` SPI, or defining custom input types), see the
616+
[LAL Extension Developer Guide](../guides/lal-extension.md).
615617

618+
<!--
619+
Developer-facing content below is kept for reference but the canonical guide is lal-extension.md.
620+
-->
621+
622+
<!--
616623
#### Approach 1: Implement `LALOutputBuilder`
617624
618625
Use this when you need full control over validation and dispatching, or when the output is not a simple `AbstractLog` subclass.
@@ -716,3 +723,4 @@ public class MyLayerSourceTypeProvider implements LALSourceTypeProvider {
716723
```
717724
718725
Register it in `META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LALSourceTypeProvider`.
726+
-->

docs/en/concepts-and-designs/mal.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,12 +248,13 @@ Extension functions are called with a `::` separator between the namespace and m
248248

249249
```
250250
metric.sum(['svc']).myext::transform(2.0)
251-
metric.genai::estimateCost()
252251
```
253252

254253
The `::` separator distinguishes extension calls from built-in `SampleFamily` methods (like `.sum()`,
255254
`.tag()`, `.filter()`). Method names only need to be unique within their namespace.
256255

256+
Available extension namespaces depend on which extension modules are deployed with the OAP server.
257+
257258
### Supported Parameter Types
258259

259260
Extension methods accept the following argument types from MAL expressions:
@@ -275,6 +276,11 @@ The MAL compiler validates extension calls at compile time:
275276
- Mismatched argument count results in a compilation error.
276277
- Type mismatches between MAL arguments and Java parameter types result in a compilation error.
277278

279+
### Building Extensions
280+
281+
For developers who want to create custom MAL extension functions, see the
282+
[MAL Extension Developer Guide](../guides/mal-extension.md).
283+
278284
## Down Sampling Operation
279285
MAL should instruct meter-system on how to downsample for metrics. It doesn't only refer to aggregate raw samples to
280286
`minute` level, but also expresses data from `minute` in higher levels, such as `hour` and `day`.

docs/en/guides/lal-extension.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# LAL Extension: Custom Input/Output Types for Developers
2+
3+
LAL (Log Analysis Language) supports custom input and output types for extending log processing
4+
beyond the built-in `Log` type. This enables receiver plugins and custom modules to define
5+
domain-specific log entities (e.g., slow SQL records, sampled traces, network profiling logs).
6+
7+
## Overview
8+
9+
LAL provides two extension mechanisms:
10+
11+
1. **`outputType` per rule** — set in YAML config to transform logs into custom entities
12+
2. **`LALSourceTypeProvider` SPI** — register default input/output types for a receiver plugin
13+
14+
Both are documented in detail in the [LAL user guide](../concepts-and-designs/lal.md#output-type),
15+
including:
16+
- Built-in output types (`SlowSQL`, `SampledTrace`)
17+
- Creating custom output types (Source subclass or `LALOutputBuilder` interface)
18+
- Custom input types for non-standard log formats
19+
- SPI registration for receiver-level defaults
20+
21+
## Quick Reference
22+
23+
### Per-rule output type (YAML config)
24+
25+
```yaml
26+
rules:
27+
- name: slow_sql
28+
outputType: SlowSQL
29+
dsl: |
30+
filter { ... }
31+
extractor {
32+
statement log.body
33+
}
34+
sink { }
35+
```
36+
37+
### LALSourceTypeProvider SPI (receiver plugin default)
38+
39+
```java
40+
public class MyLayerSourceTypeProvider implements LALSourceTypeProvider {
41+
@Override
42+
public String layer() { return "MY_LAYER"; }
43+
44+
@Override
45+
public Class<?> inputType() { return MyProtoMessage.class; }
46+
47+
@Override
48+
public Class<? extends Source> outputType() { return MyCustomBuilder.class; }
49+
}
50+
```
51+
52+
Register in `META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LALSourceTypeProvider`.
53+
54+
For complete examples and implementation details, see the
55+
[Output Type section in the LAL documentation](../concepts-and-designs/lal.md#output-type).

docs/en/guides/mal-extension.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# MAL Extension Functions for Developers
2+
3+
MAL (Meter Analysis Language) supports custom extension functions callable from scripts using the
4+
`namespace::method()` syntax. Extensions are discovered at startup via Java `ServiceLoader`, requiring
5+
no changes to the MAL compiler or grammar.
6+
7+
## Creating an Extension
8+
9+
### 1. Implement `MalFunctionExtension`
10+
11+
Create a class that implements the SPI interface and add static methods annotated with `@MALContextFunction`:
12+
13+
```java
14+
package com.example;
15+
16+
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
17+
import org.apache.skywalking.oap.meter.analyzer.v2.spi.MALContextFunction;
18+
import org.apache.skywalking.oap.meter.analyzer.v2.spi.MalFunctionExtension;
19+
20+
public class MyExtension implements MalFunctionExtension {
21+
@Override
22+
public String name() {
23+
return "myext"; // namespace used in MAL scripts
24+
}
25+
26+
@MALContextFunction
27+
public static SampleFamily scale(SampleFamily sf, double factor) {
28+
return sf.multiply(Double.valueOf(factor));
29+
}
30+
31+
@MALContextFunction
32+
public static SampleFamily filterByTag(SampleFamily sf, String key, String value) {
33+
return sf.tagEqual(key, value);
34+
}
35+
}
36+
```
37+
38+
### 2. Register via SPI
39+
40+
Create the file `META-INF/services/org.apache.skywalking.oap.meter.analyzer.v2.spi.MalFunctionExtension`:
41+
```
42+
com.example.MyExtension
43+
```
44+
45+
### 3. Use in MAL scripts
46+
47+
```yaml
48+
metricsRules:
49+
- name: scaled_metric
50+
exp: metric.sum(['svc']).myext::scale(2.0)
51+
- name: filtered_metric
52+
exp: metric.myext::filterByTag("env", "prod").sum(['svc'])
53+
```
54+
55+
## Method Requirements
56+
57+
- Methods **must** be `static`
58+
- First parameter **must** be `SampleFamily` (auto-bound to the current chain value in the expression)
59+
- Return type **must** be `SampleFamily`
60+
- Non-static or invalid methods throw `IllegalArgumentException` at startup
61+
- Duplicate namespace names throw `IllegalArgumentException` at startup
62+
- Duplicate method names within the same namespace throw `IllegalArgumentException` at startup
63+
64+
## Supported Parameter Types
65+
66+
| Java Type | MAL Argument | Example |
67+
|-----------|-------------|---------|
68+
| `String` | String literal | `"value"` |
69+
| `double` | Number literal | `2.0` |
70+
| `float` | Number literal | `3.0` |
71+
| `long` | Number literal | `100` |
72+
| `int` | Number literal | `10` |
73+
| `List<String>` | String list | `["tag1", "tag2"]` |
74+
75+
Only `List<String>` is supported for list parameters. Other generic list types (e.g., `List<Integer>`)
76+
are rejected at startup.
77+
78+
## How It Works
79+
80+
The MAL compiler generates **direct static method calls** at compile time — no reflection at runtime.
81+
Each metric gets a named variable (e.g., `_metric`), and extension calls are emitted as static calls
82+
on the variable:
83+
84+
For `metric.sum(['svc']).myext::scale(2.0)`:
85+
```java
86+
SampleFamily _metric = ((SampleFamily) samples.getOrDefault("metric", SampleFamily.EMPTY));
87+
_metric = _metric.sum(java.util.Arrays.asList(new String[]{"svc"}));
88+
_metric = com.example.MyExtension.scale(_metric, 2.0);
89+
return _metric;
90+
```
91+
92+
## Compile-Time Validation
93+
94+
The compiler validates at expression compilation time:
95+
- Namespace exists in the SPI registry
96+
- Method exists in that namespace
97+
- Argument count matches (excluding the implicit `SampleFamily` first parameter)
98+
- Argument types are compatible with the method signature
99+
100+
Any validation failure results in a compilation error with a clear message.

docs/en/guides/source-extension.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Source and scope extension for new metrics
1+
# OAL Extension: Source and Scope for New Metrics
22
From the [OAL scope introduction](../concepts-and-designs/oal.md#from), you should already have understood what a scope is.
33
If you would like to create more extensions, you need to have a deeper understanding of what a **source** is.
44

docs/menu.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -430,8 +430,12 @@ catalog:
430430
catalog:
431431
- name: "Component Library Settings"
432432
path: "/en/guides/component-library-settings"
433-
- name: "Extend An OAL Source"
433+
- name: "OAL Source and Scope Extension"
434434
path: "/en/guides/source-extension"
435+
- name: "MAL Extension Functions"
436+
path: "/en/guides/mal-extension"
437+
- name: "LAL Custom Input/Output Types"
438+
path: "/en/guides/lal-extension"
435439
- name: "Dependencies"
436440
catalog:
437441
- name: "Project Dependencies Check"

0 commit comments

Comments
 (0)