-
Notifications
You must be signed in to change notification settings - Fork 40
chore: added experimental OTLP trace exporter support #2534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
abhilash-sivan
wants to merge
130
commits into
main
Choose a base branch
from
chore-otel-span-mapper
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
130 commits
Select commit
Hold shift + click to select a range
9058592
chore: playground done
kirrg001 dc8bcd9
chore: infra cor
kirrg001 062bdc7
chore: phase 1
abhilash-sivan 79db188
chore: phase 2
abhilash-sivan 65a6346
chore: phase 3
abhilash-sivan 73c6747
chore: cleanup
abhilash-sivan 88339a3
chore: rewrite
abhilash-sivan f491602
chore: test
abhilash-sivan f828557
chore: a
abhilash-sivan 90c4036
chore: new fields
abhilash-sivan 5881174
chore: test
abhilash-sivan 3b1c1d7
test: fixed failing test
abhilash-sivan dc1d0ac
chore: cleanup
abhilash-sivan a9eadb4
chore: ui corrected
abhilash-sivan 17b0e96
chore: added kafka and pg, generalize
abhilash-sivan 93432b7
chore: update
abhilash-sivan e1db00a
chore: fix
abhilash-sivan b446627
chore: clean
abhilash-sivan c53e8c5
chore: update
abhilash-sivan 4b24288
chore: added readme
abhilash-sivan 0b8f5ec
chore: added example
abhilash-sivan bbe510f
chore: added example transformer
abhilash-sivan 1bf37de
chore: update readme to minimal
abhilash-sivan 83d38f6
chore: update logic
abhilash-sivan 31f2d33
chore: spanBuffer uses temporary converter
abhilash-sivan 2ada04a
chore: move files for using inside spanBuffer
abhilash-sivan d814ae5
chore: update
abhilash-sivan e17f7a6
chore: rearrange
abhilash-sivan 34dccf6
chore: added separate transformer
abhilash-sivan ddabdce
chore: added example otel spans
abhilash-sivan f64f968
chore: make getSpanType dynamic
abhilash-sivan e1b05e7
chore: fix status
abhilash-sivan 3150adf
chore: kafka systemName
abhilash-sivan bb0ba34
chore: v1 of multiple data key logic
abhilash-sivan ddb816e
chore: refactor
abhilash-sivan 0e62e28
chore: optimize additioanl attributes
abhilash-sivan 1f9a6aa
chore: wip more dynamic
abhilash-sivan b6857b2
chore: refatcor transformers
abhilash-sivan 63d6b16
chore: refactor main file
abhilash-sivan 2f8ae5a
chore: new otel span type file
abhilash-sivan f86ba13
chore: cleanup otlp transformer
abhilash-sivan c65abdc
chore: minor refactor for pr comment
abhilash-sivan c99ab4b
chore: udpate example app
abhilash-sivan 6f59ead
chore: refactor 1
abhilash-sivan 1f6d78e
chore: same format
abhilash-sivan bd43972
chore: remove circular dependency
abhilash-sivan a3c14ca
chore: use fn instead of string
abhilash-sivan 53be07a
chore: rearrange
abhilash-sivan 6ea5b7d
chore: refactor clean
abhilash-sivan 9c32500
chore: refactor
abhilash-sivan 0f3946e
chore: fix refactorig issues
abhilash-sivan 43a00bb
chore: rearrage
abhilash-sivan f7350f9
chore: resolve priority span type
abhilash-sivan 916e062
chore: updated structure
abhilash-sivan b9efdf9
chore: remove parentSpanContext for now
abhilash-sivan a2cfd61
chore: add version specific directory
abhilash-sivan f95968f
chore: service name resolution
abhilash-sivan 5884202
chore: instr
aryamohanan 24c8460
chore: updated the structure
aryamohanan b52c415
chore: tested kafka+ mongo
aryamohanan bcb7c4d
chore: updated
aryamohanan 3801dcd
chore: skip log spans
aryamohanan d405719
chore: fine tuning
aryamohanan 059bf5f
chore: fine tuning
aryamohanan 25613ac
chore: renaming
aryamohanan 94ec592
chore: added events and links
aryamohanan 5b53b99
chore: revert
aryamohanan 25f5dec
chore: updated
aryamohanan 24af53b
chore: more claenup
aryamohanan 9e6eafe
chore: readme update
aryamohanan 0cf6f8c
chore: cleanup and test
aryamohanan cfc3fd5
chore: updated span status logic
aryamohanan 577b333
chore: added some cleanup
aryamohanan abbe261
chore: metrics
aryamohanan 97cfc76
chore: fixed timestap and duration
abhilash-sivan c63ed0e
chore: restructure
aryamohanan 5fdba8c
chore: deleted unwanted files
aryamohanan 35da81d
chore: more cleanup
aryamohanan e1de12d
test: updated
aryamohanan 97c1bd5
chore: moved examples
aryamohanan 9397df9
chore: fine tuning
aryamohanan 29ea7cd
chore: metrics
aryamohanan 9e4bb49
chore: agentconnection
aryamohanan 76c3062
chore: agentconnection
aryamohanan 1d63335
chore: metrics handled
aryamohanan ef39a22
chore: trcaes handled
aryamohanan 5731615
chore: review comments
aryamohanan 59fda36
chore: resource
aryamohanan 106e017
chore: updated
aryamohanan 97e97b7
chore: updated
aryamohanan e3ae1c6
chore: updated codebase
aryamohanan fe54baa
chore: updated
aryamohanan dce8d7d
chore: handled dual transformation
abhilash-sivan fc574d3
chore: corrected mappings from internal format
abhilash-sivan 8daec24
test: fix tests
abhilash-sivan 1c47ac7
test: added azure blob test
abhilash-sivan eadce21
chore: improved metrics
aryamohanan 760c939
chore: improved metrics
aryamohanan ef0c426
chore: test
aryamohanan 2faa117
chore: test
aryamohanan e3578d1
chore: test
aryamohanan 4c1b0e1
chore: updated
aryamohanan c687c02
chore: updated
aryamohanan 8982b47
chore: updated
aryamohanan ee8f0df
chore: fine tuning
aryamohanan 615dcb5
chore: removed comments from agent conn
aryamohanan b93c790
chore: fixed tests
aryamohanan d56d1ae
chore: updated
aryamohanan 9f917f9
chore: handled error.type
abhilash-sivan 21d70b1
chore: config
aryamohanan 5cbc32f
chore: updated
aryamohanan 849e549
chore: updated
aryamohanan 05c8bb0
chore: cleanup
aryamohanan 1ecb7f4
chore: fine tuning
aryamohanan 607774a
chore: fine tuning
aryamohanan c803916
chore: resource
aryamohanan 0b054c3
chore: resource
aryamohanan d11d895
chore: updated http span mapping for host and port
abhilash-sivan 7da20dd
chore: clean code
aryamohanan c6482c2
chore: clean metrics
aryamohanan af86a7d
chore: clean code
aryamohanan bbfe2e2
chore: updated
aryamohanan d075b5a
chore: renamed
aryamohanan 0a22947
chore: renamed
aryamohanan c21fd18
chore: updated
aryamohanan 23a4651
chore: updated
aryamohanan 32df20e
chore: updated
aryamohanan 75b67c5
chore: updated
aryamohanan e4f95a4
chore: updated mapping
aryamohanan 7c47812
chore: updated
aryamohanan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| node_modules/ | ||
| data.txt | ||
| npm-debug.log | ||
| .DS_Store |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| # OpenTelemetry Exporter Test App | ||
|
|
||
| A comprehensive Express.js application for testing OpenTelemetry tracing with Instana backend, including HTTP, PostgreSQL, and Kafka instrumentation. | ||
|
|
||
| ## Features | ||
|
|
||
| - **HTTP tracing**: Express.js REST API with external HTTP calls | ||
| - **PostgreSQL tracing**: Database queries with pg driver | ||
| - **Kafka tracing**: Message producer and consumer | ||
| - **OpenTelemetry auto-instrumentation**: Automatic tracing for all operations | ||
| - **OTLP HTTP exporter**: Configured for Instana backend | ||
| - **Debug logging**: Console output for spans | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. **PostgreSQL** running on `localhost:5432` | ||
| - Database: `nodedb` | ||
| - User: `node` | ||
| - Password: `nodepw` | ||
|
|
||
| 2. **Kafka** running on `localhost:9092` | ||
| - Topic: `test-topic` (will be created automatically) | ||
|
|
||
| 3. **Instana** account with OTLP endpoint access | ||
|
|
||
| ## Setup | ||
|
|
||
| ### 1. Install Dependencies | ||
|
|
||
| ```bash | ||
| npm install | ||
| ``` | ||
|
|
||
| ### 2. Configure PostgreSQL | ||
|
|
||
| ```bash | ||
| # Create database and user | ||
| psql -U postgres | ||
| CREATE DATABASE nodedb; | ||
| CREATE USER node WITH PASSWORD 'nodepw'; | ||
| GRANT ALL PRIVILEGES ON DATABASE nodedb TO node; | ||
| ``` | ||
|
|
||
| ### 3. Configure Kafka | ||
|
|
||
| Make sure Kafka is running on `localhost:9092`. If using Docker: | ||
|
|
||
| ```bash | ||
| docker run -d --name kafka \ | ||
| -p 9092:9092 \ | ||
| -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \ | ||
| -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ | ||
| confluentinc/cp-kafka:latest | ||
| ``` | ||
|
|
||
| ### 4. Update Instana Configuration | ||
|
|
||
| Edit `tracing.js` and update: | ||
| - `url`: Your Instana OTLP endpoint | ||
| - `x-instana-key`: Your Instana API key | ||
|
|
||
| ### 5. Start the Application | ||
|
|
||
| ```bash | ||
| npm start | ||
| ``` | ||
|
|
||
| Or for development with otel: | ||
|
|
||
| ```bash | ||
| npm run start-otel | ||
| ``` | ||
|
|
||
| ## API Endpoints | ||
|
|
||
| ### 1. HTTP Entry + HTTP Exit | ||
|
|
||
| Tests HTTP client instrumentation with external API call. | ||
|
|
||
| ```bash | ||
| curl http://localhost:3000/external-api | ||
| ``` | ||
|
|
||
| **Expected trace:** | ||
| - HTTP server span (Express) | ||
| - HTTP client span (fetch to jsonplaceholder.typicode.com) | ||
|
|
||
| ### 2. HTTP Entry + PostgreSQL Exit | ||
|
|
||
| Tests PostgreSQL database instrumentation. | ||
|
|
||
| ```bash | ||
| curl http://localhost:3000/db | ||
| ``` | ||
|
|
||
| **Expected trace:** | ||
| - HTTP server span (Express) | ||
| - PostgreSQL query span | ||
|
|
||
| ### 3. HTTP Entry + Kafka Exit | ||
|
|
||
| Tests Kafka producer instrumentation. | ||
|
|
||
| ```bash | ||
| curl -X POST http://localhost:3000/kafka \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{"message":"Hello from OpenTelemetry!"}' | ||
| ``` | ||
|
|
||
| **Expected trace:** | ||
| - HTTP server span (Express) | ||
| - Kafka producer span | ||
| - Kafka consumer span (async, separate trace) | ||
|
|
||
| ## Tracing Details | ||
|
|
||
| The application automatically traces: | ||
|
|
||
| ### HTTP Operations | ||
| - Express route handlers | ||
| - Outgoing HTTP requests (fetch/axios) | ||
| - Request/response details | ||
|
|
||
| ### PostgreSQL Operations | ||
| - SQL queries | ||
| - Connection details | ||
| - Query parameters | ||
|
|
||
| ### Kafka Operations | ||
| - Message production | ||
| - Message consumption | ||
| - Topic and partition information | ||
| - Custom attributes via hooks | ||
|
|
||
| ## Viewing Traces | ||
|
|
||
| 1. Make requests to the API endpoints | ||
| 2. Check console output for span details | ||
| 3. View traces in your Instana dashboard | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### PostgreSQL Connection Issues | ||
|
|
||
| ```bash | ||
| # Check if PostgreSQL is running | ||
| pg_isready -h localhost -p 5432 | ||
|
|
||
| # Test connection | ||
| psql -h localhost -U node -d nodedb | ||
| ``` | ||
|
|
||
| ### Kafka Connection Issues | ||
|
|
||
| ```bash | ||
| # Check if Kafka is running | ||
| nc -zv localhost 9092 | ||
|
|
||
| # List topics | ||
| kafka-topics --list --bootstrap-server localhost:9092 | ||
| ``` | ||
|
|
||
| ### OpenTelemetry Issues | ||
|
|
||
| - Check console output for initialization messages | ||
| - Verify OTLP endpoint is accessible | ||
| - Ensure API key is correct | ||
| - Set log level to `DiagLogLevel.DEBUG` in `tracing.js` for more details | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Disable Console Exporter | ||
|
|
||
| In `tracing.js`, comment out: | ||
|
|
||
| ```javascript | ||
| // provider.addSpanProcessor(new SimpleSpanProcessor(consoleExporter)); | ||
| ``` | ||
|
|
||
| ### Change Service Name | ||
|
|
||
| In `tracing.js`, update: | ||
|
|
||
| ```javascript | ||
| [SemanticResourceAttributes.SERVICE_NAME]: 'your-service-name' | ||
| ``` | ||
|
|
||
| ### Disable Specific Instrumentations | ||
|
|
||
| In `tracing.js`, modify the `getNodeAutoInstrumentations` options: | ||
|
|
||
| ```javascript | ||
| getNodeAutoInstrumentations({ | ||
| '@opentelemetry/instrumentation-fs': { enabled: false }, | ||
| '@opentelemetry/instrumentation-dns': { enabled: false } | ||
| }) | ||
| ``` | ||
|
|
||
| ## Docker Setup (Optional) | ||
|
|
||
| For running PostgreSQL and Kafka in Docker: | ||
|
|
||
| ```bash | ||
| # PostgreSQL | ||
| docker run -d --name postgres \ | ||
| -e POSTGRES_USER=node \ | ||
| -e POSTGRES_PASSWORD=nodepw \ | ||
| -e POSTGRES_DB=nodedb \ | ||
| -p 5432:5432 \ | ||
| postgres:15 | ||
|
|
||
| # Kafka (requires Zookeeper) | ||
| docker-compose up -d | ||
| ``` | ||
|
|
||
| Create a `docker-compose.yml`: | ||
|
|
||
| ```yaml | ||
| version: '3' | ||
| services: | ||
| zookeeper: | ||
| image: confluentinc/cp-zookeeper:latest | ||
| environment: | ||
| ZOOKEEPER_CLIENT_PORT: 2181 | ||
|
|
||
| kafka: | ||
| image: confluentinc/cp-kafka:latest | ||
| depends_on: | ||
| - zookeeper | ||
| ports: | ||
| - "9092:9092" | ||
| environment: | ||
| KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | ||
| KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 | ||
| KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| /* | ||
| * (c) Copyright IBM Corp. 2026 | ||
| */ | ||
|
|
||
| 'use strict'; | ||
|
|
||
| const express = require('express'); | ||
| const { Pool } = require('pg'); | ||
| const { sendKafkaMessage, startKafkaConsumer } = require('./kafka'); | ||
|
|
||
| const app = express(); | ||
| app.use(express.json()); | ||
|
|
||
| const PORT = 3000; | ||
|
|
||
| const pool = new Pool({ | ||
| host: 'localhost', | ||
| port: 5432, | ||
| user: 'node', | ||
| password: 'nodepw', | ||
| database: 'nodedb' | ||
| }); | ||
|
|
||
| // --------------------------------------------------- | ||
| // 1. HTTP ENTRY + HTTP EXIT | ||
| // --------------------------------------------------- | ||
|
|
||
| app.get('/external-api', async (req, res) => { | ||
| try { | ||
| const response = await fetch('https://jsonplaceholder.typicode.com/todos/1'); | ||
| const data = await response.json(); | ||
|
|
||
| res.json({ | ||
| success: true, | ||
| data | ||
| }); | ||
| } catch (err) { | ||
| // eslint-disable-next-line no-console | ||
| console.error(err); | ||
|
|
||
| res.status(500).json({ | ||
| error: err.message | ||
| }); | ||
| } | ||
| }); | ||
|
|
||
| // --------------------------------------------------- | ||
| // 2. HTTP ENTRY + PG EXIT | ||
| // --------------------------------------------------- | ||
|
|
||
| app.get('/db', async (req, res) => { | ||
| try { | ||
| const result = await pool.query('SELECT NOW() as current_time'); | ||
|
|
||
| res.json({ | ||
| success: true, | ||
| rows: result.rows | ||
| }); | ||
| } catch (err) { | ||
| // eslint-disable-next-line no-console | ||
| console.error(err); | ||
|
|
||
| res.status(500).json({ | ||
| error: err.message | ||
| }); | ||
| } | ||
| }); | ||
|
|
||
| // --------------------------------------------------- | ||
| // 3. HTTP ENTRY + KAFKA EXIT | ||
| // --------------------------------------------------- | ||
|
|
||
| app.post('/kafka', async (req, res) => { | ||
| try { | ||
| const payload = req.body || { | ||
| hello: 'world' | ||
| }; | ||
|
|
||
| await sendKafkaMessage(payload); | ||
|
|
||
| res.json({ | ||
| success: true, | ||
| sent: payload | ||
| }); | ||
| } catch (err) { | ||
| // eslint-disable-next-line no-console | ||
| console.error(err); | ||
|
|
||
| res.status(500).json({ | ||
| error: err.message | ||
| }); | ||
| } | ||
| }); | ||
|
|
||
| // --------------------------------------------------- | ||
| // START | ||
| // --------------------------------------------------- | ||
|
|
||
| app.listen(PORT, async () => { | ||
| // eslint-disable-next-line no-console | ||
| console.log(`Server running on port ${PORT}`); | ||
|
|
||
| await startKafkaConsumer(); | ||
| }); | ||
|
|
||
| // Made with Bob | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need this app still?
Wondering if we can just have a separate npm script in the example-apps/collector app?
With config otlp being on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can remove this after finishing the playground work, since the example app also includes Kafka and database tests that are only required for playground testing.