Skip to content

Commit 4420d0a

Browse files
authored
feat: add mentions sink to snowflake (#3789)
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent 428fa7f commit 4420d0a

1 file changed

Lines changed: 40 additions & 0 deletions

File tree

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
DESCRIPTION >
2+
Sink pipe to export Octolens mentions data to Kafka for Snowflake ingestion.
3+
Reads from mentions datasource using FINAL to get deduplicated data.
4+
Runs daily at 00:30 UTC, exporting all data (full snapshot).
5+
6+
NODE octolens_mentions_select_fields
7+
SQL >
8+
SELECT
9+
sourceId,
10+
url,
11+
timestamp,
12+
source,
13+
author,
14+
authorProfileLink,
15+
title,
16+
body,
17+
imageUrl,
18+
relevanceScore,
19+
relevanceComment,
20+
keyword,
21+
sentimentLabel,
22+
subreddit,
23+
viewId,
24+
viewName,
25+
arrayStringConcat(viewKeywords, ',') as viewKeywords,
26+
language,
27+
projectSlug,
28+
createdAt,
29+
bookmarked,
30+
arrayStringConcat(keywords, ',') as keywords,
31+
toStartOfDay(now()) as date
32+
FROM mentions FINAL
33+
34+
TYPE SINK
35+
EXPORT_SERVICE kafka
36+
EXPORT_CONNECTION_NAME lfx-oracle-kafka-streaming
37+
EXPORT_SCHEDULE 30 0 * * *
38+
EXPORT_FORMAT csv
39+
EXPORT_STRATEGY @new
40+
EXPORT_KAFKA_TOPIC octolens_mentions_sink

0 commit comments

Comments
 (0)