feat: migrate changelog fetching from Deadlock forums to Steam API by LVL1024 · Pull Request #401 · deadlock-wiki/deadbot

LVL1024 · 2026-07-03T17:13:35Z

Here is the updated PR description in a markdown codeblock, ready to copy and paste:

Changed from using the forums to the Steam API since the devs don't post patch notes on the forums anymore, they moved to Steam posts. Also removed hero labs code.

What changed:

fetch_forum_changelogs() → fetch_steam_changelogs(), reads from steam_community_announcements filtered to patchnotes-tagged posts
_bbcode_to_text() converts Steam BBCode to plain text (paragraphs, lists, links, images, embeds, plus a catch-all for unknown tags)
MD5 hashing (steam_hash) detects when Valve edits a post after publishing. If it changes, the local .txt file is updated silently, and hotfix detection is adjusted to prevent false positives.
STEAM_MIGRATION_DATE cut-off (2026-06-04) keeps all historical forum-sourced data intact
forum_id → source_id in changelog_configs.json; title, steam_hash fields added
feedparser and bs4 dropped from pyproject.toml; tzdata added
Removed Hero Lab changelog parsing: The get_gamefile_changelogs() pipeline (~130 lines) parsed citadel_patch_notes_english.json into _HeroLab entries. Devs haven't used Hero Labs in forever.
Rewrote the changelog wikitext formatter to group entries under == Items == and == Heroes == sections (with === Hero/Item === and ==== Ability ==== sub-headings) instead of flat bullet points, matching the wiki's Patch Formatter output.
Added hide_general to parse_changelogs.py to suppress the General heading on patches with no unclassified bullets.
Added Unstable Rift and Rift Trooper to link_targets.json.

Parsed data in deadlock-data PR - Reopen deadbot PR or run deploy workflow for this branch here to reparse the data

… subheaders, and hotfix preservation

… ability subheaders, and hotfix preservation" This reverts commit b7dd29c.

…pdates fetching

Reverts the 3 commits since 26fe3d9: - b1994dc feat: exclude Hero Lab entries from reprocessing - 9b8de0e TEST: Delete existing notes - 7f2e144 feat: enhance changelog processing with pending ID support Preserves the DRY_RUN env var forwarding fix in docker-compose.yml, which fixes a pre-existing bug where the variable was set by the CI workflow but silently dropped at the container boundary.

hassayag

Sorry I haven't gotten to this sooner. Great to see this has worked nicely. Main thing I am concerned with is getting rid of any old code that was designed specifically for the old RSS feed

hassayag · 2026-07-12T22:19:46Z

+            is_steam_link = (
+                'store.steampowered.com/news/' in source_link
+                or 'steamstore-a.akamaihd.net/news/' in source_link
+                or 'steamcommunity.com/app/' in source_link
+            )


Is there any circumstance that the link would not be steam if we are pointing to the steam rss feed?

Any changelog before 2026-06-04 used the old forum links. But to keep things clean, I just made the parser skip generating wikitext for anything before that migration date entirely.

hassayag · 2026-07-12T22:25:28Z

+                                'is_hero_lab': False,
+                                'title': e.get('title'),
+                                'steam_hash': e['steam_hash'],
+                                'was_edited': True,


Interested to know how we would handle edited updates. Would it function with our current hotfix system?

I removed the manual was_edited flag entirely. Now, if Valve edits a post, the bot detects the hash change and silently updates the local .txt file. To prevent false hotfixes when overwriting, I set it so the newly fetched text is treated as the "old text" for comparison. The hotfix system still runs fine, and the wiki stays safe because the uploader skips overwriting the main page (while still appending genuinely new hotfix sections via the separate appender).

hassayag · 2026-07-12T22:26:32Z

                    final_text += f'\n\n=== Patch {patch_num} ===\n\n{entry_text}'
                    logger.debug(f'Adding Patch {patch_num} to {date_key}')

-            # Detect hotfixes - compare new patches against old content


Please be observant what your AI is spitting out, they often strip comments

Fixed, will check it better in the future.

hassayag · 2026-07-12T22:34:46Z

                link = existing_config.get('link')
+                title = existing_config.get('title')
+                steam_hash = existing_config.get('steam_hash')
            elif append_entries:


Correct me if I'm wrong as I'm not that familiar with the changelog side

Is append_entries a remnant of the old forums, which had a main forum post with additional updates added as comments? If so, have you seen a similar thing with their use of steam news?

It was originally for forum comments. But we still need it for Steam in case Valve posts multiple patch notes on the same day. Since the wiki only has one page per date, we still use append_entries to merge them under === Patch N === headers.

hassayag · 2026-07-12T22:36:36Z

+                source_id = main_entry['version']
                link = main_entry['link']
+                title = main_entry.get('title')
+                steam_hash = main_entry.get('steam_hash')
            elif existing_config:
+                source_id = existing_config.get('source_id')
                link = existing_config.get('link')
+                title = existing_config.get('title')
+                steam_hash = existing_config.get('steam_hash')
            elif append_entries:
+                source_id = append_entries[0]['version']
                link = append_entries[0]['link']
+                title = append_entries[0].get('title')
+                steam_hash = append_entries[0].get('steam_hash')


(if possible,) I would try to make use an object like {source_id, link, title, steam_hash}, although you would have to deal with the version vs source_id issue

I collapsed that whole block down by determining the source dict first: source_data = main_entry or existing_config or (append_entries[0] if append_entries else {}). It should handle the version vs source_id key discrepancy.

…gex flags, and remove dead code

LVL1024 · 2026-07-13T18:00:23Z

I also wrapped the Steam API parsing in error handling to prevent malformed news items from crashing the changelog run. Also removed some leftover forum and hero lab code.

LVL1024 · 2026-07-13T18:02:29Z

Other than fixing any other issues you might spot, we'll have to change the output of the wiki patch uploads to match closer to the formatting used there, as it has been changed.

hassayag · 2026-07-13T21:04:59Z

Looks great to me. Make any further changes and set it to ready for review, and I'll give it a final look

…anges exist

… formatter

hassayag

Seems good to me, but part of that is pure faith 😂

This reverts commit 9f16142.

LVL1024 added 2 commits July 3, 2026 17:00

feat: migrate changelog fetching from forum RSS to Steam API

47f482e

ci: re-enable changelogs in deploy workflow

0db149f

Deadbot0 mentioned this pull request Jul 3, 2026

feature/steam-patch-notes deadlock-wiki/deadlock-data#244

Closed

LVL1024 and others added 4 commits July 3, 2026 17:14

chore: update poetry.lock after removing feedparser and bs4

15e4e51

fix: add tzdata dependency for ZoneInfo in Docker

54e3a09

[skip ci] chore: updated input data

3d12d17

chore: remove Hero Lab changelog parsing (no longer used by devs)

47c1840

LVL1024 marked this pull request as draft July 3, 2026 18:58

LVL1024 linked an issue Jul 3, 2026 that may be closed by this pull request

Fix changelogs #355

Closed

LVL1024 added 8 commits July 3, 2026 19:30

feat: restructure changelog formatting with section grouping, ability…

b7dd29c

… subheaders, and hotfix preservation

TEST: add dry-run input to workflow_dispatch for wiki testing

9f16142

Revert "feat: restructure changelog formatting with section grouping,…

114866e

… ability subheaders, and hotfix preservation" This reverts commit b7dd29c.

chore: regenerate poetry.lock with Poetry 2.1.4 format

26fe3d9

feat: enhance changelog processing with pending ID support and wiki u…

7f2e144

…pdates fetching

TEST: Delete existing notes

9b8de0e

feat: exclude Hero Lab entries from reprocessing in changelog fetcher

b1994dc

LVL1024 force-pushed the feature/steam-patch-notes branch from 6363402 to e20047a Compare July 4, 2026 18:00

Merge branch 'develop' into feature/steam-patch-notes

ef0c122

LVL1024 requested a review from hassayag July 9, 2026 13:46

hassayag reviewed Jul 12, 2026

View reviewed changes

Deadbot0 and others added 7 commits July 13, 2026 14:45

[skip ci] chore: updated input data

6294611

refactor: remove was_edited mechanic, re-fetch content on Steam edits

0bf9a7f

refactor: simplify source metadata assignment in changelog fetcher

0818fdd

refactor: skip pre-Steam-migration forum changelogs in parser

6618db3

refactor: simplify source title generation in changelog parser

b08d7c3

fix: prevent false hotfixes when overwriting edited posts

3ddf9f2

fix: add error handling, deterministic fallback sorting, re.DOTALL re…

ea5ce13

…gex flags, and remove dead code

LVL1024 requested a review from hassayag July 13, 2026 18:02

LVL1024 added 4 commits July 14, 2026 22:19

refactor: rewrite changelog formatter to group entries by entity type

bf9ea4d

fix: change patch headers from === to == to match new formatter output

21a73ed

chore: fix trailing blank line and add new link targets

adfa2dc

feat: add hide_general parameter for wiki template when no general ch…

2a12b8b

…anges exist

LVL1024 marked this pull request as ready for review July 14, 2026 22:52

refactor: preserve prose lines and drop bracket headings in changelog…

e2a910a

… formatter

hassayag previously approved these changes Jul 15, 2026

View reviewed changes

Revert "TEST: add dry-run input to workflow_dispatch for wiki testing"

4cd2052

This reverts commit 9f16142.

LVL1024 dismissed hassayag’s stale review via 4cd2052 July 15, 2026 17:22

Merge branch 'develop' into feature/steam-patch-notes

ff2c422

LVL1024 merged commit b64a3a1 into develop Jul 15, 2026
2 checks passed

LVL1024 deleted the feature/steam-patch-notes branch July 15, 2026 17:36

Uh oh!

Conversation

LVL1024 commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hassayag left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LVL1024 commented Jul 13, 2026

Uh oh!

LVL1024 commented Jul 13, 2026

Uh oh!

hassayag commented Jul 13, 2026

Uh oh!

hassayag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LVL1024 commented Jul 3, 2026 •

edited

Loading