Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .github/workflows/ruby.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,16 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"

- name: Install dependencies
run: |
sudo apt-get install -yqq yamllint
run: sudo apt-get install -yqq yamllint

- name: Install Node dependencies
run: npm ci --ignore-scripts --no-fund --no-audit

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand All @@ -29,6 +31,9 @@ jobs:
- name: Run linters
run: make lint

- name: Validate configs
run: make validate

test:
strategy:
fail-fast: false
Expand All @@ -38,7 +43,7 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand All @@ -53,7 +58,7 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
/spec/reports/
/tmp/
/node_modules/
/schema/

# rspec failure tracking
.rspec_status
5 changes: 5 additions & 0 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"recommendations": [
"redhat.vscode-yaml"
]
}
5 changes: 5 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"yaml.schemas": {
"https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json": "lib/html2rss/configs/**/*.yml"
}
}
13 changes: 10 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
default: lint test
default: lint validate test

lint:
yamllint lib/html2rss/configs/ .github/
bundle exec rubocop -P -f quiet
npx prettier --check lib/**/*.yml .github/**/*.yml README.md
./node_modules/.bin/prettier --check lib/**/*.yml .github/**/*.yml README.md

validate:
bundle exec ruby bin/validate_configs
Comment on lines +8 to +9
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says "drop obsolete migrate-tests/restore-tests targets whose backing script no longer exists," but these targets (lines 42-52) are still present in the Makefile. The bin/migrate_to_dynamic_tests script referenced by migrate-tests doesn't exist, so running make migrate-tests will fail. These targets should be removed as stated in the PR description.

Copilot uses AI. Check for mistakes.

schema:
mkdir -p schema
bundle exec html2rss schema --write schema/html2rss-config.schema.json

test:
bundle exec rspec
Expand Down Expand Up @@ -46,4 +53,4 @@ restore-tests:

lintfix:
bundle exec rubocop -a
npx prettier --write lib/**/*.yml .github/**/*.yml README.md
./node_modules/.bin/prettier --write lib/**/*.yml .github/**/*.yml README.md
33 changes: 32 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,18 @@ channel:

The `type` field specifies the parameter type (currently only `string` is supported), and `default` provides the default value when no parameter is explicitly provided.

## Validation

Validate configs against the html2rss schema before committing:

```bash
# Validate all configs
make validate

# Validate a single config directly
bundle exec html2rss validate lib/html2rss/configs/github.com/releases.yml
```

## Testing

Uses **dynamic test generation** - no individual spec files needed!
Expand All @@ -55,10 +67,29 @@ make test-config CONFIG=github.com/releases.yml
make test-domain DOMAIN=github.com
```

**Adding new configs**: Just create the YAML file and run tests. No spec file needed.
**Adding new configs**: Just create the YAML file and run `make validate` then tests. No spec file needed.

**Config folder convention**: Place configs under the registrable domain folder (e.g., `example.com/` or `bbc.co.uk/`). Legacy subdomain folders (e.g., `news.example.com/`) are allowed but not preferred.

## Editor Setup (JSON Schema)

Get inline validation and autocompletion when editing configs in your IDE.
All config files already carry the schema modeline at the top:

```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
```

Any editor with [yaml-language-server](https://github.com/redhat-developer/yaml-language-server)
support (VS Code + [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml),
Neovim, Helix, …) will automatically pick up the schema when opening a config file.

The included `.vscode/settings.json` additionally associates the schema with all
configs via a glob pattern, so new files get validation before the modeline is added.

> `make schema` writes `schema/html2rss-config.schema.json` locally — useful for
> offline editing or when you want to pin against the currently installed gem version.

## Documentation

- [Main Documentation](https://html2rss.github.io/html2rss-configs/)
Expand Down
29 changes: 27 additions & 2 deletions bin/setup
Original file line number Diff line number Diff line change
@@ -1,8 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
set -vx

echo "==> Installing Ruby dependencies..."
bundle install

# Do any other automated setup that you need to do here
echo
echo "==> Installing Node dependencies (required for make lint)..."
npm install --ignore-scripts --no-fund --no-audit

echo
echo "==> Checking system tools..."
if command -v yamllint &>/dev/null; then
echo " yamllint: ok ($(yamllint --version))"
else
echo " yamllint: not found"
echo " Install with one of:"
echo " brew install yamllint"
echo " pip install yamllint"
fi

echo
echo "==> Setup complete. Common commands:"
echo
echo " make # lint + validate + test"
echo " make lint # yamllint, rubocop, prettier"
echo " make validate # validate all configs against the html2rss schema"
echo " make test # run rspec"
echo
echo " make test-config CONFIG=github.com/releases.yml # test a single config"
echo " make test-domain DOMAIN=github.com # test all configs for a domain"
echo " make schema # write schema locally (optional)"
42 changes: 42 additions & 0 deletions bin/validate_configs
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/usr/bin/env ruby
# frozen_string_literal: true

require 'bundler/setup'
require 'html2rss'
require 'yaml'

files = Dir['lib/html2rss/configs/**/*.yml']
failed = []

files.each do |file|
begin
config = YAML.safe_load_file(file, symbolize_names: true)
result = Html2rss::Config.validate(config)

if result.success?
puts "ok #{file}"
else
puts "FAIL #{file}"
result.errors.to_h.each do |key, messages|
Array(messages).each { |msg| warn " #{key}: #{msg}" }
end
failed << file
end
rescue Psych::Exception => e
puts "FAIL #{file}"
warn " parse: #{e.message}"
failed << file
rescue StandardError => e
puts "FAIL #{file}"
warn " validation: #{e.class}: #{e.message}"
failed << file
end
end

puts
if failed.empty?
puts "#{files.size} configs validated successfully."
else
warn "#{failed.size}/#{files.size} configs failed validation."
exit 1
end
1 change: 1 addition & 0 deletions lib/html2rss/configs/adfc.de/pressemitteilungen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.adfc.de/presse/pressemitteilungen/
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/apnews.com/hub.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
parameters:
section:
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/avherald.com/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://avherald.com/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/bbc.co.uk/available_episodes.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
id:
type: string
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/bbc.com/mundo.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.bbc.com/mundo
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/blog.mondediplo.net/feed.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
blog:
type: string
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/canarianweekly.com/front.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.canarianweekly.com/
time_zone: Europe/London
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cinemascore.com/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://webapp.cinemascore.com/guest/surveys
ttl: 720
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cleanenergywire.org/news.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.cleanenergywire.org/news-archive
time_zone: "Europe/Berlin"
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cnet.com/section_sub.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
parameters:
section:
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/computerbase.de/meistgelesen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
title: "computerbase.de: meistgelesen"
url: https://www.computerbase.de
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cutle.fish/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://cutle.fish/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/deraktionaer.de/meistgelesen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
title: "deraktionaer.de: meistgelesen"
url: https://deraktionaer.de/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
json: true
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/dfs.de/pressemitteilungen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.dfs.de/homepage/de/medien/presse/
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/dsw-info.de/presse.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.dsw-info.de/presse
time_zone: Europe/Berlin
Expand Down
3 changes: 3 additions & 0 deletions lib/html2rss/configs/espn.com/f1.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
strategy: default
channel:
url: https://www.espn.com/f1/
time_zone: UTC
ttl: 60
selectors:
items:
s: 'dfd'
Comment thread
gildesmarais marked this conversation as resolved.
Outdated
selector: ".headlineStack__list > li"
title:
selector: "a"
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/fia.com/documents.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.fia.com/documents/championships/fia-formula-one-world-championship-14/season/season-2026-2072
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/formula1.com/latest.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.formula1.com/en/latest/all.html
time_zone: UTC
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/github.com/releases.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
username:
type: string
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/iaapa.org/news.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.iaapa.org/news
time_zone: UTC
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/imdb.com/ratings.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
user_id:
type: string
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.ingenieur.de/karriere/arbeitsleben/heiko-mell/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/kinocheck.de/filmstarts.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://kinocheck.de/filmstarts
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/microsoft.com/azure-products.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://azure.microsoft.com/en-us/products
language: en
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/newyorker.com/magazine.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.newyorker.com/magazine
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/nomanssky.com/news.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.nomanssky.com/news/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://pankow.lebensmittel-kontrollergebnisse.de/Search
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/phys.org/weekly.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://phys.org/weekly-news/
time_zone: Europe/London
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/rbb24.de/meistgeklickt.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
title: "rbb24.de: meistgeklickt"
url: https://rbb24.de/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/robinwood.de/aktuelles.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.robinwood.de/was-gibt-es-neues/aktuelles
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/s3.amazonaws.com/popular_movies.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
# This generates a RSS of the daily updated
# https://github.com/sjlu/popular-movies
channel:
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/sebastianvettel.de/news.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://sebastianvettel.de/news/
Expand Down
Loading
Loading