Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .github/workflows/ruby.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,16 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"

- name: Install dependencies
run: |
sudo apt-get install -yqq yamllint
run: sudo apt-get install -yqq yamllint

- name: Install Node dependencies
run: npm ci --ignore-scripts --no-fund --no-audit

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand All @@ -29,6 +31,9 @@ jobs:
- name: Run linters
run: make lint

- name: Validate configs
run: make validate

test:
strategy:
fail-fast: false
Expand All @@ -38,7 +43,7 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand All @@ -53,7 +58,7 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Set up Ruby
uses: ruby/setup-ruby@v1
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
/spec/reports/
/tmp/
/node_modules/
/schema/

# rspec failure tracking
.rspec_status
2 changes: 1 addition & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
require:
plugins:
- rubocop-rspec
- rubocop-performance

Expand Down
5 changes: 5 additions & 0 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"recommendations": [
"redhat.vscode-yaml"
]
}
5 changes: 5 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"yaml.schemas": {
"https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json": "lib/html2rss/configs/**/*.yml"
}
}
13 changes: 10 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
default: lint test
default: lint validate test

lint:
yamllint lib/html2rss/configs/ .github/
bundle exec rubocop -P -f quiet
npx prettier --check lib/**/*.yml .github/**/*.yml README.md
./node_modules/.bin/prettier --check lib/**/*.yml .github/**/*.yml README.md

validate:
bundle exec ruby bin/validate_configs
Comment on lines +8 to +9
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says "drop obsolete migrate-tests/restore-tests targets whose backing script no longer exists," but these targets (lines 42-52) are still present in the Makefile. The bin/migrate_to_dynamic_tests script referenced by migrate-tests doesn't exist, so running make migrate-tests will fail. These targets should be removed as stated in the PR description.

Copilot uses AI. Check for mistakes.

schema:
mkdir -p schema
bundle exec html2rss schema --write schema/html2rss-config.schema.json

test:
bundle exec rspec
Expand Down Expand Up @@ -46,4 +53,4 @@ restore-tests:

lintfix:
bundle exec rubocop -a
npx prettier --write lib/**/*.yml .github/**/*.yml README.md
./node_modules/.bin/prettier --write lib/**/*.yml .github/**/*.yml README.md
52 changes: 48 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This repository contains `html2rss` feed configurations for many websites.

## Dynamic Parameters

Configs must include a `parameters` section to define default values for dynamic parameters:
Parameterized configs should include a `parameters` section with default values:

```yaml
parameters:
Expand All @@ -38,7 +38,30 @@ channel:
# ... rest of config
```

The `type` field specifies the parameter type (currently only `string` is supported), and `default` provides the default value when no parameter is explicitly provided.
The `type` field specifies the parameter type (currently only `string` is supported), and `default` provides the default value used by this repository's validation and fetch tests.

Notes:

- Only configs that use `%<param>s` placeholders need a `parameters` section.
- Callers can still override those defaults at runtime with `html2rss feed ... --params ...`.
- Dynamic substitution applies to `channel` and `headers`; selectors are not parameterized by this feature.

## Validation

Use both schema-aware editing and runtime validation before committing:

```bash
# Validate all configs with the runtime validator
make validate

# Validate a single config directly
bundle exec html2rss validate lib/html2rss/configs/github.com/releases.yml

# Export the current JSON Schema locally for editor use
make schema
```

The JSON Schema is useful for editor autocompletion and basic structural checks. Runtime validation remains authoritative for merged defaults and cross-field rules.

## Testing

Expand All @@ -55,12 +78,33 @@ make test-config CONFIG=github.com/releases.yml
make test-domain DOMAIN=github.com
```

**Adding new configs**: Just create the YAML file and run tests. No spec file needed.
**Adding new configs**: Create the YAML file, run `make validate`, then run the generated tests. No dedicated spec file is needed.

**Config folder convention**: Place configs under the registrable domain folder (e.g., `example.com/` or `bbc.co.uk/`). Legacy subdomain folders (e.g., `news.example.com/`) are allowed but not preferred.

## Editor Setup (JSON Schema)

Get inline validation and autocompletion when editing configs in your IDE.
All config files already carry the schema modeline at the top:

```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
```

Any editor with [yaml-language-server](https://github.com/redhat-developer/yaml-language-server)
support (VS Code + [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml),
Neovim, Helix, …) will automatically pick up the schema when opening a config file.

The included `.vscode/settings.json` additionally associates the schema with all
configs via a glob pattern, so new files get validation before the modeline is added.

> `make schema` writes `schema/html2rss-config.schema.json` locally — useful for
> offline editing or when you want to pin against the currently installed gem version.

## Documentation

- [Main Documentation](https://html2rss.github.io/html2rss-configs/)
- [Selectors Reference](https://html2rss.github.io/ruby-gem/reference/selectors/)
- [Dynamic Parameters](https://html2rss.github.io/ruby-gem/how-to/dynamic-parameters/)
- [CLI Reference](https://html2rss.github.io/ruby-gem/reference/cli-reference/)
- [Contributing Guide](https://html2rss.github.io/get-involved/contributing)
- [Sponsorship Page](https://html2rss.github.io/get-involved/sponsoring)
42 changes: 29 additions & 13 deletions bin/rspec_changed_configs
Original file line number Diff line number Diff line change
@@ -1,18 +1,34 @@
#!/usr/bin/env ruby
# frozen_string_literal: true

changed_files = `git diff --name-only origin/master | grep 'lib/html2rss/configs/.*/.*.yml'`.split("\n")

if changed_files.any?
# Use dynamic test file with environment variable to filter changed configs
config_names = changed_files.map { |file| file.sub('lib/html2rss/configs/', '') }

# Test each changed config individually
config_names.each do |config_name|
puts "Testing changed config: #{config_name}"
system("bundle exec rspec --example '#{config_name}' --tag fetch spec/html2rss/configs_dynamic_spec.rb")
end
else
puts 'No changed config files found'
CONFIG_PATH = %r{\Alib/html2rss/configs/.+/.+\.yml\z}
SCHEMA_MODELINE = /\A[+-]# yaml-language-server: \$schema=/

def comment_only_change?(file)
diff_lines = `git diff --unified=0 origin/master -- #{file}`.lines
changed_lines = diff_lines.select { |line| line.start_with?('+', '-') }
content_lines = changed_lines.reject { |line| line.start_with?('+++', '---') }

content_lines.any? && content_lines.all? { |line| line.match?(SCHEMA_MODELINE) }
end

changed_files = `git diff --name-only origin/master`
.split("\n")
.grep(CONFIG_PATH)

test_files = changed_files.reject { |file| comment_only_change?(file) }

if test_files.empty?
puts 'No changed config files require fetch tests'
exit 0
end

failed = false

test_files.each do |file|
config_name = file.sub('lib/html2rss/configs/', '')
puts "Testing changed config: #{config_name}"
failed ||= !system("bundle exec rspec --example '#{config_name}' --tag fetch spec/html2rss/configs_dynamic_spec.rb")
end

exit 1 if failed
29 changes: 27 additions & 2 deletions bin/setup
Original file line number Diff line number Diff line change
@@ -1,8 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
set -vx

echo "==> Installing Ruby dependencies..."
bundle install

# Do any other automated setup that you need to do here
echo
echo "==> Installing Node dependencies (required for make lint)..."
npm install --ignore-scripts --no-fund --no-audit

echo
echo "==> Checking system tools..."
if command -v yamllint &>/dev/null; then
echo " yamllint: ok ($(yamllint --version))"
else
echo " yamllint: not found"
echo " Install with one of:"
echo " brew install yamllint"
echo " pip install yamllint"
fi

echo
echo "==> Setup complete. Common commands:"
echo
echo " make # lint + validate + test"
echo " make lint # yamllint, rubocop, prettier"
echo " make validate # validate all configs against the html2rss schema"
echo " make test # run rspec"
echo
echo " make test-config CONFIG=github.com/releases.yml # test a single config"
echo " make test-domain DOMAIN=github.com # test all configs for a domain"
echo " make schema # write schema locally (optional)"
40 changes: 40 additions & 0 deletions bin/validate_configs
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env ruby
# frozen_string_literal: true

require 'bundler/setup'
require 'html2rss'
require 'yaml'

files = Dir['lib/html2rss/configs/**/*.yml']
failed = []

files.each do |file|
config = YAML.safe_load_file(file, symbolize_names: true)
result = Html2rss::Config.validate(config)

if result.success?
puts "ok #{file}"
else
puts "FAIL #{file}"
result.errors.to_h.each do |key, messages|
Array(messages).each { |msg| warn " #{key}: #{msg}" }
end
failed << file
end
rescue Psych::Exception => error
puts "FAIL #{file}"
warn " parse: #{error.message}"
failed << file
rescue StandardError => error
puts "FAIL #{file}"
warn " validation: #{error.class}: #{error.message}"
failed << file
end

puts
if failed.empty?
puts "#{files.size} configs validated successfully."
else
warn "#{failed.size}/#{files.size} configs failed validation."
exit 1
end
1 change: 1 addition & 0 deletions lib/html2rss/configs/adfc.de/pressemitteilungen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.adfc.de/presse/pressemitteilungen/
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/apnews.com/hub.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
parameters:
section:
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/avherald.com/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://avherald.com/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/bbc.co.uk/available_episodes.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
id:
type: string
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/bbc.com/mundo.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.bbc.com/mundo
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/blog.mondediplo.net/feed.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
parameters:
blog:
type: string
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/canarianweekly.com/front.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.canarianweekly.com/
time_zone: Europe/London
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cinemascore.com/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://webapp.cinemascore.com/guest/surveys
ttl: 720
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cleanenergywire.org/news.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.cleanenergywire.org/news-archive
time_zone: "Europe/Berlin"
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cnet.com/section_sub.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
parameters:
section:
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/computerbase.de/meistgelesen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
title: "computerbase.de: meistgelesen"
url: https://www.computerbase.de
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/cutle.fish/index.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://cutle.fish/
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/deraktionaer.de/meistgelesen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
title: "deraktionaer.de: meistgelesen"
url: https://deraktionaer.de/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
json: true
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/dfs.de/pressemitteilungen.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.dfs.de/homepage/de/medien/presse/
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/dsw-info.de/presse.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.dsw-info.de/presse
time_zone: Europe/Berlin
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/espn.com/f1.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.espn.com/f1/
time_zone: UTC
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/fia.com/documents.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
---
channel:
url: https://www.fia.com/documents/championships/fia-formula-one-world-championship-14/season/season-2026-2072
Expand Down
1 change: 1 addition & 0 deletions lib/html2rss/configs/formula1.com/latest.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json
channel:
url: https://www.formula1.com/en/latest/all.html
time_zone: UTC
Expand Down
Loading
Loading