Adds ReadabilityOptions struct with RemoveSelectors []string field for specifying CSS selectors of elements to remove before readability extraction
Adds ReadabilityWithOptions() function that applies DOM cleanup before parsing
Existing Readability() delegates to ReadabilityWithOptions with zero-value options (fully backward compatible)
Uses goquery (new dependency) for CSS selector-based DOM manipulation
Motivation
Sites like The Verge use infinite scroll that loads additional full articles below the current one. When Readability() extracts content, these extra articles pollute the result. This change lets callers specify selectors to remove before extraction, eliminating the need to reimplement the readability pipeline downstream.
TestReadabilityWithOptions_RemoveSelectors — verifies removed elements are excluded from extraction
TestReadabilityWithOptions_NoSelectors — verifies empty options behave like Readability()
TestRemoveSelectors — unit test for the HTML cleaning function
TestRemoveSelectors_MultipleSelectors — verifies multiple selectors work together
All existing tests continue to pass
## Summary
- Adds `ReadabilityOptions` struct with `RemoveSelectors []string` field for specifying CSS selectors of elements to remove before readability extraction
- Adds `ReadabilityWithOptions()` function that applies DOM cleanup before parsing
- Existing `Readability()` delegates to `ReadabilityWithOptions` with zero-value options (fully backward compatible)
- Uses goquery (new dependency) for CSS selector-based DOM manipulation
## Motivation
Sites like The Verge use infinite scroll that loads additional full articles below the current one. When `Readability()` extracts content, these extra articles pollute the result. This change lets callers specify selectors to remove before extraction, eliminating the need to reimplement the readability pipeline downstream.
Closes #60
## Test plan
- [x] `TestReadabilityWithOptions_RemoveSelectors` — verifies removed elements are excluded from extraction
- [x] `TestReadabilityWithOptions_NoSelectors` — verifies empty options behave like `Readability()`
- [x] `TestRemoveSelectors` — unit test for the HTML cleaning function
- [x] `TestRemoveSelectors_MultipleSelectors` — verifies multiple selectors work together
- [x] All existing tests continue to pass
Sites with infinite scroll (e.g. The Verge) load additional articles
into the DOM, which get included in readability extraction. Add
ReadabilityOptions.RemoveSelectors to strip elements by CSS selector
before parsing, avoiding the need to reimplement the readability
pipeline downstream.
Closes#60
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Claude
merged commit c982b61bab into main2026-02-19 01:11:38 +00:00
Claude
deleted branch feature/readability-remove-selectors2026-02-19 01:11:38 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
ReadabilityOptionsstruct withRemoveSelectors []stringfield for specifying CSS selectors of elements to remove before readability extractionReadabilityWithOptions()function that applies DOM cleanup before parsingReadability()delegates toReadabilityWithOptionswith zero-value options (fully backward compatible)Motivation
Sites like The Verge use infinite scroll that loads additional full articles below the current one. When
Readability()extracts content, these extra articles pollute the result. This change lets callers specify selectors to remove before extraction, eliminating the need to reimplement the readability pipeline downstream.Closes #60
Test plan
TestReadabilityWithOptions_RemoveSelectors— verifies removed elements are excluded from extractionTestReadabilityWithOptions_NoSelectors— verifies empty options behave likeReadability()TestRemoveSelectors— unit test for the HTML cleaning functionTestRemoveSelectors_MultipleSelectors— verifies multiple selectors work together