Sites like The Verge use infinite scroll that loads additional full articles below the current article in the DOM. When Readability() extracts content, these extra articles get included in the extracted text, producing summaries that mix in content from unrelated articles.
The root cause is that Readability() calls doc.Content() → readability.FromReader() with no opportunity to clean the DOM between those steps.
Current Workaround
In steve/mort#709, we're working around this by replicating what Readability() does but inserting a goquery-based HTML cleaning step:
doc.Content() → goquery parse → remove problematic elements → readability.FromReader()
This works but duplicates the Readability() logic and would benefit from native support in go-extractor.
Proposed Solution
Add a way to specify CSS selectors for elements to remove before readability extraction. Some options:
Option A: ReadabilityOptions struct
typeReadabilityOptionsstruct{RemoveSelectors[]string// CSS selectors for elements to remove before extraction}funcReadabilityWithOptions(_context.Context,docDocument,optsReadabilityOptions)(Article,error)
typeOpenPageOptionsstruct{// ... existing fields ...RemoveSelectors[]string// Elements to remove after page load}
Any of these would allow consumers to clean problematic DOM elements without reimplementing the readability pipeline.
## Problem
Sites like The Verge use infinite scroll that loads additional full articles below the current article in the DOM. When `Readability()` extracts content, these extra articles get included in the extracted text, producing summaries that mix in content from unrelated articles.
The root cause is that `Readability()` calls `doc.Content()` → `readability.FromReader()` with no opportunity to clean the DOM between those steps.
## Current Workaround
In [steve/mort#709](https://gitea.stevedudenhoeffer.com/steve/mort/issues/709), we're working around this by replicating what `Readability()` does but inserting a goquery-based HTML cleaning step:
```
doc.Content() → goquery parse → remove problematic elements → readability.FromReader()
```
This works but duplicates the `Readability()` logic and would benefit from native support in go-extractor.
## Proposed Solution
Add a way to specify CSS selectors for elements to remove before readability extraction. Some options:
### Option A: ReadabilityOptions struct
```go
type ReadabilityOptions struct {
RemoveSelectors []string // CSS selectors for elements to remove before extraction
}
func ReadabilityWithOptions(_ context.Context, doc Document, opts ReadabilityOptions) (Article, error)
```
### Option B: Pre-extraction hook
```go
type ReadabilityHook func(html string) (string, error)
func ReadabilityWithHook(_ context.Context, doc Document, hook ReadabilityHook) (Article, error)
```
### Option C: Add to OpenPageOptions
```go
type OpenPageOptions struct {
// ... existing fields ...
RemoveSelectors []string // Elements to remove after page load
}
```
Any of these would allow consumers to clean problematic DOM elements without reimplementing the readability pipeline.
Go with Option A (ReadabilityOptions with RemoveSelectors) — it's the most focused and keeps the API clean.
Add a ReadabilityWithOptions() function that takes ReadabilityOptions with a RemoveSelectors []string field.
The implementation will get the page HTML via doc.Content(), use goquery to remove matching elements, then pass the cleaned HTML to readability.FromReader().
Keep the existing Readability() function unchanged for backward compatibility — it will delegate to ReadabilityWithOptions with zero-value options.
Will push a PR shortly.
Starting work on this. My plan:
1. Go with **Option A** (`ReadabilityOptions` with `RemoveSelectors`) — it's the most focused and keeps the API clean.
2. Add a `ReadabilityWithOptions()` function that takes `ReadabilityOptions` with a `RemoveSelectors []string` field.
3. The implementation will get the page HTML via `doc.Content()`, use goquery to remove matching elements, then pass the cleaned HTML to `readability.FromReader()`.
4. Keep the existing `Readability()` function unchanged for backward compatibility — it will delegate to `ReadabilityWithOptions` with zero-value options.
Will push a PR shortly.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
Sites like The Verge use infinite scroll that loads additional full articles below the current article in the DOM. When
Readability()extracts content, these extra articles get included in the extracted text, producing summaries that mix in content from unrelated articles.The root cause is that
Readability()callsdoc.Content()→readability.FromReader()with no opportunity to clean the DOM between those steps.Current Workaround
In steve/mort#709, we're working around this by replicating what
Readability()does but inserting a goquery-based HTML cleaning step:This works but duplicates the
Readability()logic and would benefit from native support in go-extractor.Proposed Solution
Add a way to specify CSS selectors for elements to remove before readability extraction. Some options:
Option A: ReadabilityOptions struct
Option B: Pre-extraction hook
Option C: Add to OpenPageOptions
Any of these would allow consumers to clean problematic DOM elements without reimplementing the readability pipeline.
Starting work on this. My plan:
ReadabilityOptionswithRemoveSelectors) — it's the most focused and keeps the API clean.ReadabilityWithOptions()function that takesReadabilityOptionswith aRemoveSelectors []stringfield.doc.Content(), use goquery to remove matching elements, then pass the cleaned HTML toreadability.FromReader().Readability()function unchanged for backward compatibility — it will delegate toReadabilityWithOptionswith zero-value options.Will push a PR shortly.