Add DOM cleanup option before readability extraction #60

New Issue

Claude · 2026-02-19T01:05:08Z

Claude commented

2026-02-19 01:05:08 +00:00

Problem

Sites like The Verge use infinite scroll that loads additional full articles below the current article in the DOM. When Readability() extracts content, these extra articles get included in the extracted text, producing summaries that mix in content from unrelated articles.

The root cause is that Readability() calls doc.Content() → readability.FromReader() with no opportunity to clean the DOM between those steps.

Current Workaround

In steve/mort#709, we're working around this by replicating what Readability() does but inserting a goquery-based HTML cleaning step:

doc.Content() → goquery parse → remove problematic elements → readability.FromReader()

This works but duplicates the Readability() logic and would benefit from native support in go-extractor.

Proposed Solution

Add a way to specify CSS selectors for elements to remove before readability extraction. Some options:

Option A: ReadabilityOptions struct

type ReadabilityOptions struct {
    RemoveSelectors []string // CSS selectors for elements to remove before extraction
}

func ReadabilityWithOptions(_ context.Context, doc Document, opts ReadabilityOptions) (Article, error)

Option B: Pre-extraction hook

type ReadabilityHook func(html string) (string, error)

func ReadabilityWithHook(_ context.Context, doc Document, hook ReadabilityHook) (Article, error)

Option C: Add to OpenPageOptions

type OpenPageOptions struct {
    // ... existing fields ...
    RemoveSelectors []string // Elements to remove after page load
}

Any of these would allow consumers to clean problematic DOM elements without reimplementing the readability pipeline.

## Problem Sites like The Verge use infinite scroll that loads additional full articles below the current article in the DOM. When `Readability()` extracts content, these extra articles get included in the extracted text, producing summaries that mix in content from unrelated articles. The root cause is that `Readability()` calls `doc.Content()` → `readability.FromReader()` with no opportunity to clean the DOM between those steps. ## Current Workaround In [steve/mort#709](https://gitea.stevedudenhoeffer.com/steve/mort/issues/709), we're working around this by replicating what `Readability()` does but inserting a goquery-based HTML cleaning step: ``` doc.Content() → goquery parse → remove problematic elements → readability.FromReader() ``` This works but duplicates the `Readability()` logic and would benefit from native support in go-extractor. ## Proposed Solution Add a way to specify CSS selectors for elements to remove before readability extraction. Some options: ### Option A: ReadabilityOptions struct ```go type ReadabilityOptions struct { RemoveSelectors []string // CSS selectors for elements to remove before extraction } func ReadabilityWithOptions(_ context.Context, doc Document, opts ReadabilityOptions) (Article, error) ``` ### Option B: Pre-extraction hook ```go type ReadabilityHook func(html string) (string, error) func ReadabilityWithHook(_ context.Context, doc Document, hook ReadabilityHook) (Article, error) ``` ### Option C: Add to OpenPageOptions ```go type OpenPageOptions struct { // ... existing fields ... RemoveSelectors []string // Elements to remove after page load } ``` Any of these would allow consumers to clean problematic DOM elements without reimplementing the readability pipeline.

Claude referenced this issue from a commit

2026-02-19 01:05:37 +00:00

fix: sanitize HTML before readability to prevent infinite scroll pollution

Claude commented

2026-02-19 01:07:13 +00:00

Starting work on this. My plan:

Go with Option A (ReadabilityOptions with RemoveSelectors) — it's the most focused and keeps the API clean.
Add a ReadabilityWithOptions() function that takes ReadabilityOptions with a RemoveSelectors []string field.
The implementation will get the page HTML via doc.Content(), use goquery to remove matching elements, then pass the cleaned HTML to readability.FromReader().
Keep the existing Readability() function unchanged for backward compatibility — it will delegate to ReadabilityWithOptions with zero-value options.

Will push a PR shortly.

Starting work on this. My plan: 1. Go with **Option A** (`ReadabilityOptions` with `RemoveSelectors`) — it's the most focused and keeps the API clean. 2. Add a `ReadabilityWithOptions()` function that takes `ReadabilityOptions` with a `RemoveSelectors []string` field. 3. The implementation will get the page HTML via `doc.Content()`, use goquery to remove matching elements, then pass the cleaned HTML to `readability.FromReader()`. 4. Keep the existing `Readability()` function unchanged for backward compatibility — it will delegate to `ReadabilityWithOptions` with zero-value options. Will push a PR shortly.

Claude referenced this issue from a commit

2026-02-19 01:09:35 +00:00

feat: add ReadabilityWithOptions for DOM cleanup before extraction

Claude referenced a pull request that will close this issue

2026-02-19 01:09:41 +00:00

feat: add ReadabilityWithOptions for DOM cleanup #61

Claude closed this issue

2026-02-19 01:11:38 +00:00

Sign in to join this conversation.