steve/go-extractor
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add RemoveHidden option for display:none element stripping #63

Merged
steve merged 1 commits from feature/readability-remove-hidden into main 2026-02-20 14:10:59 +00:00
Conversation 0 Commits 1 Files Changed 3 +164
Claude commented 2026-02-20 14:06:32 +00:00
Collaborator
Copy Link

Summary

  • Adds RemoveHidden bool field to ReadabilityOptions that evaluates JavaScript on the live page to remove all elements with computed display: none before readability extraction
  • Defines an internal pageEvaluator interface so the concrete Playwright-backed document supports JS evaluation without changing the Document interface
  • Adds PageEvaluate method to the document struct
  • Includes tests for the happy path, error propagation from JS evaluation, and the unsupported-Document case

Closes #62

Context

Some websites embed hidden display: none elements as anti-AI-scraping honeypots containing prompt injection attacks. These are invisible to users but get picked up by readability extraction. CSS selectors can't target computed styles, so JavaScript evaluation with getComputedStyle is needed.

Test plan

  • go test ./... passes
  • Manual test against a page with known hidden honeypot elements (e.g. Together AI blog post mentioned in #62)
## Summary - Adds `RemoveHidden bool` field to `ReadabilityOptions` that evaluates JavaScript on the live page to remove all elements with computed `display: none` before readability extraction - Defines an internal `pageEvaluator` interface so the concrete Playwright-backed document supports JS evaluation without changing the `Document` interface - Adds `PageEvaluate` method to the `document` struct - Includes tests for the happy path, error propagation from JS evaluation, and the unsupported-Document case Closes #62 ## Context Some websites embed hidden `display: none` elements as anti-AI-scraping honeypots containing prompt injection attacks. These are invisible to users but get picked up by readability extraction. CSS selectors can't target computed styles, so JavaScript evaluation with `getComputedStyle` is needed. ## Test plan - [x] `go test ./...` passes - [ ] Manual test against a page with known hidden honeypot elements (e.g. Together AI blog post mentioned in #62)
Claude added 1 commit 2026-02-20 14:06:32 +00:00
feat: add RemoveHidden option to strip display:none elements before extraction
All checks were successful
CI / vet (pull_request) Successful in 34s
Details
CI / test (pull_request) Successful in 1m1s
Details
CI / build (pull_request) Successful in 1m5s
Details
65cf6b027f 
When RemoveHidden is true, JavaScript is evaluated on the live page to
remove all elements with computed display:none before readability
extraction. This defends against anti-scraping honeypots that embed
prompt injections in hidden DOM elements.

The implementation uses an optional pageEvaluator interface so that the
concrete document (backed by Playwright) supports it while the Document
interface remains unchanged.

Closes #62

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
steve merged commit 8a2de65e31 into main 2026-02-20 14:10:59 +00:00
steve deleted branch feature/readability-remove-hidden 2026-02-20 14:11:00 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
Something isn't working correctly
 claude
Trigger — Steve has marked this issue for Claude to work on.
 claude:blocked
Blocked — Waiting on a dependency (another issue/repo).
 claude:review
In Review — PR created, awaiting human review/merge.
 claude:running
Active — A Claude instance is currently working on this.
 documentation
Documentation improvements or additions
 enhancement
New feature or improvement to existing functionality
 performance
Performance optimization or resource usage improvement
 priority/critical
Showstopper — security vulnerability, data loss, or crash in production
 priority/high
Important — significant bug or high-value improvement
 priority/low
Nice to have — minor improvement or cleanup
 priority/medium
Normal — standard improvement or non-critical bug
 security
Security-related issue or vulnerability
 testing
Test coverage, test infrastructure, or test improvements
 type/epic
Parent issue grouping related stories/tasks
 type/refactor
Code restructuring without behavior change
 type/task
Concrete implementation work item
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Claude fizi steve
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/go-extractor#63
Delete Branch "feature/readability-remove-hidden"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?