steve/go-extractor
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Epic: Improve headless browser stealth against anti-bot detection #68

New Issue
Closed
opened 2026-02-24 01:13:06 +00:00 by Claude · 2 comments
Claude commented 2026-02-24 01:13:06 +00:00
Collaborator
Copy Link

Summary

The current stealth system (added in #58 / #59) is insufficient against stricter anti-bot services like archive.ph. The NewInteractiveBrowser (Chromium) successfully loads archive.ph, but NewBrowser (Firefox, used by mort's summary system) gets HTTP 429 blocked.

Investigation reveals several architectural gaps:

Aspect NewInteractiveBrowser (WORKS) NewBrowser (FAILS)
Browser engine Chromium (explicit) Firefox (default)
Viewport 1280×720 (explicit) None set
Stealth args --disable-blink-features applied Not applied (Firefox ignores Chromium args)
New headless mode Channel: "chromium" applied Not applied (Firefox)

The 12 stealth init scripts were written exclusively for Chromium but are injected unconditionally into all browser engines including Firefox. Firefox receives scripts that reference window.chrome, Chrome PDF plugins, HeadlessChrome UA stripping, etc. — all are no-ops or actively suspicious on Firefox. Meanwhile, Firefox-specific headless detection vectors are completely unaddressed.

Additionally, every browser session has identical static fingerprint values (WebGL renderer, plugin list, connection stats), making fingerprint-based detection trivial across sessions.

Sub-tasks

  • #69 — Add Firefox-specific stealth and split browser-conditional init scripts
  • #70 — Set default viewport for NewBrowser and align User-Agent with browser engine
  • #71 — Randomize static fingerprint values across browser sessions

Prior Art

  • #58 — Initial stealth scripts (Chromium-focused)
  • #59 — Additional stealth improvements
  • Downstream: steve/mort summary system blocked by archive.ph (HTTP 429)
## Summary The current stealth system (added in #58 / #59) is insufficient against stricter anti-bot services like archive.ph. The `NewInteractiveBrowser` (Chromium) successfully loads archive.ph, but `NewBrowser` (Firefox, used by mort's summary system) gets HTTP 429 blocked. Investigation reveals several architectural gaps: | Aspect | `NewInteractiveBrowser` (WORKS) | `NewBrowser` (FAILS) | |--------|--------------------------------|----------------------| | Browser engine | **Chromium** (explicit) | **Firefox** (default) | | Viewport | 1280×720 (explicit) | **None set** | | Stealth args | `--disable-blink-features` applied | **Not applied** (Firefox ignores Chromium args) | | New headless mode | `Channel: "chromium"` applied | **Not applied** (Firefox) | The 12 stealth init scripts were written exclusively for Chromium but are injected unconditionally into all browser engines including Firefox. Firefox receives scripts that reference `window.chrome`, Chrome PDF plugins, `HeadlessChrome` UA stripping, etc. — all are no-ops or actively suspicious on Firefox. Meanwhile, Firefox-specific headless detection vectors are completely unaddressed. Additionally, every browser session has identical static fingerprint values (WebGL renderer, plugin list, connection stats), making fingerprint-based detection trivial across sessions. ## Sub-tasks - [x] #69 — Add Firefox-specific stealth and split browser-conditional init scripts - [x] #70 — Set default viewport for `NewBrowser` and align User-Agent with browser engine - [x] #71 — Randomize static fingerprint values across browser sessions ## Prior Art - #58 — Initial stealth scripts (Chromium-focused) - #59 — Additional stealth improvements - Downstream: `steve/mort` summary system blocked by archive.ph (HTTP 429)
Claude added the enhancementpriority/hightype/epic labels 2026-02-24 01:13:48 +00:00
Claude commented 2026-02-24 01:18:16 +00:00
Author
Collaborator
Copy Link

Starting work on #69 (split browser-conditional init scripts and add Firefox-specific stealth). This is the first sub-task of this epic.

Starting work on #69 (split browser-conditional init scripts and add Firefox-specific stealth). This is the first sub-task of this epic.
Claude commented 2026-02-24 01:26:34 +00:00
Author
Collaborator
Copy Link

Starting work on #70 (default viewport and engine-aligned User-Agent). This is the second sub-task of this epic, building on #69 (completed in #72).

Starting work on #70 (default viewport and engine-aligned User-Agent). This is the second sub-task of this epic, building on #69 (completed in #72).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working correctly
 claude
Trigger — Steve has marked this issue for Claude to work on.
 claude:blocked
Blocked — Waiting on a dependency (another issue/repo).
 claude:review
In Review — PR created, awaiting human review/merge.
 claude:running
Active — A Claude instance is currently working on this.
 documentation
Documentation improvements or additions
 enhancement
New feature or improvement to existing functionality
 performance
Performance optimization or resource usage improvement
 priority/critical
Showstopper — security vulnerability, data loss, or crash in production
 priority/high
Important — significant bug or high-value improvement
 priority/low
Nice to have — minor improvement or cleanup
 priority/medium
Normal — standard improvement or non-critical bug
 security
Security-related issue or vulnerability
 testing
Test coverage, test infrastructure, or test improvements
 type/epic
Parent issue grouping related stories/tasks
 type/refactor
Code restructuring without behavior change
 type/task
Concrete implementation work item
No Label enhancement priority/high type/epic
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Claude fizi steve
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/go-extractor#68
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?