The archive.ph submission flow had several defects that caused Mort's
summary fallback to return placeholder "Working..." pages instead of
real archived content, or hang for the full timeout:
- Context cancellation in the poll loop fell through to a final
WaitForNetworkIdle and returned the doc as success. The function now
returns a typed error (ErrArchiveIncomplete on deadline, wrapped
ctx.Err() on caller cancel).
- The poll only checked doc.URL() — if archive.ph's JS got wedged on
/wip/<id>, the loop spun until timeout. Completion now also requires
a DOM marker (#HEADER, [id^="SHARE"], .TEXT-BLOCK) so URL-only
transitions don't satisfy the check.
- The final URL is now validated against an alphanumeric ID pattern,
rejecting /wip/, /submit, /newest/ and the front page.
- 5-second blind sleep before polling replaced with a bounded
WaitForNetworkIdle that short-circuits when already archived.
- Form selectors now use a cascade (input[name='url'] →
input[type='url'] → input.input-url → input[name='anyway'], and
similar for the submit button) so a single archive.ph markup change
doesn't kill the flow. Errors name which selectors were tried.
- Default timeout lowered from 1 hour to 5 minutes (still overridable
via context deadline). Exposed as DefaultTimeout.
- Poll progress is now logged at slog.Info every 30s so production logs
surface stuck flows.
- Front-page 5xx now retries twice with 1s/4s backoff before failing.
- New exported sentinels: ErrArchiveIncomplete, ErrArchiveSelectorMissing.
- Tests cover URL validator (incl. /wip/, /newest/, short IDs, o-prefix),
selector cascade, DOM completion detector, transient status
classification, and ctx cancellation paths via a thread-safe mutating
mock document. Full integration with a live browser remains hand-tested.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds OpenPageOptions.AllowNonOKStatus. When set, openPage no longer closes
the page on non-2xx (other than 404) and Open returns both a usable Document
and ErrInvalidStatusCode. archive.IsArchived and Archive opt in, so callers
can PromoteToInteractive the captcha page, hand it to a human solver, and
demote back to extract content from the same browser instance — avoiding
the cf_clearance fingerprint-binding issue that re-challenges any fresh
retry browser.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
DuckDuckGo intermittently serves a CAPTCHA modal ("Unfortunately, bots
use DuckDuckGo too...") instead of search results. The result selector
matches zero elements on that page, so callers used to get
([]Result{}, nil) — silent empty results that look like "no matches."
Detect the challenge via the BEM class .anomaly-modal__title and return
a typed ErrBlocked so callers can distinguish blocked from no-results
and react (retry, fallback to another engine, surface to user).
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Playwright's bundled Chromium has a distinct build fingerprint (build ID,
uniform WebGL/codec lists, HeadlessChrome residue) that anti-bot services
increasingly flag. Driving a system-installed Google Chrome via Playwright's
channel option sheds that signal and aligns sec-ch-ua with UA more cleanly.
Changes:
- Add BrowserOptions.Channel string field (chrome, chrome-beta, chromium,
msedge; empty = default).
- When stealth+headless+Chromium and Channel is empty, default to "chrome"
(was "chromium"). Explicit Channel values always win, so callers can opt
back to "chromium" or pick another channel.
- Merge Channel in mergeOptions.
- Expose --channel/--ch flag on cmd/browser for A/B fingerprint testing.
Callers must have the chosen browser installed on the host
(e.g. `playwright install chrome`). Firefox and WebKit paths are untouched.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
The hardcoded DefaultChromiumUserAgent said Chrome/131 while the
installed Chromium was v136. Chromium's sec-ch-ua header is generated
from the real engine version, so sites comparing User-Agent against
sec-ch-ua detected the mismatch as bot traffic and returned 403.
Now the User-Agent is built after browser launch using browser.Version(),
keeping the Chrome/N token in sync with sec-ch-ua's Chromium;v="N".
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Chromium's Cookies() API can lose or normalize Secure, SameSite, and
HttpOnly attributes during the AddCookies → navigate → Cookies()
round-trip. This caused cookies like cf_clearance (set with
Secure=true, SameSite=None) to be overwritten with Chromium's defaults
(Secure=false, SameSite=Lax).
Now updateCookies() looks up existing cookies in the jar first. For
cookies that already exist, only Value and Expires are updated —
security attributes are preserved from the original. New cookies from
the server are still written with all their attributes.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Adds SetDefaultTimeout(time.Duration) to the InteractiveBrowser interface,
delegating to Playwright's Page and BrowserContext SetDefaultTimeout and
SetDefaultNavigationTimeout methods. This allows callers to set a timeout
so Playwright operations return an error instead of blocking forever when
the browser process crashes or the remote server becomes unresponsive.
Closes#86
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Playwright requires cookie expires to be either -1 (session cookie) or
a positive unix timestamp. When a cookie has no expiry (zero time.Time),
.Unix() returns -62135596800 which Playwright rejects. Cookies with
non-positive timestamps (e.g. Cloudflare's __cf_bm) also fail.
Now treats zero time or non-positive unix timestamps as session cookies
by setting expires to -1.
Fixes#84
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Replace deprecated Locator.Type() with PressSequentially() (node.go)
- Close page on Goto failure to prevent resource leak (playwright.go)
- Fix teardown order: close context before browser (playwright.go)
- Clean up resources on NewPage failure (interactive.go)
- Spawn cleanup goroutine on context cancellation in both constructors
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Makes MouseMove accept an optional steps variadic parameter that maps
to Playwright's MouseMoveOptions.Steps, generating intermediate
mousemove events for human-like drag behavior.
Closes#81
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Enables drag operations (mousedown → mousemove → mouseup) needed for
slider captchas and other drag-based interactions.
Closes#79
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Allow transferring ownership of a Playwright page between Document and
InteractiveBrowser modes without tearing down the browser. This enables
handing a live page to a human (e.g. for captcha solving) and resuming
scraping on the same page afterward.
Closes#76
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The Secure field was dropped in both Playwright<->internal cookie
conversion functions, causing cookies with __Secure-/__Host- prefixes
to be rejected by Chromium. Additionally, batch AddCookies meant one
invalid cookie would fail browser creation entirely.
Changes:
- Map Secure field in cookieToPlaywrightOptionalCookie and
playwrightCookieToCookie
- Add cookies one-by-one with slog.Warn on failure instead of
failing the entire batch
- Add unit tests for both conversion functions
Closes#75
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace static stealthChromiumScripts and stealthFirefoxScripts slices
with builder functions that accept hardware profile structs. Each browser
session now randomly selects from a pool of 6 realistic profiles per
engine, and Chromium connection stats receive per-session jitter (±20ms
RTT, ±2 Mbps downlink). This prevents anti-bot systems from correlating
sessions via identical WebGL, connection, mozInnerScreen, and
hardwareConcurrency fingerprints.
Closes#71
Co-Authored-By: Claude Opus 4.6 <[email protected]>
NewBrowser previously had no viewport (strong headless signal) and used a
Firefox User-Agent unconditionally, even for Chromium instances (detectable
mismatch).
Add per-engine UA constants (DefaultFirefoxUserAgent, DefaultChromiumUserAgent)
and auto-select the matching UA in initBrowser when the caller hasn't set one
explicitly. Keep DefaultUserAgent as a backward-compatible alias.
Add 1920x1080 default viewport to NewBrowser (most common desktop resolution).
NewInteractiveBrowser keeps its existing 1280x720 default but also gains
engine-aware UA selection.
Closes#70
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The stealth system previously injected all 12 init scripts unconditionally
into every browser engine. Chromium-specific scripts (window.chrome stubs,
ANGLE WebGL strings, CDP cleanup, HeadlessChrome UA strip) were no-ops or
actively suspicious on Firefox, while Firefox-specific headless vectors
were unaddressed.
Split stealthInitScripts into three categories:
- stealthCommonScripts (4): webdriver, outerWidth/Height, permissions, Notification
- stealthChromiumScripts (8): existing Chromium-specific scripts
- stealthFirefoxScripts (5): new Firefox-specific stealth:
- navigator.webdriver getOwnPropertyDescriptor hardening
- WebGL renderer spoof with Mesa/Intel strings
- mozInnerScreenX/Y non-zero spoof
- navigator.hardwareConcurrency normalization
- PDF.js plugin list override
browser_init.go now selects common + engine-specific scripts based on
opt.Browser. Tests updated with per-category validation and cross-
contamination checks.
Closes#69
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Adds a new site extractor for pizzint.watch, which tracks pizza shop
activity near the Pentagon as an OSINT indicator. The extractor fetches
the dashboard API and exposes DOUGHCON levels, restaurant activity, and
spike events.
Includes a CLI tool with an HTTP server mode (--serve) for embedding
the pizza status in dashboards or status displays.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The weather extractor used positional CSS selectors (div:first-child,
div:nth-child(2)) to locate the header and hourly container within the
widget section. When DuckDuckGo inserts advisory banners (e.g. wind
advisory), the extra div shifts positions and breaks extraction of
current temp, hourly data, humidity, and wind.
Replace with structural selectors:
- div:not(:has(ul)) for the header (first div without a list)
- div:has(> ul) for the hourly container (div with direct ul child)
These match elements by their content structure rather than position,
so advisory banners no longer break extraction.
Fixes#64
Co-Authored-By: Claude Opus 4.6 <[email protected]>
When RemoveHidden is true, JavaScript is evaluated on the live page to
remove all elements with computed display:none before readability
extraction. This defends against anti-scraping honeypots that embed
prompt injections in hidden DOM elements.
The implementation uses an optional pageEvaluator interface so that the
concrete document (backed by Playwright) supports it while the Document
interface remains unchanged.
Closes#62
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Sites with infinite scroll (e.g. The Verge) load additional articles
into the DOM, which get included in readability extraction. Add
ReadabilityOptions.RemoveSelectors to strip elements by CSS selector
before parsing, avoiding the need to reimplement the readability
pipeline downstream.
Closes#60
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add 7 new init scripts to cover WebGL fingerprinting, missing Chrome
APIs, permissions behavior, CDP artifacts, and HeadlessChrome UA string.
Enable Chromium's new headless mode (Channel: "chromium") when stealth
is active to use the full UI layer that is harder to detect.
Closes#58
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add anti-bot-detection evasion support to reduce blocking by sites like
archive.ph. Stealth mode is enabled by default for all browsers and applies
common evasions: navigator.webdriver override, plugin/mimeType spoofing,
window.chrome stub, and outerWidth/outerHeight fixes. For Chromium,
--disable-blink-features=AutomationControlled is also added.
New BrowserOptions fields:
- Stealth *bool: toggle stealth presets (default true)
- LaunchArgs []string: custom browser launch arguments
- InitScripts []string: JavaScript injected before page scripts
Closes#56
Co-Authored-By: Claude Opus 4.6 <[email protected]>
DuckDuckGo's weather widget uses randomized CSS module class names that
don't match the BEM-style selectors the extractor was using. Replace all
class-based selectors with structural and attribute-based selectors:
- Identify widget via article:has(img[src*='weatherkit'])
- Use positional selectors (div:first-child, p:first-of-type, etc.)
- Extract icon hints from img[alt] attributes
- Parse precipitation from span > span structure
- Derive CurrentTemp from first hourly entry (no standalone element)
- Derive HighTemp/LowTemp from first daily forecast entry
- Use text-matching for Humidity/Wind labels
Fixes#53
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add HourlyForecast struct and Hourly field to WeatherData for hourly
temperature/condition data. Add Precipitation (int, -1 if unavailable)
and IconHint (from aria-label/title/alt attributes) to both DayForecast
and HourlyForecast. This enables downstream consumers like mort to
replace inline DuckDuckGo scraping with a single GetWeather() call.
Closes#51
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Extract firmware information from Bambu Lab's firmware download pages
by parsing the __NEXT_DATA__ JSON blob embedded in the page. Supports
all printer models (X1, P1, A1, A1 mini, H2D, H2S, P2S, X1E, H2D Pro).
Provides GetLatestFirmware() and GetAllFirmware() methods that return
version, release date, release notes, download URL, and MD5 checksum.
Closes#45
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add sites/recipe package with ExtractRecipe() that works on any recipe
URL. Parses JSON-LD structured data (@type: Recipe) first, with DOM
fallback. Handles @graph containers, arrays, HowToStep objects, ISO
8601 durations, and various author/yield/image formats.
Closes#29
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add sites/steam package with GetGamePrice() and SearchGames() methods.
Handles regular prices, discounted games, and free-to-play titles.
Includes age gate bypass logic and currency detection.
Closes#28
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add sites/coingecko package with GetPrice() method that extracts
structured crypto price data (name, symbol, price, 24h/7d change,
market cap, volume, 24h high/low) from CoinGecko coin pages.
Includes mock-based tests and parseLargeNumber helper for T/B/M suffixes.
Closes#27
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add weather.go with GetWeather() for extracting structured weather data
(location, temp, conditions, forecast) and stock.go with GetStockQuote()
and GetStockChart() for stock data extraction and chart screenshots.
Both include mock-based tests. CSS selectors may need tuning against
the live site since DuckDuckGo's React-rendered widgets use dynamic
class names.
Closes#25, #26
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Create exported extractortest package with MockBrowser, MockDocument,
and MockNode that support selector-based responses for testing site
extractors without a real browser.
Add extraction tests for DuckDuckGo (result parsing, empty results, no
links, full search flow) and Powerball (drawing parsing, next drawing
parsing with billion/million, error cases, full GetCurrent flow).
Closes#21
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Wrap staticCookieJar in struct with sync.RWMutex for thread safety
- Add SameSite field to Cookie struct with Strict/Lax/None constants
- Update Playwright cookie conversion functions for SameSite
- Replace hardcoded 4-country switch with dynamic country code generation
Closes#20, #22, #23
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Return errors for required fields (ID, price) and log warnings for
optional fields (title, description, unit price) across all site
extractors instead of silently discarding them with _ =.
Closes#24
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Extract identical numericOnly inline functions from powerball and
megamillions into shared sites/internal/parse.NumericOnly with tests
- Extract duplicated DuckDuckGo result parsing from Search() and
GetResults() into shared extractResults() helper
Closes#13, #14
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Define DefaultUserAgent (Firefox/147.0) in playwright.go and reference
it from NewBrowser, NewInteractiveBrowser, and CLI flags. Previously
three different UA strings existed (two at 142.0, one outdated at 133.0).
Closes#17
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Change ShowBrowser from bool to *bool so nil means "don't override"
in mergeOptions(), fixing the bug where it always overwrote the base
- Add Bool() helper for convenient *bool construction
- Align NewInteractiveBrowser default from Chromium to Firefox to match
NewBrowser
- Update README example and CLI flags for the *bool change
Closes#15, #16
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- playwright.go: check error from page.Context().Cookies() before
iterating over results, preventing silent failures
- archive.go: replace time.Sleep(5s) with context-aware select using
time.After, allowing the operation to be cancelled promptly
Closes#7, #18
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Fix archive cmd passing only archive-specific Flags instead of the
merged flags variable that includes browser flags (#8)
- Move defer DeferClose() after error checks in 6 locations to prevent
calling Close on nil values (#19):
- sites/duckduckgo/cmd/duckduckgo/main.go
- sites/duckduckgo/duckduckgo.go
- sites/google/cmd/google/main.go
- sites/wegmans/cmd/wegmans/main.go
- sites/wegmans/wegmans.go
- sites/aislegopher/aislegopher.go
Closes#8, #19
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Change SearchPage.GetResults() to return ([]Result, error) so ForEach
errors are no longer silently discarded
- Fix Search() to return the ForEach error instead of nil
- Update cmd caller to check GetResults() errors
Closes#5, #6
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add length check before slicing article.Content[:32], matching the
safe truncation pattern already used in cmd/browser/main.go.
Closes#9
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- document.go: check if resp is nil before calling resp.Status() in
Refresh(), since Playwright's Reload() can return a nil response
- archive.go: check SelectFirst() results for nil before calling
Type() and Click(), preventing panics when DOM elements are missing
Closes#10, #11
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace string interpolation in SetAttribute with Playwright's Evaluate
argument passing mechanism. This structurally eliminates the injection
surface — arbitrary name/value strings are safely passed as JavaScript
arguments rather than interpolated into the expression string.
The vulnerable escapeJavaScript helper (which only escaped \ and ') is
removed since it is no longer needed.
Closes#12
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add project documentation:
- README.md with installation, usage examples, API reference, and project structure
- CLAUDE.md with developer guide, architecture overview, conventions, and issue label docs
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The go.sum file was not tracked, causing CI to fail with
"missing go.sum entry" errors during build/test/vet.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The gitea.com/actions/setup-go mirror only has tags up to v3.
v3 supports go-version-file which is all we need.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
GitHub is returning 500 errors for actions/checkout and actions/setup-go.
Switch to Gitea's own mirrors at gitea.com/actions/ to avoid the dependency.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Fix Nodes.First() panic on empty slice (return nil)
- Fix ticker leak in archive.go (create once, defer Stop)
- Fix cookie path matching for empty and root paths
- Fix lost query params in google.go (u.Query().Set was discarded)
- Fix type assertion panic in useragents.go
- Fix dropped date parse error in powerball.go
- Remove unreachable dead code in megamillions.go and powerball.go
- Simplify document.go WaitForNetworkIdle, remove unused root field
- Remove debug fmt.Println calls across codebase
- Replace panic(err) with stderr+exit in all cmd/ programs
- Fix duckduckgo cmd: remove useless defer, return error on bad safesearch
- Fix archive cmd: ToConfig returns error instead of panicking
- Add 39+ unit tests across 6 new test files
- Add Gitea Actions CI workflow (build, test, vet in parallel)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Exposes low-level mouse, keyboard, screenshot, navigation, and cookie
extraction APIs via a new InteractiveBrowser interface. Designed for
interactive browser proxy sessions where direct page control is needed.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Refactored Playwright initialization to ensure context propagation. Updated `NewPlayWrightBrowser` and related methods to accept `context.Context` for better cancellation and timeout handling. Improved error resilience and concurrency during browser setup.
Refined price parsing logic to strip trailing periods from units (e.g., "lb." -> "lb") for better handling. Added logging for debugging extracted response data.
Introduced the `UseLocalOnly` option to prevent connections to a remote Playwright server and enforce usage of the local server. Updated relevant connection logic to respect this new option.
Introduced a 30-second timeout for connecting to the Playwright server. Added logging for connection attempts to improve debugging and enhance connection reliability.
Adjusted HTML selectors for improved compatibility and updated price parsing logic to handle additional formats. Added logging to provide better debugging insights during price extraction.
Introduced the `DontLaunchOnConnectFailure` option to prevent browser launch on connection failure when connecting to a Playwright server. Enhanced environment variable handling for server address based on browser type, improving flexibility and reliability in connection scenarios.
Introduced `PlayWrightServerAddress` to `PlayWrightBrowserOptions`, allowing the browser to connect to a remote Playwright server if specified. Defaults to the `PLAYWRIGHT_SERVER_ADDRESS` environment variable. Updated initialization logic to handle both local launches and remote connections seamlessly.
Introduced the `OpenSearch` method and `SearchPage` interface to streamline search operations and allow for loading more results dynamically. Updated dependencies and modified the DuckDuckGo CLI to utilize these enhancements.
This commit introduces optional viewport dimensions and dark mode support to the PlayWrightBrowserOptions struct and its usage. It ensures more control over browser display settings and improves flexibility when configuring browser contexts. Additionally, visibility checking logic in SetHidden was refined to avoid redundant operations.
The method and its implementation now align with setting an element's "hidden" property instead of "visible." This change improves code clarity and consistency with expected behavior.
This commit introduces two new methods, SetVisible and SetAttribute, to the Node interface. These methods allow toggling element visibility and setting attributes dynamically. Additionally, a helper function, escapeJavaScript, was added to ensure proper escaping of JavaScript strings.
Introduce `ErrCommandNoReactions` to allow commands to opt out of success reactions. Adjust bot behavior to respect this error and prevent reactions when applicable, ensuring cleaner and more controlled responses. Add error handling and safeguard workers against panics.
This update enhances the `Item` structure to include `UnitPrice` and `Unit` fields. Additional logic is implemented to extract and parse unit pricing details from the HTML, improving data accuracy and granularity.
Added price field to Item struct in AisleGopher and implemented logic to extract price data. Updated Wegmans parser to validate URL structure by ensuring the second segment is "product". These changes improve data accuracy and error handling.
Introduce functionality to retrieve item details, including name and price, from Wegmans using a browser-based scraper. This includes a CLI tool to execute searches and robust error handling for URL validation and browser interactions.
Replaced the overly complex CSS selector with a simplified "h2" selector for extracting titles. This change improves maintainability and ensures accurate title extraction from the updated DOM structure.
Implemented a DuckDuckGo search module with configurable SafeSearch and regional settings. Added a CLI tool to perform searches via DuckDuckGo using browser automation, supporting flags for customization.
Moved the local package import to align with standard Go import grouping conventions. This improves code readability and maintains a consistent structure.
Introduce a Google search integration, including a Go package for performing searches with configurable parameters (e.g., language, region) and a CLI tool for executing search queries. Refactor archive CLI import ordering for consistency.
steve
