🕸 GermainUX Website Crawler
GermainUX Website Crawler proactively evaluates pages across a website or web application, helping teams identify quality and availability issues before they affect users.
Issues Detected
|
Issue |
Description |
|---|---|
|
Broken links |
Links that lead to missing pages, invalid destinations, or failed responses. |
|
Unresponsive pages |
Pages that time out or fail to return a valid response. |
|
Forbidden content |
Pages or resources that return access-denied responses or contain prohibited content. |
|
Broken integrations |
External services, embedded resources, or application integrations that fail to load or respond correctly. |
For an eCommerce website, GermainUX can crawl product, category, search, cart, and other customer-facing pages to identify issues that could disrupt product discovery, checkout, or conversion.
⚙️ Website Support
GermainUX supports both static and dynamic websites.
|
Crawling method |
Recommended use |
|---|---|
|
HTTP request and response |
The preferred method when page quality can be evaluated directly from the server response. |
|
Headless browser |
Used for dynamic websites and applications whose content depends on JavaScript execution or browser interaction. |
Website Crawler can monitor websites and applications built with technologies such as WordPress, Shopify, and Salesforce Experience Cloud.
🚀 Crawl a Website
GermainUX provides two ways to start a website crawl:
|
Option |
Description |
|---|---|
|
Crawl Now |
Runs an immediate crawl using a streamlined set of configuration options. |
|
Schedule Crawling |
Creates a reusable crawler with scheduling and advanced configuration options. |
⚡ Crawl Now
Use Crawl Now to analyze a website immediately and review the results when the crawl completes. Its simplified settings are sufficient for most common use cases.
Go to GermainUX Workspace > Analytics > Website Crawler, enter the starting URL, and launch the crawl.
📅 Schedule Crawling
Use this option to execute a crawl automatically at a configurable frequency.
Scheduled crawlers also support advanced settings, including:
|
Setting |
|---|
|
Custom HTTP headers |
|
Authentication |
|
Connection settings |
|
Crawl limits |
|
URL inclusion and exclusion rules |
|
Failure-status handling |
Go to GermainUX Workspace > Analytics > Website Crawler, and then click +.
🎚 Crawler Settings
|
Setting |
Description |
|---|---|
|
URL |
Starting URL from which the crawler begins discovering pages. |
|
Stay on Domain |
Controls whether the crawler follows links outside the starting domain. |
|
Store Successfully Visited URLs |
Controls whether GermainUX stores successful results in addition to failed results. |
|
HTTP Failure Status Code |
Minimum HTTP status code that GermainUX considers a failure. |
|
Maximum Crawling Depth |
Maximum number of link levels the crawler can follow from the starting page. A blank value removes the depth limit. |
|
Maximum URLs to Crawl |
Maximum number of URLs the crawler can visit. A blank value removes this limit. |
|
Crawler Threads |
Number of concurrent threads used to crawl the website. Increasing this value can shorten execution time but consumes more resources and increases traffic to the target website. |
|
Ignore URLs |
URLs or regular-expression patterns that the crawler must not visit. |
|
Excluded Failure Status Codes |
HTTP status codes that should not be treated as failures. |
|
Blocked URLs |
Specific URLs that should be excluded from crawling or failure reporting. |
📍 Stay on Domain
|
Value |
Behavior |
|---|---|
|
True |
Visits only URLs within the starting domain and its permitted subdomains. For example, |
|
False |
Allows the crawler to follow links to external domains. |
Enable this setting for most audits to prevent the crawler from expanding into unrelated external websites.
💾 Store Successfully Visited URLs
|
Value |
Behavior |
|---|---|
|
True |
Stores Website URL Availability data for both successful and failed URLs. |
|
False |
Stores data only for URLs that failed or were unavailable. |
⛔ Ignore URLs
Use Ignore URLs to prevent the crawler from visiting known, irrelevant, sensitive, or potentially disruptive destinations.
Regular expressions are supported. For example, the following pattern excludes URLs containing drive.google.com:
.*drive\.google\.com.*
Validate regular expressions carefully to avoid unintentionally excluding valid pages.
✅ Excluded Failure Status Codes
Use this setting when a particular HTTP status is expected and should not be reported as a failure.
For example, you can exclude selected 404 or 503 responses when they are known, intentional, and not relevant to the audit.
🔒 Blocked URLs
Use Blocked URLs to exclude specific pages or resources that should not be crawled or included in failure results. This can reduce false positives caused by known external services, security controls, or intentionally unavailable pages.
For example, you can exclude an irrelevant Cloudflare-hosted URL whose expected response would otherwise generate unnecessary noise.
📋 Operational Recommendations
|
Recommendation |
|---|
|
Confirm that you are authorized to crawl the target website. |
|
Start with conservative URL, depth, and thread limits. |
|
Use Stay on Domain unless external-link validation is required. |
|
Exclude logout, deletion, purchase, form-submission, and other state-changing URLs. |
|
Avoid crawling production applications at a frequency that could affect performance. |
|
Configure authentication and custom headers securely. |
|
Review exclusions regularly to ensure that genuine issues are not being hidden. |
Additional crawler parameters are available. Please contact us if you need help selecting the appropriate crawling method or configuration.
Service: Automation
Feature Availability: 2024.1