> ## Documentation Index > Fetch the complete documentation index at: https://docs.germainux.com/llms.txt > Use this file to discover all available pages before exploring further. # Website Crawler > GermainUX Website Crawler proactively evaluates pages across a website or web app, helping teams find quality and availability issues before users do. Components [`Engine`](https://docs.germainux.com/main/germain-ux-engine) Availability 2024.1 or later > **[Issues Detected](#issues-detected)** > > Find checkout-blocking problems > **[Website Support](#website-support)** > > Static and dynamic sites > **[Crawler Settings](#crawler-settings)** > > Tune crawl behavior > **[Operational Recommendations](#operational-recommendations)** > > Choose the right method ## Overview GermainUX Website Crawler proactively evaluates pages across a website or web application, helping teams identify quality and availability issues before they affect users. ## Issues Detected | Issue | Description | | - | - | | **Broken links** | Links that lead to missing pages, invalid destinations, or failed responses. | | **Unresponsive pages** | Pages that time out or fail to return a valid response. | | **Forbidden content** | Pages or resources that return access-denied responses or contain prohibited content. | | **Broken integrations** | External services, embedded resources, or application integrations that fail to load or respond correctly. | For an eCommerce website, GermainUX can crawl product, category, search, cart, and other customer-facing pages to identify issues that could disrupt product discovery, checkout, or conversion. ![Website Crawler Report emailed or viewable on the workspace](https://mintcdn.com/germain-ux/npHkPTFCfiH3hzwe/images/main/crawler/image-20260825-175919.png?fit=max\&auto=format\&n=npHkPTFCfiH3hzwe\&q=85\&s=2f7d9ee004573232005cf565d694a415) Website Crawler Report emailed or viewable on the workspace - GermainUX ## Website Support GermainUX supports both static and dynamic websites. | Crawling method | Recommended use | | - | - | | **HTTP request and response** | The preferred method when page quality can be evaluated directly from the server response. | | **Headless browser** | Used for dynamic websites and applications whose content depends on JavaScript execution or browser interaction. | Website Crawler can monitor websites and applications built with technologies such as WordPress, Shopify, and Salesforce Experience Cloud. ## Crawl a Website GermainUX provides two ways to start a website crawl: | Option | Description | | - | - | | **Crawl Now** | Runs an immediate crawl using a streamlined set of configuration options. | | **Schedule Crawling** | Creates a reusable crawler with scheduling and advanced configuration options. | ### Crawl Now Use **Crawl Now** to analyze a website immediately and review the results when the crawl completes. Its simplified settings are sufficient for most common use cases. Go to **GermainUX Workspace > Analytics > Website Crawler**, enter the starting URL, and launch the crawl. ![Immediate Website Crawling](https://mintcdn.com/germain-ux/npHkPTFCfiH3hzwe/images/main/crawler/image-20250814-172458.png?fit=max\&auto=format\&n=npHkPTFCfiH3hzwe\&q=85\&s=ad7bc7da6ee25ffb86a91538855afcfc) Immediate Website Crawling - GermainUX ### Schedule Crawling Use this option to execute a crawl automatically at a configurable frequency. Scheduled crawlers also support advanced settings, including: * Custom HTTP headers * Authentication * Connection settings * Crawl limits * URL inclusion and exclusion rules * Failure-status handling Go to **GermainUX Workspace > Analytics > Website Crawler**, and then click **+**. ![Schedule Website Crawling](https://mintcdn.com/germain-ux/npHkPTFCfiH3hzwe/images/main/crawler/image-20250711-082440.png?fit=max\&auto=format\&n=npHkPTFCfiH3hzwe\&q=85\&s=e11d4f9f7c17153c9cb02a8d3109fdba) Schedule Website Crawling - GermainUX ## Crawler Settings | Setting | Description | | - | - | | **URL** | Starting URL from which the crawler begins discovering pages. | | **Stay on Domain** | Controls whether the crawler follows links outside the starting domain. | | **Store Successfully Visited URLs** | Controls whether GermainUX stores successful results in addition to failed results. | | **HTTP Failure Status Code** | Minimum HTTP status code that GermainUX considers a failure. | | **Maximum Crawling Depth** | Maximum number of link levels the crawler can follow from the starting page. A blank value removes the depth limit. | | **Maximum URLs to Crawl** | Maximum number of URLs the crawler can visit. A blank value removes this limit. | | **Crawler Threads** | Number of concurrent threads used to crawl the website. Increasing this value can shorten execution time but consumes more resources and increases traffic to the target website. | | **Ignore URLs** | URLs or regular-expression patterns that the crawler must not visit. | | **Excluded Failure Status Codes** | HTTP status codes that should not be treated as failures. | | **Blocked URLs** | Specific URLs that should be excluded from crawling or failure reporting. | ### Stay on Domain | Value | Behavior | | - | - | | **True** | Visits only URLs within the starting domain and its permitted subdomains. For example, `google.com` and `drive.google.com` belong to the same domain hierarchy. | | **False** | Allows the crawler to follow links to external domains. | Enable this setting for most audits to prevent the crawler from expanding into unrelated external websites. ### Store Successfully Visited URLs | Value | Behavior | | - | - | | **True** | Stores Website URL Availability data for both successful and failed URLs. | | **False** | Stores data only for URLs that failed or were unavailable. | ### Ignore URLs Use **Ignore URLs** to prevent the crawler from visiting known, irrelevant, sensitive, or potentially disruptive destinations. Regular expressions are supported. For example, the following pattern excludes URLs containing `drive.google.com`: ```text theme={null} .*drive\.google\.com.* ``` Validate regular expressions carefully to avoid unintentionally excluding valid pages. ### Excluded Failure Status Codes Use this setting when a particular HTTP status is expected and should not be reported as a failure. For example, you can exclude selected `404` or `503` responses when they are known, intentional, and not relevant to the audit. ### Blocked URLs Use **Blocked URLs** to exclude specific pages or resources that should not be crawled or included in failure results. This can reduce false positives caused by known external services, security controls, or intentionally unavailable pages. For example, you can exclude an irrelevant Cloudflare-hosted URL whose expected response would otherwise generate unnecessary noise. ![Website Crawler: Blocked URLs](https://mintcdn.com/germain-ux/npHkPTFCfiH3hzwe/images/main/crawler/image-20250711-082619.png?fit=max\&auto=format\&n=npHkPTFCfiH3hzwe\&q=85\&s=daed8c73d9c9a503bee29f6357d5041e) ## Operational Recommendations * Confirm that you are authorized to crawl the target website. * Start with conservative URL, depth, and thread limits. * Use **Stay on Domain** unless external-link validation is required. * Exclude logout, deletion, purchase, form-submission, and other state-changing URLs. * Avoid crawling production applications at a frequency that could affect performance. * Configure authentication and custom headers securely. * Review exclusions regularly to ensure that genuine issues are not being hidden. Additional crawler parameters are available. Please [contact us](https://docs.germainux.com/main/support.md) if you need help selecting the appropriate crawling method or configuration. ## Get Help > **[Contact GermainUX Support](https://docs.germainux.com/main/support.md)** > > The Germain Team can help you set this up. > **Can't find what you need?** > > Search all GermainUX docs with **Ctrl + K**, or use the feedback buttons at the bottom of this page to tell us what is missing. The content of this document is proprietary to GERMAIN SOFTWARE. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.