Site Crawl
Crawl your website to automatically build your AI agent's knowledge base. Access it from Admin > Site Crawl in your admin panel.
Overview
The site crawler visits your website pages, extracts their content, and creates searchable AI embeddings. This lets your AI agent answer questions based on the information on your website.
How It Works
The crawler runs a 3-phase pipeline:
| Phase | Description | Progress |
|---|---|---|
| 1. Crawl | Visits pages and saves HTML | 0-50% |
| 2. Analyze | Detects repeating content (headers, footers, menus) | 50-60% |
| 3. Embed | Generates searchable AI embeddings | 60-100% |
Crawler Settings
| Setting | Description |
|---|---|
| Website URL | Starting URL for the crawl. Pre-filled from your saved website address; changes you make here are saved back as your website address |
| Max Pages | Maximum pages to crawl: 1 to 1000 (default 100) |
| Max Depth | How many links deep to follow: 1 to 10 (default 3) |
| Fresh start | Delete the pages and knowledge saved by earlier crawls first. On by default on this page |
Whether a site needs a headless browser is detected by the crawler itself. To force the browser, or to send cookies for pages behind a login, use the crawler options described in Crawler Options.
The page also shows how many crawls you have left today, this week and this month.
Fresh start
Fresh start (delete existing crawl data) removes everything earlier crawls saved for your account before the new crawl begins: the crawled pages and the knowledge built from them. Documents you uploaded are not touched.
- Leave it ticked when you are replacing your content, for example after a redesign or a move to a new address.
- Untick it to keep what is there and add to it.
- The box is ticked each time you open the page, so check it before you click Start Crawl.
Reset State
Reset State appears next to Start Crawl only after a crawl ended in an error or was stopped. Click it to clear that result so the page goes back to its normal idle state. It does not delete any crawled content; it only clears the crawl's status.
Storefront password
If your Shopify store is behind its password page (a store that has not launched yet), the crawler cannot read it. When that is detected, a notice appears above Start Crawl (and on the first step of the Setup Wizard):
- Click Enter password.
- Type the Storefront password. You can find it in Shopify under Online Store → Preferences → Restrict store access.
- Click Save. The password is checked against your store before it is kept.
If the saved password stops working, for example because it was changed in Shopify, the notice says so; click Enter password and enter the new one. When a password is already saved the button reads Change password. If you save a password and the store turns out not to be password protected, a short note says so and no password is needed.
Start Crawl
- Enter or verify your Website URL
- Adjust Max Pages and Max Depth as needed
- Keep Fresh start ticked only if replacing all content
- Click Start Crawl
- Monitor progress through all 3 phases
- Review results when complete
The crawl runs in the background — you can close the page and return later to check progress.
Failed URLs
Some URLs may fail during crawling:
- Pages blocked by robots.txt
- Login-required pages
- Dead links (404 errors)
Those pages are left out of your knowledge base. If the crawl as a whole fails, its error message shows on this page.
Best Practices
- Crawl first, then fine-tune — Start with a broad crawl, then add documents and FAQs to fill gaps
- Use Fresh start deliberately — Only when completely replacing all crawled content
- Check your robots.txt — Make sure it doesn't block the crawler from important pages
- Monitor failed URLs — Review failures to identify pages that need cookies or browser mode (see Crawler Options)
Related Pages
- AI Documents - Upload PDFs and documents
- AI FAQs - Add FAQs to improve responses
- Crawler Options - Advanced crawler settings
- Improve AI Responses - Tips for better AI answers