Run a Website Crawl
Go to Knowledge Base in the sidebar and click Upload Document, then switch to the Scrape URL tab.
Enter the address to crawl under Data source setting URL. Tick This page only if you just want that one page, or leave it unticked to crawl the whole site from that starting point.
Use Ignore URLs when indexing to add any URL patterns that shouldn’t be crawled — common ones like /privacy-policy/, /cookie-policy/, /sitemap.xml and date archive pages are suggested by default.
Set the Category to the chatbot(s) this content should apply to, then click Upload to start the crawl.
To refresh previously crawled content, tick Rescrape Crawler before running it again.
Completed
If you are still experiencing issues, our support team is here to help.
Was this article helpful?