Your agents can read more of your website

We’ve improved how your agents learn from your website. They can now find more pages and read content on sites they previously struggled with.

• More of your content. Your agents can find more blog posts, help articles and pages deeper in your site, including content that needs a browser to load. You can also add just one section of a site, such as /docs.
• Fewer distractions. Common cookie banners and labels like “Skip to main content” are left out of the saved pages. Copies of your site in other languages, such as de.example.com or /de when it opens in /en, are left out too.
• More reliable updates. If we can’t reach a page during a refresh, your agents keep using the last saved version. The rest of your site updates as usual.
• See what’s happening. Under Knowledge sources, you can see when a website is being scraped, whether something went wrong and what content your agents still have. You can scrape it again from the website’s details.

Websites your agents use refresh every week. Adding a website to an agent also triggers a refresh if its content is more than a week old.