Web sources

A web source is a website you crawl so its content is indexed and added to your agents’ knowledge. Once a source is crawled and ready, agents in the workspace can draw on its pages to answer users.

Web sources live under Sources, alongside Documents and Resource libraries, and belong to the workspace.

Optional feature — ask to have it enabled. Web sources are turned on per workspace. If you don’t see Web sources under Sources in Studio, the feature isn’t enabled for your workspace yet — contact the Bayes team to request it.

How agents use web sources

A crawled web source is indexed as knowledge in the workspace. Once its Embedding Status is Ready, agents can retrieve relevant pages from it to answer users. Use tags to organize your sources and control how they are used. Note that, unlike an uploaded document, a crawled web source is never offered to users as a download — its pages have no stored file — so the reserved public-documents tag has no download effect on a web source.

The full flow at a glance

The walkthrough below replays every step in the real interface. Use Prev / Next to move at your own pace; each step highlights the button to click and the area to watch.

Step by step

1. Open Web sources

In Studio, open the left sidebar, expand the Sources section, then select Web sources.

2. Crawl a website

1. Click Crawl Website (top-right of the page).
2. In the Crawl a Website dialog, enter the Website URL (for example https://example.com). The entire website is indexed in the background.
3. Optionally give the source a Name (otherwise the site is used).
4. Click Start Crawling.

The new source appears in the table right away.

3. Wait until it’s ready

Each source shows an Embedding Status that moves through these values:

• Crawling — pages are being fetched. Once counting starts it reads Crawling — N pages.
• Embedding — the fetched pages are being indexed.
• Ready — indexing is done; the source is fully usable by your agents.
• Failed — indexing failed; re-run it with Reprocess (see step 4).

Only a Ready source is fully usable by your agents. The table also shows the number of Pages crawled and the last update time.

4. Manage a source

Open the ⋮ menu at the end of a source’s row to:

• View — see the source’s details.
• Edit — rename the source and add tags (see below).
• Reprocess — re-run indexing on a source. This item appears only when the Embedding Status is Failed, and starts again immediately (no confirmation).
• Stop crawl — cancel a crawl that’s still running. This item appears only while the source is Crawling and takes effect immediately (no confirmation).
• Recrawl website — crawl the site again to pick up changes. Shown when the source is not currently crawling. (Stop crawl and Recrawl website never appear together — one replaces the other depending on whether a crawl is running.)
• Delete — remove the source and its indexed content.

5. See the crawled pages

Click the arrow (chevron) at the start of a source’s row to expand it and list every page that was crawled, each linking to its URL.

6. Tag a source

Tags let you group and control sources.

1. In the source’s ⋮ menu, choose Edit.
2. In the edit dialog, click Add tag and pick a tag.
3. Click Update.

The tag then appears in the source’s Tags column.

7. Create and manage tags

1. On the Web sources page, click Tags (top-right) to open the tags panel.
2. Click Create tag.
3. Enter a Name and an optional Description, then click Create.
4. From the tags panel you can Edit or Delete any tag at any time.

The reserved public-documents tag only makes uploaded Documents downloadable by users in a conversation. A crawled web source can’t be downloaded even when tagged, because its pages have no stored file.

Tips

• Crawl the most relevant sections of a site — the whole site is indexed, so a focused, well-structured site gives the best answers.
• Recrawl after the website changes so your agents stay up to date.
• Use tags to keep large sets of sources organized and to control how your agents use them.

Troubleshooting

• An agent doesn’t use a source — check that its Embedding Status is Ready; a source still Crawling or Embedding isn’t available yet.
• The site changed but answers are stale — open the ⋮ menu and choose Recrawl website.
• A crawl is stuck or was started by mistake — open the ⋮ menu and choose Stop crawl (shown only while the source is still Crawling).
• A source shows Failed — open the ⋮ menu and choose Reprocess to re-run indexing.
• A crawl won’t start — the Website URL must be a valid https:// address.