Connect custom website to Rovo

This connector allows you to do a search crawl and index of your website to show in Rovo Search results and use in Rovo Chat and Agents.

What is indexed?

The custom website connector indexes these objects:

• Web pages (mime type: text/html)

• Text files (mime type: text/plain)

• Name

• URL

• Created date

• Last updated date

• Description

• Page content

Before you begin

• This connector can only crawl secure (https) websites that you own.

• To ensure you own the domain or subdomain, you need to be able to edit the robots.txt file in the website you’d like to crawl.

• We encourage you to review the pages that are available on your website. All Rovo users will have access to all content available to the crawler (including content using any configured authentication).

Editing your robots.txt

You need to be able to edit the robots.txt file on your website. If you’re unsure what a robots.txt file is, see How to write a robots.txt file, or talk to your website admin.

At minimum, you’ll need to add the following to your existing robots.txt file on your website:

Note that User-agent: * will not permit crawling alone - the atlassian-bot line must be included.

You can additionally add specific allow or disallow rules to the robots.txt and the connector will follow these rules, for example:

This rule would allow Rovo to crawl and index every public page on your site except the content under /not-useful/.

Connecting and crawling your website

To get to the setup screen for your custom website in Atlassian Admin:

1. Go to Atlassian Administration. Select your organization if you have more than one.

2. Select Apps > AI settings > Rovo.

3. Under Sites, next to the site you want to connect, select Add connector.

4. Select Custom website and press Next.

To setup your crawl:

1. Enter a website name for the site you’d like to crawl.

2. Add the full URL of the domain. Include the protocol (https://).

3. Choose how often Rovo should index your site.

4. Choose your authentication method and fill in any applicable fields. See Authentication options below for more details.

5. Review and agree to the data usage information.

6. Select Connect.

Next steps

After you’ve finished setting up the crawl:

1. The crawling and indexing of your site will start immediately.

2. Pages will start to show in Search incrementally for you and your team over the next few hours.

3. Depending on the number of pages on your website, it may take some time for all your website’s content to be indexed and appear in Search.

Authentication options

The connector currently supports crawling sites with:

• No authentication (public sites)

• Basic authentication (username/password, without a login page)

• Form authentication (a login page requiring a username/password)

Basic and form authentication are suitable when your organisation has sites that aren’t public, but also don’t require individual permissions (for example, some intranets or internal knowledge bases).

You will still need to edit the robots.txt file on your authenticated site.

Form authentication

To setup form authentication, you’ll need to get some details about the login form you see on your website.

• Login URL: This is the URL where the user would see a login screen. It can be a subdomain or a different URL to the Domain URL. It must start with https://.

• 4 selector fields:

  • Username selector: Input field where the user would type in their username

  • Password selector: Input field where the user would type in their password

  • Username submit selector (optional): If the password field is not visible initially on the login screen, this is the button that the user has to press after entering a username. An example is the Atlassian login screen (you may need to open this in an incognito window to see).

  • Password submit selector: The button to submit the username and password

What are selector fields?

Selectors identify the username and password fields in the webpage’s code to enable the connector to log in to your site.

Some example valid selectors are:

• #email

• #password

To set up form-based authentication, you’ll need to copy and fill in at least three selectors (username, password, and password submit) to setup form authentication.

How to find the selectors from your site

To copy the selectors for your login screen from Chrome:

1. Open your Login URL in a non-logged-in browser (or an incognito window).

2. Right-click the applicable text box or the submit button, then select Inspect.

3. An elements panel will appear on the right side, with an element highlighted. This element is usually:

   1. <input> for the username and password fields

   2. <button> for the submit fields

4. Right-click on the highlighted element of the code, and select Copy > Copy selector.

5. Paste it into the appropriate field in admin.atlassian.com.