Optimizely
CRAWLER
Quickly Index Your Site Content and Start Searching
Index your website pages, PDFs, Office documents and more with SearchStax Site Search Crawler.
CRAWLER
Quickly Index Your Site Content and Start Searching
Index your website pages, PDFs, Office documents and more with SearchStax Site Search Crawler.
Easily ingest data
The Site Search Crawler can automatically discover all web pages, content and documents on your website by crawling through your site’s links and internal navigation as well as XML sitemaps generated by your CMS.
The Site Search Crawler can automatically discover all web pages, content and documents on your website by crawling through your site’s links and internal navigation as well as XML sitemaps generated by your CMS.
Crawl across your website
Site Search Crawler can index any public-facing content on your website - regardless of what content management system you’re using.
Popular CMSs and DXPs
Beyond webpages
Build search experiences fast with Crawler
Start now and schedule for later
You can start crawling your site immediately so you can get up and running quickly. Then schedule daily crawls to keep your site search indexes up-to-date.
Data extraction
Site Search Crawler extracts page content, titles and metadata automatically. You can also:
- Add your own metadata fields (like OG tags, author info and publish date)
- Use Xpath to extract data from specific sections of your site
- Use CSS selectors to pull data from specific HTML tags
- Define exclusions to avoid crawling certain portions of your site
- Respect robots.txt to ensure only the right content goes into your index
Start now and schedule for later
You can start crawling your site immediately so you can get up and running quickly. Then schedule daily crawls to keep your site search indexes up-to-date.
Data extraction
Site Search Crawler extracts page content, titles and metadata automatically. You can also:
- Add your own metadata fields (like OG tags, author info and publish date)
- Use Xpath to extract data from specific sections of your site
- Use CSS selectors to pull data from specific HTML tags
- Define exclusions to avoid crawling certain portions of your site
- Respect robots.txt to ensure only the right content goes into your index
Gear up with marketer-friendly tech
Site Search Crawler enables marketing teams to create engaging search experiences across their entire digital ecosystem.
Click to crawl
Simply enter a starter URL and Crawler will start populating your search index.
Daily updates
As your content continually changes, Crawler will keep your search results fresh with a daily crawl.
Clear visibility
Easily monitor your crawls to ensure pages are indexed so your site health and search quality stay high.
There are more Site Search features to explore
Connectors
Connect to any web CMS, DXP or custom HTML site with our Crawler, pre-built connectors and APIs.
Search Experience
Give your visitors instant access to exactly the information they’re looking for.
Boost and Promote
Boost your top performing content on your search results page with easy-to-use tools for marketers