Yes, we can block ALL search engine bots from indexing all or part of your website. Contact your Solutions Manager if your robots.txt needs to specifically disallow search engine bots.

Your options:

• If your website has gated content (requiring a password/subscription to access), consider whether to allow search engines to catalog that gated content. If the publisher feels that the data is too sensitive to be captured by a search engine and possibly shared, ePublishing can lock out search engines from indexing just that locked/gated content.
• You may want to allow search engines to index your content's titles and teasers, but not the actual body of your articles. This is controlled by a setting under advanced: search.engine.request.policy.enabled. If it is set to false, the search engines will not be able to "read" or index the full content of your website. By default, this is set to true.
  • In addition to setting search.engine.request.policy.enabled to true, you can also fine-tune this further by specifying which search engines can crawl your locked content. Options are: Googlebot, Baidu, Bing, Yahoo, and Yandex. This is controlled by the Editorial Content area search.engine.request.policy. Simply remove the # in front of the crawlers you want to allow to include them in the policy, click Save, and click Clear Site Cache. If enabled, once the crawler reads your site, it will then be able to index your locked content.

Please contact your Solutions Manager if you wish to restrict your website's presence on search engines.