How to use the Scan feature

The Cookie Policy and Consent section of your questionnaire will ask you to scan your website. This is to identify third party technologies that your website uses.

Scanner best practices:

1. Use a sitemap xml URL for your scan- If you have one, we highly recommend adding your website's sitemap URL to perform the scan, as this can provide a more comprehensive list of scripts (and cookies) the website uses. It is recommended that you use a .xml sitemap URL (not html). At the bottom of this article, you can find popular web building platforms with instructions on how to identify your sitemap xml URL.

2. (Temporarily) Disable bot protection - If you are receiving a scan error message, try temporarily disabling bot prevention services such as Cloudflare Bot Fight mode.  Because Usercentrics needs to scan your website to identify all scripts that are placing cookies on your visitor's browsers, tools that block scanners must be temporarily disabled. Be sure to re-enable any bot protection features after the scan is complete. 

If you are using Cloudflare and want to whitelist the Usercentrics scan tool, add the following to your WAF rules:

[begin]
(cf.client.bot) or (cf.verified_bot_category in {"Search Engine Crawler" "Search Engine Optimization" "Monitoring & Analytics" "Advertising & Marketing" "Page Preview" "Academic Research" "Security" "Accessibility" "Webhooks" "Feed Fetcher"}) or (http.user_agent contains "letsencrypt" and http.request.uri.path contains "acme-challenge") or (http.user_agent eq "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/605.1.15 (compatible; Usercentrics/0.1; +http://usercentrics.com/dps-scanner/)")
[end]

More information on Usercentrics's scanner: https://support.usercentrics.com/hc/en-us/articles/14028762867228-How-to-whitelist-Usercentrics-DPS-Scanner-IP-Addresses?page=1&utm_source=chatgpt.com

Additional security features that may create scan issues include WordFence, Sucuri, or PatchStack.  We have also seen hosting companies block the scanner from loading (although this tends to be rather rare). 

3. Manually add Services - Ensuring you list all third party scripts that you are loading on your site is an essential step. Here's a step by step guide on how you can identify what scripts you're loading on your own website. Not seeing a Service in the scan results that you know your website is using? Click the purple "Add New Service" button and select which pre-defined (or custom) Service(s) you would like to have added.  You can also add custom Services as well.

Pre-defined Services are third party technologies that have been pre-answered already by Usercentrics.  

Custom Services are third party technologies that you are loading on your website that is not available within the pre-defined list of Services. 
- When adding a Custom Service, you will need to fill out the details of this Service.  You can reach out to this third party to help assist you. Also, you can use tools such as ChatGPT to help fill in these details (be sure to check sources for accuracy).
- You must set up manual script control for any non-Essential Custom Services that you add to your license to ensure consent is captured first from your visitor prior to loading the script.

The "Add Services" button can be found at the bottom of the "Cookie Policy and Consent Tool" portion of your Termageddon license (screenshot below):

4. Retry after some time passes - The Usercentrics scanner runs a significant number of scans each and every day. There's a chance that you ran the scan at a time where thousands of others were also trying to run the scan at that exact time.  You can always wait and come back and retry (for example wait 20 minutes). You can also reach out to support if you'd like more assistance. 

How to find your Sitemap:

Below are some common web building platforms and their respective sitemap URL:

WordPress websites

Default: "/wp-sitemap.xml"

Didn't work? Try: "/sitemap.xml"

If you are using WordPress, add "/sitemap.xml" to your website domain. 

So for example, if your website URL is https://fancyapples.com, then you would list https://fancyapples.com/sitemap.xml

Test it! Type in your sitemap URL into a browser and see if the page loads. The URL may load, or it may auto-forward to the correct sitemap url (such as "/wp-sitemap.xml" or "/sitemap_index.xml"). Once you've confirmed the sitemap URL, you can add it to your Scan within the questionnaire.

Are you still not seeing a sitemap on your WordPress website? You may want to consider adding one (here's a third party article discussing how to do this).

Squarespace websites

Default: "/sitemap.xml"

If you are using Squarespace, add "/sitemap.xml" to your website domain. 

So for example, if your website URL is https://fancyapples.com, then you would list https://fancyapples.com/sitemap.xml

Test it! Type in your sitemap URL into a browser and see if the page loads. Once confirmed you can add it to the "Scan URL" field within the questionnaire.

Shopify websites

Default: "/sitemap.xml"

If you are using Shopify, add "/sitemap.xml" to your website domain. 

So for example, if your website URL is https://fancyapples.com, then you would list https://fancyapples.com/sitemap.xml

Test it! Type in your sitemap URL into a browser and see if the page loads. Once confirmed, you can add it to the "Scan URL" field within the questionnaire.

Shopify provides more information on its sitemap URLs in this article.