Robots.txt missing some entries
Problem reported by Simon Sprott - 7/30/2017 at 2:26 AM
The community area link for "Report Abuse" "Edit Reply" etc launch a java script form, but also present a link which appends 'ThreadPanels' to the url and returns the same page. When spiders crawl the site they keep traversing these links recursively.
This causes a number of issues, they sometime keep digging quite a long way generating quite a lot of traffic, also the SEO/crawl reports I've looked at don't like them.
Could you add a rule to the robots.txt file you generate to stop spiders looking at links with 'ThreadPanels' in them.
Maybe something like this
Disallow: /community/*/ThreadPanels/
Sample urls being requested.
http ://www.example.com/community/a10/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E

2 Replies

Reply to Thread
Derek Curtis Replied
Employee Post
This will be resolved in the release on December 3rd in the next version of SmarterTrack.
Derek Curtis COO SmarterTools Inc. (877) 357-6278 www.smartertools.com
Andrew Barker Replied
Employee Post Marked As Resolution
This issue was resolved in SmarterTrack Build 6910.
Andrew Barker Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com

Reply to Thread