2
Robots.txt missing some entries
Problem reported by Simon Sprott - 7/30/2017 at 2:26 AM
Resolved
The community area link for "Report Abuse" "Edit Reply" etc launch a java script form, but also present a link which appends 'ThreadPanels' to the url and returns the same page. When spiders crawl the site they keep traversing these links recursively.
 
This causes a number of issues, they sometime keep digging quite a long way generating quite a lot of traffic, also the SEO/crawl reports I've looked at don't like them.
 
Could you add a rule to the robots.txt file you generate to stop spiders looking at links with 'ThreadPanels' in them.
Maybe something like this
Disallow: /community/*/ThreadPanels/
 
Sample urls being requested.
http ://www.example.com/community/a10/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
http ://www.example.com/community/a10/ThreadPanels/ThreadPanels/ThreadPanels/ThreadPanels/%3Ca%20target='_blank'%20href='https :/www.example.com/my-downloads'%3Ehttps :/www.example.com/my-downloads%3C/a%3E
etc
 

2 Replies

Reply to Thread
0
Derek Curtis Replied
Employee Post
This will be resolved in the release on December 3rd in the next version of SmarterTrack.
Derek Curtis COO SmarterTools Inc. www.smartertools.com
0
Andrew Barker Replied
Employee Post Marked As Resolution
This issue was resolved in SmarterTrack Build 6910.
Andrew Barker Software Developer SmarterTools Inc. www.smartertools.com

Reply to Thread