Comments on: What is your robots.txt file telling your competitors about you? http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/ Sat, 13 Mar 2010 00:07:03 +0000 http://wordpress.org/?v=2.7 hourly 1 By: Evans http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-9426 Evans Mon, 05 May 2008 20:32:36 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-9426 Nice article! Its actually surprising how many people rely solely on their robots.txt file to help protect very vital information in the systems from being accessed. Bear in mind that not all crawlers and bots respect the robots.txt files on web servers. In fact, I've seen a worrying number of sites where a Chinese originated crawler has been hammering sites disregarding the instructions in those robots.txt file. Webmasters and site owners should remember that security is not a destination, but an ever ending journey. Evans Nice article! Its actually surprising how many people rely solely on their robots.txt file to help protect very vital information in the systems from being accessed.

Bear in mind that not all crawlers and bots respect the robots.txt files on web servers. In fact, I’ve seen a worrying number of sites where a Chinese originated crawler has been hammering sites disregarding the instructions in those robots.txt file.

Webmasters and site owners should remember that security is not a destination, but an ever ending journey.

Evans

]]> By: 69 ways to jump on the online money market - Strip Blog | Strip Blogging http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6973 69 ways to jump on the online money market - Strip Blog | Strip Blogging Mon, 25 Feb 2008 18:43:28 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6973 [...] What is your robots.txt file telling your competitors about you? [...] [...] What is your robots.txt file telling your competitors about you? [...]

]]> By: Weekly Round Up - Stories you Might Have Missed » Hybrid Search Engine Marketing | SEO Consulting to Help Rank Better http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6644 Weekly Round Up - Stories you Might Have Missed » Hybrid Search Engine Marketing | SEO Consulting to Help Rank Better Fri, 15 Feb 2008 11:16:20 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6644 [...] What is your robots.txt file telling your competitors about you? - Jennifer Slegg [...] [...] What is your robots.txt file telling your competitors about you? - Jennifer Slegg [...]

]]> By: paisley http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6632 paisley Thu, 14 Feb 2008 19:11:26 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6632 try noindex noarchive and nofollow.. metas for exclusion also on the example above by jim matheson.. be sure and put up an index page on the /noindex/ directory.. or chmod your directory where it won't be read.. try noindex noarchive and nofollow.. metas for exclusion

also on the example above by jim matheson.. be sure and put up an index page on the /noindex/ directory.. or chmod your directory where it won’t be read..

]]> By: (EMP) E-Marketing Performance » : » Team Reading List 2.14.08 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6625 (EMP) E-Marketing Performance » : » Team Reading List 2.14.08 Thu, 14 Feb 2008 15:42:38 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6625 [...] What is your robots.txt file telling your competitors about you? [...] [...] What is your robots.txt file telling your competitors about you? [...]

]]> By: Jenstar http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6623 Jenstar Thu, 14 Feb 2008 15:25:04 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6623 Andy, I bet you there are a ton of small webmasters, particularly the self-taught DIY ones whose skills go not much beyond how to set up a basic website, out there that have no idea what you just said or how to do it... they simply pay their ten bucks a month hosting and their coding efforts don't go beyond simple HTML with an editor ;) I think this is the reason many "mom and pop" website owners dismally fail the robots.txt thing, because they don't know how to password protect directories because their html editor program doesn't have it as an option. Maybe "how to password protect a directory" should be an upcoming article! Andy, I bet you there are a ton of small webmasters, particularly the self-taught DIY ones whose skills go not much beyond how to set up a basic website, out there that have no idea what you just said or how to do it… they simply pay their ten bucks a month hosting and their coding efforts don’t go beyond simple HTML with an editor ;)

I think this is the reason many “mom and pop” website owners dismally fail the robots.txt thing, because they don’t know how to password protect directories because their html editor program doesn’t have it as an option. Maybe “how to password protect a directory” should be an upcoming article!

]]> By: MMMeeja: Andy Murdoch http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6616 MMMeeja: Andy Murdoch Thu, 14 Feb 2008 09:46:17 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6616 I would hope that professional webmasters would do all of their development and testing on private networks. A linux server with Apache, MySql etc costs peanuts and means you can test your sites with reduced security, greater debugging output and automated tools. Even if you need to show off your site designs to remote customers, you can do that over VPN. Remember - when you've put something on the public internet, you can't undo that. I would hope that professional webmasters would do all of their development and testing on private networks.

A linux server with Apache, MySql etc costs peanuts and means you can test your sites with reduced security, greater debugging output and automated tools. Even if you need to show off your site designs to remote customers, you can do that over VPN.

Remember - when you’ve put something on the public internet, you can’t undo that.

]]> By: Jenstar http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6590 Jenstar Wed, 13 Feb 2008 20:41:58 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6590 I don't think seeing what is in a competitor's robots.txt file is breaching ethics... after all, it is a publicly accessible file on the site, as would be any page on a website unless it is password protected or required authentication somehow. However, it would be a personal decision if you would go to a new directory you spied in the robots.txt to snoop what was there, keeping in mind that the competitor could potentially track it back to you if you did view the "secret but unprotected" directory without a proxy. I don’t think seeing what is in a competitor’s robots.txt file is breaching ethics… after all, it is a publicly accessible file on the site, as would be any page on a website unless it is password protected or required authentication somehow. However, it would be a personal decision if you would go to a new directory you spied in the robots.txt to snoop what was there, keeping in mind that the competitor could potentially track it back to you if you did view the “secret but unprotected” directory without a proxy.

]]> By: Paul Burani, Clicksharp Marketing http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6587 Paul Burani, Clicksharp Marketing Wed, 13 Feb 2008 19:14:31 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6587 Now THAT is some insight. As always, the white hats will have to decide if snooping around competitors' robots.txt files constitutes a breach of ethics--but I could venture a guess... And it's good defensive practice anyway. Now THAT is some insight. As always, the white hats will have to decide if snooping around competitors’ robots.txt files constitutes a breach of ethics–but I could venture a guess… And it’s good defensive practice anyway.

]]> By: g1smd http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/comment-page-1/#comment-6578 g1smd Wed, 13 Feb 2008 12:39:53 +0000 http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#comment-6578 Since you don't need to specify the full URL in robots.txt, just enough to match the path, I often use just the first three letters, or so, in the robots.txt file. For example, the scripts folder is disallowed with: Disallow: /scr even though it might actually be called /script or /scripts or /script3. Since you don’t need to specify the full URL in robots.txt, just enough to match the path, I often use just the first three letters, or so, in the robots.txt file.

For example, the scripts folder is disallowed with:
Disallow: /scr
even though it might actually be called /script or /scripts or /script3.

]]>