Google Crawler Issues
As it turns out, the Google Crawler can't read a robots.txt file if it's been written with Visual Web Developer.
ALL of my sites that contain a robots.txt file have been removed from google crawl. I have seen a topic on the 'net about robots.txt being encoded as utf-8 rather than ANSI, but nothing I could do would cause google to crawl my sites.
In the short term, I am recommending that if you want google to crawl your site, you should remove robots.txt from your root directory.
Note that bing.com does not have this problem.
Two useful sites to use in determining if your site is getting crawled...
If any of you geniuses know how to make Google crawl an ASP.NET site with a robots.txt file, let me know.
Yahoo ID: Commander_Bombast
Opportunity is missed by most because it is dressed in overalls and looks like work.
--THOMAS ALVA EDISON
Posted 40 months ago (Thread Starter)
More...
Moderator
Bryan Valencia
Prestwood IT,
Home office in Manteca, CA
Approved member.
Member subscribes to this thread with a verified email.
About Bryan Valencia
Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.
Post ID #13834, 2 replies
Thread Started 1/15/2010 9:34:04 PM
View Counter=2441
Last Reply Posted 2/22/2010 11:43:48 PM)
Hi Bryan,
I don't get it. I think I'm missing something. Why don't you just create a robots.txt file in notepad and FTP it to the site?
Posted 39 months ago
Location=Prestwood IT office in Citrus Heights, CA,
Joined=13 years ago
MB Posts =1398
KB Posts =1490
More...
Moderator
Mike Prestwood
Prestwood IT
Prestwood IT office in Citrus Heights, CA
Approved member.
Member subscribes to this thread with a verified email.
About Mike Prestwood
Mike Prestwood is a drummer, an author, and creator of the PrestwoodBoards online community. He is the President & CEO of Prestwood IT Solutions play-drums.com and has authored 3 drum books . If you have a project you wish to discuss with Mike, you can send him a private message through his PrestwoodBoards home page or call him 9AM to 4PM PST at 916-726-5675 x205.
Web Presence
Facebook Visit Me )
Twitter Visit Me )
LinkedIn Visit Me )
Post ID #13924 (Level 1.1)
Reply to 13834
Thread Started 2/1/2010 6:32:01 PM
Most Recent Post
Why can't I create it in VWD? Why does it work fine for BING and other search engines? Why when I create a plain text file in VWD, is it stored as anything other than plain text? Sorry, what you suggest is a workaround , which is fine, but workarounds are for bugs . What I'm doing here is notifying people of a bug.
Yahoo ID: Commander_Bombast
Opportunity is missed by most because it is dressed in overalls and looks like work.
--THOMAS ALVA EDISON
Posted 39 months ago (Thread Starter)
More...
Moderator
Bryan Valencia
Prestwood IT,
Home office in Manteca, CA
Approved member.
Member subscribes to this thread with a verified email.
About Bryan Valencia
Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.
Post ID #13988 (Level 2.1)
Reply to 13924
and Parent is 13834
Thread Started 2/22/2010 11:40:27 PM
Revive Thread!
Add a comment to revive this old thread and make this archived thread more useful.
Thread #13834 Counter 2441
Since 1/15/2010
Have a question? Need our services? Contact us now.
Design
Print This
PrestwoodBoards