I.T. Discussion Community!
-Collapse +Expand
Design
Search Design Group:

Advanced
-Collapse +Expand Design Store
PRESTWOODSTORE

Prestwood eMagazine

December Edition
Subscribe now! It's Free!
Enter your email:

   ► MB LobbyWebsite Design & Hosting BoardWebsite Design HTML, XML, CSS, Graphics Topic   Print This     

Google Crawler Issues

Google Crawler Issues in Website Design HTML, XML, CSS, Graphics topic (part of our Website Design & Hosting group).

Quick Search: Crawler   Google Crawler   Google Crawler Issues  
Moderator
Bryan Valencia
209 Software
Home office in Manteca, CA

As it turns out, the Google Crawler can't read a robots.txt file if it's been written with Visual Web Developer.

ALL of my sites that contain a robots.txt file have been removed from google crawl.  I have seen a topic on the 'net about robots.txt being encoded as utf-8 rather than ANSI, but nothing I could do would cause google to crawl my sites.

In the short term, I am recommending that if you want google to crawl your site, you should remove robots.txt from your root directory.

Note that bing.com does not have this problem.

Two useful sites to use in determining if your site is getting crawled...

If any of you geniuses know how to make Google crawl an ASP.NET site with a robots.txt file,  let me know.

Yahoo ID: Commander_Bombast

Opportunity is missed by most because it is dressed in overalls and looks like work.

--THOMAS ALVA EDISON

 

 Posted 9 years ago (Thread Starter)
Comment Quote
About Bryan Valencia -Collapse +Expand
Visit Profile
Approved member.
Member subscribes to this thread with a verified email.
About Bryan Valencia

Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.

Web Presence
LinkedIn, Bryan On LinkedIn (Visit Me)
Website, Software Services (http://209software.com/)
Website, Visual Studio Journey (http://vstudiojourney.blogspot.com/)

Post ID #13834, 3 reply
Thread Started 1/15/2010 9:34:04 PM
View Counter=6490
Last Reply Posted 2/2/2019 2:51:33 AM)
Location=Home office in Manteca, CA 
Joined=12 years ago   MB Posts=23   KB Posts=27   KB Comments=4   BLOG, Topics=2  
Moderator
Mike Prestwood
Prestwood IT
Prestwood IT office in Citrus Heights, CA

Hi Bryan,

I don't get it. I think I'm missing something. Why don't you just create a robots.txt file in notepad and FTP it to the site?

--
Mike Prestwood
Prestwood IT Solutions

 Posted 9 years ago
Comment Quote
About Mike Prestwood -Collapse +Expand
Visit Profile
Approved member.
Member subscribes to this thread with a verified email.
About Mike Prestwood

Mike Prestwood is a drummer, an author, and creator of the PrestwoodBoards online community. He is the President & CEO of Prestwood IT Solutions. Prestwood IT provides Coding, Website, and Computer Tech services. Mike has authored 6 computer books and over 1,200 articles. As a drummer, he maintains play-drums.com and has authored 3 drum books. If you have a project you wish to discuss with Mike, you can send him a private message through his PrestwoodBoards home page or call him 9AM to 4PM PST at 916-726-5675 x205.

Web Presence
Facebook, Prestwood IT Facebook page -- fan page. (Visit Me)
Twitter, Follow Prestwood IT on Twitter. (Visit Me)
LinkedIn, Prestwood IT company page on LinkedIn. (Visit Me)
YouTube, Prestwood IT YouTube Channel (Visit My Channel)
Website, My drum website where I sell my drum books. (http://www.play-drums.com)

Post ID #13924 (Level 1.1)  Reply to 13834
Thread Started 2/1/2010 6:32:01 PM
Location=Prestwood IT office in Citrus Heights, CA 
Joined=19 years ago   MB Posts=1410   KB Posts=1805   KB Comments=69   BLOG, Topics=4  
Moderator
Bryan Valencia
209 Software
Home office in Manteca, CA
  1. Why can't I create it in VWD?
  2. Why does it work fine for BING and other search engines?
  3. Why when I create a plain text file in VWD, is it stored as anything other than plain text?

Sorry, what you suggest is a workaround, which is fine, but workarounds are for bugs.  What I'm doing here is notifying people of a bug.

Yahoo ID: Commander_Bombast

Opportunity is missed by most because it is dressed in overalls and looks like work.

--THOMAS ALVA EDISON

 

 Posted 9 years ago (Thread Starter)
Comment Quote
About Bryan Valencia -Collapse +Expand
Visit Profile
Approved member.
Member subscribes to this thread with a verified email.
About Bryan Valencia

Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.

Web Presence
LinkedIn, Bryan On LinkedIn (Visit Me)
Website, Software Services (http://209software.com/)
Website, Visual Studio Journey (http://vstudiojourney.blogspot.com/)

Post ID #13988 (Level 2.1)  Reply to 13924 and Parent is 13834
Thread Started 2/22/2010 11:40:27 PM
Location=Home office in Manteca, CA 
Joined=12 years ago   MB Posts=23   KB Posts=27   KB Comments=4   BLOG, Topics=2  
Most Recent Post
dencypriya
-- --

Hiii...

  • Site Errors: This section of the report shows the main issues for the past 90 days that prevented Googlebot from accessing your entire site (click any box to display its chart).
     
  • URL Errors: This section lists specific errors Google encountered when trying to crawl specific desktop or phone pages. Each main section in the URL Errors reports corresponds to the different crawling mechanisms Google uses to access your pages, and the errors listed are specific to those kinds of pages.

openstack Training | VMware Training | Android Training | Oracle SCM Online Training | Salesforce Admin Training |Salesforce Admin Online | MongoDB Admin Training | MongoDB Admin Online Training

 Posted 38 months ago
Comment Quote
About dencypriya -Collapse +Expand
Visit Profile
Membership pending.
Member subscribes to this thread but email is NOT VERIFIED.
Email Not Verified!
Once email is verified, we will review and approve the account.
Web Presence Hidden.
Once above is taken care of, full Profile content will display including back links, about me, my message, custom Profile html, social networking links, message board signature, company profile, etc.

Post ID #15431 (Level 2) and Parent is 13834
Reply Posted 10/17/2016 11:50:36 PM
Location=-- -- 
Joined=38 months ago   MB Posts=1  

Revive Thread!

Add a comment to revive this old thread and make this archived thread more useful.

Write a Comment...
Full Editor
...
Sign in...

If you are a member, Sign In. Or, you can Create a Free account now.


Anonymous Post (text-only, no HTML):

Enter your name and security key.

Your Name:
Security key = P1151A1
Enter key:
Icon: A Post    Thread    Idea    Important!    Cool    Sad    No    Yes    Includes a Link...   
Thread #13834 Counter
6490
Since 1/15/2010

Follow PrestwoodBoards on: 


©1995-2019 PrestwoodBoards  [Security & Privacy]
Professional IT Services: Coding | Websites | Computer Tech