Date:  10/07/2007 10:21:12 PM Msg ID:  003524
From:  FoxWeb Support Thread:  003523
Subject:  Re: Intercepting Googlebot
Your FoxWeb error log should contain the SPECID in the QUERY_STRING item of the Server Variables page.  You should be able to reconstruct the URL this way and try it from a browser.
FoxWeb Support Team
support@foxweb.com email
Sent by Joe Goldsmith on 10/05/2007 10:15:27 PM:
Perhaps this is an interesting question to ponder. I have a retail site with about 3000 items for sale. Each item is kept in a table row. I have a page that, when called, shows all the details about a specific item.

Now, I have all of my items for purchase in a special XML file that the googlebot opens and scans and catalogs all items for sale. This is a great way for Google to index all the items for searches. An XML snippit looks like this:

 <url>
  <loc>http://www.justdeejays.com/product-specifications.fwx?SPECID=5C9D17C3F0804040</loc>
  <lastmod>2007-07-26</lastmod>
  <priority>0.84</priority>
  <changefreq>monthly</changefreq>
 </url>


Thus all items in this XML file has a <URL>-</URL> for each of all 3000 items. The problem I am having is that somewhere there is an error(s) that the googlebot encounters and is logged in the FoxWeb error pages. However, the error pages do not tell me what item IDs are causing the errors.

So is there a way to identify when the googlebot scans my site and is there a way to log item IDs that fail the scan?

Any help would be appreciated.