PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)
-   -   Spidering....links found : 0 (http://www.phpdig.net/forum/showthread.php?t=889)

Dave A 08-24-2004 11:30 AM

O files found.
 
links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

When I get this I usually delete the site from the admin page and then go back and respider it using different settings. Most times it works a treat.

But I am a new boy ay using this software...

vinyl-junkie 08-24-2004 01:31 PM

Quote:

Originally Posted by rispbiz
There only a few sites that I seem to have this problem with. Strangly enough one of the others off hand is http://www.hotmail.com

I'd be willing to bet that hotmail.com won't allow you to index them. There's a thread here in the forum about something similar, but I'm unable to find it at the moment.

rispbiz 08-24-2004 03:08 PM

That could be with hotmail.com but the result is the same with www.hotdial.net which does allow indexing and it is like the spider doesnt even try indexing the url.

Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://www.hotdial.net/
Exclude paths :
-
- @NONE@
No link in temporary table

--------------------------------------------------------------------------------

links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

If possiable could someone try indexing this site with there phpdig and let me know if they have a problem indexing it. Then I would know that if another phpdig site cant index it then it would more than likley be a problem with the url rather than an issue with my spider.

If other phpdig se can index it then I would have to figure out where the problem is in my engine.

Thanks for any help.
2-surf.net

vinyl-junkie 08-24-2004 03:53 PM

Quote:

Originally Posted by rispbiz
That could be with hotmail.com but the result is the same with www.hotdial.net which does allow indexing and it is like the spider doesnt even try indexing the url.

I don't know about that. I got the same results as you. I've been helping in the forum for quite a while, and usually if a site can be indexed, I won't have a problem doing so.

rispbiz 08-24-2004 04:06 PM

Very Strange!
 
I will work with the webmaster and see if I can get him to change the index page, and then try to reindex. Maybe there is something on the page that php dig doesn't like.

Thank you for trying the url for me and quick responses.

With both of us not being able to index the site leaves a lot of questions. Is it the site or something to do with phpdig. HUH???

Thank You
2-surf.net

vinyl-junkie 08-24-2004 05:15 PM

Quote:

Originally Posted by rispbiz
With both of us not being able to index the site leaves a lot of questions. Is it the site or something to do with phpdig. HUH???

I'm betting it's a problem with the site.

Good luck on getting it solved!

rispbiz 08-25-2004 09:45 AM

Found Problem
 
Here is what the problem was, Which makes no sense.

The website had a robot.txt file in the that only had this line.

Disallow: 4.15.191.215

I had webmaster remove the robots.txt and it indexed fine.

How come the Disallow: is causing a problem with the sipder?

Thank You,
2-Surf.net

Charter 08-25-2004 10:44 AM

Quote:

Originally Posted by rispbiz
Here is what the problem was, Which makes no sense.

The website had a robot.txt file in the that only had this line.

Disallow: 4.15.191.215

I had webmaster remove the robots.txt and it indexed fine.

How come the Disallow: is causing a problem with the sipder?

Thank You,
2-Surf.net

That is not standard robots.txt format.

vinyl-junkie 08-25-2004 04:53 PM

I think they might have been trying to ban someone from visiting their site. Trouble is, you don't do that via robots.txt. You do it via .htaccess.


All times are GMT -8. The time now is 10:31 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.