How long?

sjSJ · 05-11-2004, 05:07 AM

I have installed PhpDig on my winxp box running Apache .
I have installed a robots.txt in the root dir.
When trying to spider my site I get a list of all the directories which are off limits and the URL becomes locked.

How long will "Spidering" take ? Must be less than 500kb of html pages. Should this take minutes or longer?

Regards
Steve

vinyl-junkie · 05-11-2004, 07:09 PM

Welcome to the forum, sjSJ. We're glad you joined us.

At the risk of stating the obvious, the length of time it takes to spider will be dependent on how many pages there are to spider.

That said, it really should take only a few seconds per page.

Is nothing happening at all for you when the spider starts? You should be getting displays every few seconds of the each page that phpdig is processing.

It's possible you don't have your folder permissions set correctly. Check the documentation to see if that might be the problem. Another possibility is that the spidering simply got hung. That happened to me the first time I tried to spider one of my sites. However, I cleared all the tables, started the spider again, and everything turned out okay.

Hopefully, one of these suggestions will be the key. Let us know if you still have problems. We'll be glad to help.

05-11-2004, 05:07 AM	#1
sjSJ Green Mole Join Date: May 2004 Posts: 1	How long? I have installed PhpDig on my winxp box running Apache . I have installed a robots.txt in the root dir. When trying to spider my site I get a list of all the directories which are off limits and the URL becomes locked. How long will "Spidering" take ? Must be less than 500kb of html pages. Should this take minutes or longer? Regards Steve

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Long Time Spidering Crashes	mikiwiki	Troubleshooting	1	10-26-2006 10:25 AM
Long Call help.	GunMuse	How-to Forum	0	07-26-2006 11:11 AM
Long post, possible issues, and maybe the data to solve some?	Paul D. Buck	Troubleshooting	4	02-03-2005 12:59 PM
PDFs in directory listed as one long entry	motopsycho	External Binaries	4	03-09-2004 06:07 PM

05-11-2004, 07:09 PM	#2
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Welcome to the forum, sjSJ. We're glad you joined us. At the risk of stating the obvious, the length of time it takes to spider will be dependent on how many pages there are to spider. That said, it really should take only a few seconds per page. Is nothing happening at all for you when the spider starts? You should be getting displays every few seconds of the each page that phpdig is processing. It's possible you don't have your folder permissions set correctly. Check the documentation to see if that might be the problem. Another possibility is that the spidering simply got hung. That happened to me the first time I tried to spider one of my sites. However, I cleared all the tables, started the spider again, and everything turned out okay. Hopefully, one of these suggestions will be the key. Let us know if you still have problems. We'll be glad to help.