Spidering Stops, Site is Locked after 7 Pages

nevsie · 02-28-2005, 12:55 PM

Hi,

I am using PHPDIG and like the programme... However, i have a problem with the spidering stopping after a set amount of pages. One site it might be 7 pages another it might be 10, but never more. The site then stays in the state of being locked.

I have set:
Depth to '10'
Links per '0'
'LIMIT_TO_DIRECTORY',false
'PHPDIG_IN_DOMAIN',true

Other comments have been that you should use a browser such as FireFox to stop timeout issues... I am using Firefox, although i never see the second page where the details of page spidering arise... Does this mean the browser is timing out... If so can i change the timeout period, as i cannot access shell to run PHP DIG.

Alternatively could my host be ending the spidering. Whether it be at the mysql end or at the spidering end? I have contacted them and asked, but as of yet no reply.

Any help is truely appreciated.

Thanks N

nevsie · 03-02-2005, 04:30 AM

Checked with the hosting company and they have nothing on the mysql or scan prevention side that would stop this.

Therefore i can only assume it is to do with the browser timing out. If this is the case, what can i do about it. I have tried in various browsers from mozilla, firefox, IE and netscape...

Any help is appreciated

julien · 03-02-2005, 10:14 AM

Hi Nevsie,
Just to understand : you have installed phpdig on your website and you run it with your browser trhough the admin panel (something like www.yoursite.com/phpdig/admin/) ?
Your hosting company say that they don't block (nor timeout) the execution of the script so you think it comes from the browser ?
...
Maybe Charter can help you...

Is it possible for the browser to "timeout" the loading of a page ?

I don't know...
sorry,
Julien.

nevsie · 03-02-2005, 03:52 PM

Hi All,

As much as my hosts admitted it was not them doing anything... I was contacted by one of their higher techies...

He loved the script, and had not seen it before!!!

But he informed me that they have a maximum script execution time!!! So they did stop the script after all!!! Thats the answer, so cheers guys...

The solution is to scan groups, sub groups, or single pages at a time. Then it works great.

Cheers, N

02-28-2005, 12:55 PM	#1
nevsie Green Mole Join Date: Feb 2005 Posts: 3	Spidering Stops, Site is Locked after 7 Pages Hi, I am using PHPDIG and like the programme... However, i have a problem with the spidering stopping after a set amount of pages. One site it might be 7 pages another it might be 10, but never more. The site then stays in the state of being locked. I have set: Depth to '10' Links per '0' 'LIMIT_TO_DIRECTORY',false 'PHPDIG_IN_DOMAIN',true Other comments have been that you should use a browser such as FireFox to stop timeout issues... I am using Firefox, although i never see the second page where the details of page spidering arise... Does this mean the browser is timing out... If so can i change the timeout period, as i cannot access shell to run PHP DIG. Alternatively could my host be ending the spidering. Whether it be at the mysql end or at the spidering end? I have contacted them and asked, but as of yet no reply. Any help is truely appreciated. Thanks N

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Spider stops before all pages are indexed	halide	Troubleshooting	3	07-19-2005 12:26 AM
Indexing stops after a few pages	Sibona	Troubleshooting	1	05-03-2005 10:27 AM
Locked Site	olivier	Troubleshooting	3	02-11-2005 06:21 AM
Spidering stops after 6 level 1 pages	Steve	Troubleshooting	3	05-20-2004 04:56 PM
Indexing always stops with locked site	jamie	Troubleshooting	3	01-06-2004 10:29 AM

03-02-2005, 04:30 AM	#2
nevsie Green Mole Join Date: Feb 2005 Posts: 3	Checked with the hosting company and they have nothing on the mysql or scan prevention side that would stop this. Therefore i can only assume it is to do with the browser timing out. If this is the case, what can i do about it. I have tried in various browsers from mozilla, firefox, IE and netscape... Any help is appreciated

03-02-2005, 10:14 AM	#3
julien Green Mole Join Date: Feb 2005 Posts: 6	Hi Nevsie, Just to understand : you have installed phpdig on your website and you run it with your browser trhough the admin panel (something like www.yoursite.com/phpdig/admin/) ? Your hosting company say that they don't block (nor timeout) the execution of the script so you think it comes from the browser ? ... Maybe Charter can help you... Is it possible for the browser to "timeout" the loading of a page ? I don't know... sorry, Julien.

03-02-2005, 03:52 PM	#4
nevsie Green Mole Join Date: Feb 2005 Posts: 3	Hi All, As much as my hosts admitted it was not them doing anything... I was contacted by one of their higher techies... He loved the script, and had not seen it before!!! But he informed me that they have a maximum script execution time!!! So they did stop the script after all!!! Thats the answer, so cheers guys... The solution is to scan groups, sub groups, or single pages at a time. Then it works great. Cheers, N