PhpDig.net - View Single Post

slintz · 08-10-2004, 12:28 PM

I've read the other posts re: slow spidering behavior and found nothing matching my situation. Please help!

After inserting traces and such into the code, I've found a consistent delay of 10 - 15 seconds for each page being indexed which occurs across a specific function call:

Code:

FILE:      robot_functions.php 
FUNCTION:  phpdigGetUrl()
STATEMENT: $answer = fgets($fp,8192);

(I've reformatted the code substantially, so I can't provide a specific line number. The fgets() occurs close to the top of the while (!$stop && !feof($fp)) { ... })

Code:

OS:      Win 2000
HTTPD:   Apache 2.0.49 (Win32)
PHP:     5.0.0
MYSQL:   4.1.3b-beta
PHPDIG:  1.8.3

As a speed check, I ran wget (cygwin) to mirror a piece of my own local site to my own drive. PhpDig took about 4 minutes to index what wget did in less than 10. Although they do different things, spidering and wget'ing are very similar which indicates that a 25:1 timing differential should not be expected...

Thanks much!

08-10-2004, 12:28 PM	#1
slintz Green Mole Join Date: Aug 2004 Posts: 4	speciffically slow spidering at fgets() I've read the other posts re: slow spidering behavior and found nothing matching my situation. Please help! After inserting traces and such into the code, I've found a consistent delay of 10 - 15 seconds for each page being indexed which occurs across a specific function call: Code: FILE: robot_functions.php FUNCTION: phpdigGetUrl() STATEMENT: $answer = fgets($fp,8192); (I've reformatted the code substantially, so I can't provide a specific line number. The fgets() occurs close to the top of the while (!$stop && !feof($fp)) { ... }) Code: OS: Win 2000 HTTPD: Apache 2.0.49 (Win32) PHP: 5.0.0 MYSQL: 4.1.3b-beta PHPDIG: 1.8.3 As a speed check, I ran wget (cygwin) to mirror a piece of my own local site to my own drive. PhpDig took about 4 minutes to index what wget did in less than 10. Although they do different things, spidering and wget'ing are very similar which indicates that a 25:1 timing differential should not be expected... Thanks much!