PhpDig.net - Spidering Problems on a Windows Server Website

Page 2 of 2

Show 40 post(s) from this thread on one page

PhpDig.net (http://www.phpdig.net/forum/index.php)

- Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)

- - Spidering Problems on a Windows Server Website (http://www.phpdig.net/forum/showthread.php?t=496)

vinyl-junkie

02-14-2004 12:32 PM

With just that code in a web page, I get this:

Quote:

This is
a test.

Charter

02-14-2004 12:53 PM

Hi. In robot_functions.php $first_words is created as follows:

PHP Code:


		
			
$first_words = $titre_resume."\n".ereg_replace('(@@@.*)','',wordwrap($page_desc['content'].$text[0], SUMMARY_LENGTH, '@@@'));

Try checking the first_words column of the spider table to verify that something like title \n some text appears in the column.

vinyl-junkie

02-14-2004 01:06 PM

This is actually a very small website (only 20 spider-able files right now), so I was able to browse the entire "spider" table. There is no entry in the first_words field with "\n" anywhere in that field.

Charter

02-14-2004 01:29 PM

>> There is no entry in the first_words field with "\n" anywhere in that field.

Hi. The \n is actually a newline. Did you see anything like the following in the first_words column?

title

some text

vinyl-junkie

02-14-2004 05:07 PM

Yes, but only just a few entries. Most just have the page name. For example, the first_words fields for one entry has ContactMe.asp. Here's one that had the page description as part of that field.

Quote:

CD Trustee On The Web: Introduction Would you like to have your CD Trustee music database display on your website as dynamic content pages? I've written a tutorial to show you how.

If this could be broken out between title and text, it would be like so:

Quote:

CD Trustee On The Web: Introduction

Would you like to have your CD Trustee music database display on your website as dynamic content pages? I've written a tutorial to show you how.

vinyl-junkie

02-14-2004 05:30 PM

Don't know if this helps or not (should have mentioned it earlier), but when I spidered this site it seemed to take a really long time (about 3 or 4 minutes), and we're talking about a site with around 20 pages total. With my Unix site, I could spider a few hundred pages in the same length of time.

Also, just for grins I decided to update the spidering just now and got this error:

Quote:

HTTP/1.1 502 Gateway Error Server: Microsoft-IIS/5.0 Date: Sun, 15 Feb 2004 02:18:08 GMT Connection: close Content-Length: 186 Content-Type: text/html

CGI Timeout

The specified CGI application exceeded the allowed time for processing. The server has deleted the process.

Charter

02-14-2004 05:48 PM

Hi. It looks like you may be running into the issue listed in point one of this thread, for which a full solution is currently not available. Perhaps the 'undefined offset' error is related to CGI timeout. One suggestion is to delete the site from the admin panel, empty the tables, and set LIMIT_DAYS to zero in the config.php file and then crawl on a per page basis where search depth is zero (available since version 1.6.5) or one. Also, if interested see this thread about the LIMIT_DAYS constant.

vinyl-junkie

02-14-2004 06:41 PM

I decided to clear the tables and reset that config.php value you suggested, then spider the site the conventional way. If that hadn't worked, I would have gone and spidered the pages individually as you suggested. It did work though, and the site searches seem to be okay now.

A Windows site is definitely touchy. :rolleyes: Now I know why my main website is on Unix.

Charter

02-20-2004 06:44 PM

OT: Thanks vinyl-junkie for helping here and elsewhere! :D

All times are GMT -8. The time now is 12:56 PM.

Page 2 of 2

Show 40 post(s) from this thread on one page