Spidering Problems on a Windows Server Website

[vinyl-junkie]

I haven't forgotten about the fact that I said I would try and solve that authentication problem for a site on a Windows server. Instead, I wanted to bypass the authentication process for now and just see if I could spider this website at all. Sure enough, I ran into a problem. Here's the error I'm getting.

Fatal error: Call to undefined function: is_executable() in c:\hosting\webhost4life\member\vinyljunkie2\search\admin\robot_functions.ph p on line 665

The line number that references is here, in bold:

if (USE_IS_EXECUTABLE_COMMAND == 1) {
$is_exec_command_msword = is_executable(PHPDIG_PARSE_MSWORD);

I found the code in config.php which references USE_IS_EXECUTABLE_COMMAND and changed the value for that variable to zero, but all I got was a bunch more errors after refreshing my phpdig database and trying to start over again.

Any ideas what the problem could be?

[mbruere]

Hi. PhpDig 1.6.4+ has an option in the config file to bypass is_executable. Change define('USE_IS_EXECUTABLE_COMMAND','1'); to define('USE_IS_EXECUTABLE_COMMAND','0'); in the config file.

Regards,

Mathieu

[vinyl-junkie]

Yes, I know. As I stated in my first post, I changed that to zero and just got a bunch of different errors. I have no idea why. Would it be helpful if I posted those error messages too?

[vinyl-junkie]

I'm replying to myself here. After posting earlier, I went on reading some other posts in the forum and came across a couple of others with problems similar to mine with that is_executable variable. OK, I thought. I'll just double-check what I've done and attempt to spider my site one more time.

I refreshed all the phpdig database tables, then attempted to spider my site again. This time, the spidering page showed up like this:

SITE : http://www.techtipscentral.net/
Exclude paths :
- test/
- 1ClickDBFree/
Unable to create temp directory

For what it's worth, I ended up having to drop that table and then re-create it to get the phpdig database refreshed so I could try spidering again.

Now I'm sure that I have that is_executable variable being set properly, so what is the problem this time?

[vinyl-junkie]

Sorry for so many posts, but I'm replying to myself again. I found this thread in the forums, which describes the exact same problem I'm having, including the error messages.

I tried what they said they did, creating an admin/temp directory, and also Charter's suggestion about commenting out that section of code. I'm still getting that long list of errors given in that prior post. Worse yet, it seems to only be spidering the root directory and then stopping. I have *way* more content than just one page. Help!

[Charter]

Hi. Is safe mode on? What errors do you receive?

[vinyl-junkie]

Re: safe mode. I have no idea. Is that something I would have to ask my web host? I didn't find anything about that in their knowledge base. If safe mode is on, is there no way around it?

Here are the errors I'm getting (full directory extension is replaced by "pathname" here):

Warning: fopen(../admin/temp/27583251.tmp): failed to open stream: Permission denied in c:\pathname\search\admin\robot_functions.php on line 705

Warning: fwrite(): supplied argument is not a valid stream resource in c:\pathname\search\admin\robot_functions.php on line 707

Warning: fclose(): supplied argument is not a valid stream resource in c:\pathname\search\admin\robot_functions.php on line 709

Warning: filesize(): Stat failed for ../admin/temp/27583251.tmp (errno=2 - No such file or directory) in c:\pathname\search\admin\robot_functions.php on line 710

Notice: Undefined variable: revisit_after in c:\pathname\search\admin\spider.php on line 374
1:http://www.techtipscentral.net/
(time : 00:00:06)
No link in temporary table

[Charter]

Hi. Check your phpinfo for safe_mode to see if it's on or off. Also check that the following directories have 777 permission:

[PHPDIG_DIR]/text_content
[PHPDIG_DIR]/include
[PHPDIG_DIR]/admin/temp

[vinyl-junkie]

safe_mode is off for both local and master value.
safe_mode_exec_dir is off for both.
safe_mode_gid is off for both.
safe_mode_include_dir has "no value" for both.
open_basedir has "no value" for both.
disable_functions has "no value" for both.
disable_classes has "no value for both.

All three of those directories you had me check have permission level 777.

I was reading some of the stuff from the link you posted. Is it possible that the reason I'm unable to execute these commands is that I created the PHPDIG directories with the username/password that I was given for my web host account, but the database username/password are different? When it comes to this sort of thing, I'm way out of my element.

[Charter]

Hi. What is allow_url_fopen set to in the phpinfo?

[vinyl-junkie]

That is set to On for both local and master values.

[Charter]

>> ...I created the PHPDIG directories with the username/password that I was given for my web host account...

Hi. Do you mean that you logged into your account and then created the directories? Who's the owner of the directories? Perhaps send an email to your host. For some reason it seems that the directories are not accessible to the script.

[vinyl-junkie]

I contacted my Web host, and they changed the permissions for my account. That did the trick, and now I have spidered my site!

I just have to add that I have one of the best Web hosts around. I turned in the support ticket about 10:15 p.m. tonight, and received a response about 45 minutes later. This isn't the first time either that I have received fast service like that. Pretty cool, huh?

Now, to look at that authentication screen issue with IIS that I said I would work on....

[vinyl-junkie]

But...

And here is where Windows is so darn picky, I'm testing the actual search page now and getting the following error:

Undefined offset: 1 in pathname\search\libs\search_function.php on line 468

Line 468 in search_function.php looks like this:

list($title,$text) = explode("\n",$first_words);

I've discovered already with PHP scripts that Windows is very fussy about making sure variables are initialized before calling a function. However, I initialized $title and $text to nulls at the start of the function, but that didn't help. Any ideas on how to fix this?

[Charter]

Hi. What do you get when you call the following from the browser using your Windows account?

PHP Code:

<?php
$test = "This is\na test.";
list($first,$second) = explode("\n",$test);
echo $first . "<br>" . $second;
?>