User Submitted URLs Mod Version 1 - Page 2

davey147 · 08-29-2004, 04:19 PM

I got the above working, well kind of,

I set the cron job up, and set it to send me the results, which come back to me and the say successfull.

But when i look to see if they have actually been added, they havent, so i have no extra sites in my search engine,

Any Ideas

rispbiz · 08-29-2004, 06:58 PM

What happens if you try to run your cron in a shell.

The best way to make sure any cron is going to work is to first try the command in a shell. If that works then use the command in a cron

I have been running the above code in a shell for about 3 days non-stop with no problem. I am still unsure of why I still can not do the same thing from a txt file but this seems to work well. There is no doubt that every server has there own personality its all a matter of finding out what works best on your server. Which sometimes can be nerve wrecking.

Thank You
2-surf.net

davey147 · 08-30-2004, 03:13 AM

Hi,

Unfortionatly i dont have shell access i dont think, i dont really know what it is,

But the cron is working as i get the report email saying that is has done what it should be doing.

But nevermind thanks anyway

cr0bar · 08-31-2004, 09:15 AM

Hey.. i'm having problems with both of these scripts. At the moment i'm trying to get the .pl one to work
it executes and says its completed with no error.. but it comes up instantly.. does this mean its not working. Also.. the url's aren't been added to the site.

cr0bar · 08-31-2004, 09:15 AM

ps.. i have shell axx

rispbiz · 08-31-2004, 10:35 AM

The pl above will not show errors if there is errors like doucment already indexed or reasons why phpdig does not completely index sites according to your config file they will not show with this pl.

Its shows finished indexing when the spider.php is finished with the url reguardless of what the spider did.

On thing you can do to test the script is run this commmand php -f /path/to/admin/spider.php http://www.somedomain.com

If the results are good then the script should be working fine.

Just for referance I am currently indexing 1 level and 1 link. and it takes average of 30 seconds to complete one site.

Thanks
2-surf.net

cr0bar · 08-31-2004, 10:53 AM

ok.. it takes like .5 of a second. lol

something is up.. but i don't have a clue what. i've added sites to the db.. but when it runs.. it just deletes them and doesn't add to the site =/

rispbiz · 08-31-2004, 11:04 AM

Yes,

There is a problem indexing if it is doing it in that small amount of time.

Sounds like the pl is working if it is deleting the urls.

What happens when you index from shell command

php -f /path/to/admin/spider.php http://www.somedomain.com

Thanks
2-Surf.net

cr0bar · 08-31-2004, 11:16 AM

aphrodite# php -f /usr/nfs/domains/f/****MASKED****/html/admin/spider.php http://www.url.com
aphrodite#

straight away

cr0bar · 08-31-2004, 11:25 AM

my provider installed php on that box for me.. just basic pack. cos this shell server is nfs'd to another box for web.

So.. that line is executed on one machine.. but the website runs off another.

Do i need any extentions added to php on that machine?

rispbiz · 08-31-2004, 01:41 PM

That response does not look correct.

when you index in the shell it should look like the same results as if you were indexing it through the web browser.

Thank you,
2-surf.net

08-29-2004, 06:58 PM	#17
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	Cron Job Tip What happens if you try to run your cron in a shell. The best way to make sure any cron is going to work is to first try the command in a shell. If that works then use the command in a cron I have been running the above code in a shell for about 3 days non-stop with no problem. I am still unsure of why I still can not do the same thing from a txt file but this seems to work well. There is no doubt that every server has there own personality its all a matter of finding out what works best on your server. Which sometimes can be nerve wrecking. Thank You 2-surf.net __________________ Sometimes the shortest way home is the longest way around! Thank you PhpDig for a great search engine! www.2-surf.net Last edited by rispbiz; 08-29-2004 at 07:03 PM.

08-31-2004, 10:35 AM	#21
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	.pl Above The pl above will not show errors if there is errors like doucment already indexed or reasons why phpdig does not completely index sites according to your config file they will not show with this pl. Its shows finished indexing when the spider.php is finished with the url reguardless of what the spider did. On thing you can do to test the script is run this commmand php -f /path/to/admin/spider.php http://www.somedomain.com If the results are good then the script should be working fine. Just for referance I am currently indexing 1 level and 1 link. and it takes average of 30 seconds to complete one site. Thanks 2-surf.net __________________ Sometimes the shortest way home is the longest way around! Thank you PhpDig for a great search engine! www.2-surf.net

08-31-2004, 11:04 AM	#23
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	Right there is a indexing prob. Yes, There is a problem indexing if it is doing it in that small amount of time. Sounds like the pl is working if it is deleting the urls. What happens when you index from shell command php -f /path/to/admin/spider.php http://www.somedomain.com Thanks 2-Surf.net __________________ Sometimes the shortest way home is the longest way around! Thank you PhpDig for a great search engine! www.2-surf.net Last edited by rispbiz; 08-31-2004 at 11:09 AM.

08-31-2004, 01:41 PM	#26
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	Results That response does not look correct. when you index in the shell it should look like the same results as if you were indexing it through the web browser. Thank you, 2-surf.net __________________ Sometimes the shortest way home is the longest way around! Thank you PhpDig for a great search engine! www.2-surf.net

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
um new user help	mspmsp	How-to Forum	1	09-16-2004 07:35 PM
User Titles	Charter	The Mole Hole	0	01-30-2004 11:55 PM
Mod Proxy (English version)	Iltud	Mod Submissions	1	09-20-2003 12:56 PM
Mod Proxy (Version française)	Iltud	Mod Submissions	1	09-20-2003 12:53 PM

08-29-2004, 04:19 PM	#16
davey147 Green Mole Join Date: Feb 2004 Posts: 6	I got the above working, well kind of, I set the cron job up, and set it to send me the results, which come back to me and the say successfull. But when i look to see if they have actually been added, they havent, so i have no extra sites in my search engine, Any Ideas

08-30-2004, 03:13 AM	#18
davey147 Green Mole Join Date: Feb 2004 Posts: 6	Hi, Unfortionatly i dont have shell access i dont think, i dont really know what it is, But the cron is working as i get the report email saying that is has done what it should be doing. But nevermind thanks anyway

08-31-2004, 09:15 AM	#19
cr0bar Green Mole Join Date: Aug 2004 Posts: 5	Hey.. i'm having problems with both of these scripts. At the moment i'm trying to get the .pl one to work it executes and says its completed with no error.. but it comes up instantly.. does this mean its not working. Also.. the url's aren't been added to the site.

08-31-2004, 10:53 AM	#22
cr0bar Green Mole Join Date: Aug 2004 Posts: 5	ok.. it takes like .5 of a second. lol something is up.. but i don't have a clue what. i've added sites to the db.. but when it runs.. it just deletes them and doesn't add to the site =/

08-31-2004, 11:16 AM	#24
cr0bar Green Mole Join Date: Aug 2004 Posts: 5	aphrodite# php -f /usr/nfs/domains/f/**MASKED**/html/admin/spider.php http://www.url.com aphrodite# straight away

08-31-2004, 11:25 AM	#25
cr0bar Green Mole Join Date: Aug 2004 Posts: 5	my provider installed php on that box for me.. just basic pack. cos this shell server is nfs'd to another box for web. So.. that line is executed on one machine.. but the website runs off another. Do i need any extentions added to php on that machine?