Spider test for me

DrKamikaze83 · 02-17-2004, 02:13 AM

Can somebody spider http://www.ebay.com and http://www.dovebid.com and show me the result.

Spidering doesn't work for myself.

thanks
Alex

DrKamikaze83 · 02-17-2004, 06:36 AM

hi,

i have tried it for many sites in the Inet and it doesn't work.

As a last i tried it on my localhost.
On my localhost everything works wonderful.

I don't know what the problem is.

Charter · 02-17-2004, 12:16 PM

Hi. Do you get any errors when trying to index online? Is safe_mode set to on?

DrKamikaze83 · 02-18-2004, 12:08 AM

in phpinfo safe mode is off, but maybe there is something in the script tat i have forgotten to change.

Online there is no spidering possible. At any site in the internet he only detect the host like www.ebay.com and no pages. It is reagardlessof which page. It's always the same.

hi have read and tried all threads for safe_mode, but i can't arrive to get it work.
Please help me.

thanks
Alex

Charter · 02-18-2004, 12:49 AM

Hi. Setup a small three page demo like below and then index the main.html page using a search depth of one, and then wait several minutes before touching the browser. What do you see onscreen after several minutes?

http://www.domain/testdir/main.html

<html>
<body>
main page
<a href="page1.html">page1</a>
<a href="page2.html">page2</a>
</body>
</html>

http://www.domain/testdir/page1.html

<html>
<body>
page one
</body>
</html>

http://www.domain/testdir/page2.html

<html>
<body>
page two
</body>
</html>

DrKamikaze83 · 02-18-2004, 01:39 AM

hi i tried it, but it didn't work. I atarted phpdig from my localhost.

site (spidering): http://maggiv8.funpic/Test/main.html

result:
Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://maggiv8.funpic.de/
Exclude paths :
- @NONE@
No link in temporary table

--------------------------------------------------------------------------------

links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

What can i try next?

Regards
Alex

Charter · 02-18-2004, 01:54 AM

Hi. Did you configure the connect.php file that is online and try to crawl http://maggiv8.funpic.de/Test/main.html from online? The database variables in the online connect.php file need to match the online database.

DrKamikaze83 · 02-18-2004, 01:58 AM

i don't understand, what i should do know.

I have only loaded the 3 Test-files up. The other things, like database and phpdig, are on my localhost on my PC.

Regards
Alex

Charter · 02-18-2004, 02:06 AM

Hi. Perhaps try editing your hosts file like in this thread or in this thread.

DrKamikaze83 · 02-18-2004, 02:28 AM

hii charter,

i looked the two at threads.

i think, this on is the problem. http://www.phpdig.net/showthread.php?threadid=514
I didn't understand, what oscure is mentioning.

Can you give me a exact description what i have to do.

Thanks
Alex

DrKamikaze83 · 02-19-2004, 04:53 AM

hi,

i have uploaded now all to this site http://maggiv8.funpic.de/
from that site i spidered www.ebay.com.

Results:

Warning: set_time_limit,getmyuid,getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/spider.php on line 16

Spidering in progress...

Warning: set_time_limit,getmyuid,getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 97

--------------------------------------------------------------------------------
SITE : http://www.ebay.com/
Exclude paths :
- help/confidence/
- help/policies/
- disney/

Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655
1:http://www.ebay.com/
(time : 00:00:08)
+ +
level 1...

Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655
2:http://www.ebay.com/mainc1.html?ssPageName=VisitorPage
(time : 00:00:20)
+

Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655
3:http://www.ebay.com/PayPal/
(time : 00:00:27)

level 2...

Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655
4:http://www.ebay.com/es/
(time : 00:00:38)

No link in temporary table

--------------------------------------------------------------------------------

links found : 4
http://www.ebay.com/
http://www.ebay.com/mainc1.html?ssPageName=VisitorPage
http://www.ebay.com/PayPal/
http://www.ebay.com/es/
Optimizing tables...
Indexing complete !

Now i need to get it work on my PC. Help me please.

Thanks
Alex

Charter · 02-19-2004, 07:40 AM

Hi. The warnings from your online account are because your host has disabled certain functions. You can remove set_time_limit from line 16 of spider.php and from line 97 of robot_functions.php and remove the commented out line 655 in robot_functions.php.

As to crawling from your PC, perhaps try editing your Hosts file. Just do a search for the Hosts file and then add a line to the file with a text editor, something like the following:

Code:

127.0.0.1          localhost
put.the.ip.here    maggiv8.funpic.de

DrKamikaze83 · 02-19-2004, 07:56 AM

are the host data

HOST-RESOURCES-(TYPES/MIB)

or are it the http_vhost files?

is it import where it have to be written in the files?

Thanks
Alex

Charter · 02-19-2004, 08:05 AM

Hi. I've seen it as just Hosts, no extension, but I'm not sure with your OS/setup. The first entry should probably be the localhost one, but again it might depend on your OS/setup.

DrKamikaze83 · 02-19-2004, 08:13 AM

what do you mean with OS? Operating System? i have Win2000 and Apache Server 1.3.29 !

02-17-2004, 02:13 AM	#1
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	Spider test for me Can somebody spider http://www.ebay.com and http://www.dovebid.com and show me the result. Spidering doesn't work for myself. thanks Alex

02-17-2004, 12:16 PM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Do you get any errors when trying to index online? Is safe_mode set to on? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-18-2004, 12:08 AM	#4
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	in phpinfo safe mode is off, but maybe there is something in the script tat i have forgotten to change. Online there is no spidering possible. At any site in the internet he only detect the host like www.ebay.com and no pages. It is reagardlessof which page. It's always the same. hi have read and tried all threads for safe_mode, but i can't arrive to get it work. Please help me. thanks Alex Last edited by DrKamikaze83; 02-18-2004 at 12:37 AM.

02-18-2004, 12:49 AM	#5
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Setup a small three page demo like below and then index the main.html page using a search depth of one, and then wait several minutes before touching the browser. What do you see onscreen after several minutes? http://www.domain/testdir/main.html <html> <body> main page <a href="page1.html">page1</a> <a href="page2.html">page2</a> </body> </html> http://www.domain/testdir/page1.html <html> <body> page one </body> </html> http://www.domain/testdir/page2.html <html> <body> page two </body> </html> __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-18-2004, 01:54 AM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Did you configure the connect.php file that is online and try to crawl http://maggiv8.funpic.de/Test/main.html from online? The database variables in the online connect.php file need to match the online database. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-17-2004, 06:36 AM	#2
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	hi, i have tried it for many sites in the Inet and it doesn't work. As a last i tried it on my localhost. On my localhost everything works wonderful. I don't know what the problem is.

02-18-2004, 01:39 AM	#6
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	hi i tried it, but it didn't work. I atarted phpdig from my localhost. site (spidering): http://maggiv8.funpic/Test/main.html result: Spidering in progress... -------------------------------------------------------------------------------- SITE : http://maggiv8.funpic.de/ Exclude paths : - @NONE@ No link in temporary table -------------------------------------------------------------------------------- links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! -------------------------------------------------------------------------------- [Back] to admin interface. What can i try next? Regards Alex

02-18-2004, 01:58 AM	#8
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	i don't understand, what i should do know. I have only loaded the 3 Test-files up. The other things, like database and phpdig, are on my localhost on my PC. Regards Alex

02-18-2004, 02:06 AM	#9
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Perhaps try editing your hosts file like in this thread or in this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-18-2004, 02:28 AM	#10
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	hii charter, i looked the two at threads. i think, this on is the problem. http://www.phpdig.net/showthread.php?threadid=514 I didn't understand, what oscure is mentioning. Can you give me a exact description what i have to do. Thanks Alex

02-19-2004, 04:53 AM	#11
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	hi, i have uploaded now all to this site http://maggiv8.funpic.de/ from that site i spidered www.ebay.com. Results: Warning: set_time_limit,getmyuid,getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/spider.php on line 16 Spidering in progress... Warning: set_time_limit,getmyuid,getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 97 -------------------------------------------------------------------------------- SITE : http://www.ebay.com/ Exclude paths : - help/confidence/ - help/policies/ - disney/ Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655 1:http://www.ebay.com/ (time : 00:00:08) + + level 1... Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655 2:http://www.ebay.com/mainc1.html?ssPageName=VisitorPage (time : 00:00:20) + Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655 3:http://www.ebay.com/PayPal/ (time : 00:00:27) level 2... Warning: getmypid,dl,leak() has been disabled for security reasons in /usr/export/www/vhosts/funnetwork/hosting/maggiv8/admin/robot_functions.php on line 655 4:http://www.ebay.com/es/ (time : 00:00:38) No link in temporary table -------------------------------------------------------------------------------- links found : 4 http://www.ebay.com/ http://www.ebay.com/mainc1.html?ssPageName=VisitorPage http://www.ebay.com/PayPal/ http://www.ebay.com/es/ Optimizing tables... Indexing complete ! Now i need to get it work on my PC. Help me please. Thanks Alex

02-19-2004, 07:40 AM	#12
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. The warnings from your online account are because your host has disabled certain functions. You can remove set_time_limit from line 16 of spider.php and from line 97 of robot_functions.php and remove the commented out line 655 in robot_functions.php. As to crawling from your PC, perhaps try editing your Hosts file. Just do a search for the Hosts file and then add a line to the file with a text editor, something like the following: Code: 127.0.0.1 localhost put.the.ip.here maggiv8.funpic.de __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-19-2004, 07:56 AM	#13
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	are the host data HOST-RESOURCES-(TYPES/MIB) or are it the http_vhost files? is it import where it have to be written in the files? Thanks Alex

02-19-2004, 08:05 AM	#14
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. I've seen it as just Hosts, no extension, but I'm not sure with your OS/setup. The first entry should probably be the localhost one, but again it might depend on your OS/setup. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-19-2004, 08:13 AM	#15
DrKamikaze83 Green Mole Join Date: Feb 2004 Posts: 17	what do you mean with OS? Operating System? i have Win2000 and Apache Server 1.3.29 !

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Test Server	RaGe	Mod Requests	0	05-10-2004 05:01 PM
Googlebot/Test	Charter	IPs, SEs, & UAs	4	04-30-2004 12:50 PM
How do i index my test server.	silverfish	Troubleshooting	41	03-30-2004 08:43 PM
Test-Search for (little) Intelligent Php-Dig Fuzzy	Rolandks	Feedback & News	3	10-30-2003 01:21 AM