Spider Issues

LauzonRSD · 11-02-2005, 07:58 AM

I am trying to spider my site, http://massmma.net

Latest stable version of phpDig is on the same domain.

I added my site to the index list, http://massmma.net

Now it says 1 host, but 0 pages.

DataBase status
Hosts : 1 Entries
Pages : 0 Entries
Index : 0 Entries
Keywords : 0 Entries
Temporary table : 0 Entries

I asked if I was on a load balanced server, and was told I was not.

Thanks

LauzonRSD · 11-02-2005, 08:05 AM

When I added my site to the index list, http://massmma.net, with a depth of 5, and Link Per of 0.

When I tell it to dig, it gives the display

Spidering in progress... [Stop spider]
Optimizing tables...
Indexing complete ! [Back] to admin interface.

Also, I can get it to Spider Another site, subfighter.com.

Charter · 11-02-2005, 08:50 AM

A test index of your site gave the following:

Code:

Spidering in progress... [Stop spider]
SITE : http://massmma.net/
Exclude paths :
- @NONE@
1:http://massmma.net/
(time : 00:00:08)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
level 1...
2:http://massmma.net/readMore.php?id=67
(time : 00:00:29)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
3:http://massmma.net/readMore.php?id=8
(time : 00:00:42)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
level 2...
links found : 3
http://massmma.net/
http://massmma.net/readMore.php?id=67
http://massmma.net/readMore.php?id=8
Optimizing tables...
Indexing complete ! [Back] to admin interface.

It looks like your site is hosted by WinWeb, and it's a guess, but I'd say WinWeb is a reseller, and if so, it could be that whatever servers they offer do have load balancing on them.

If I had to take a guess, I'd guess that WinWeb is reselling through HE (see similar DC pics here) and that HE does have load balancing on its servers (see dedicated server features here).

If you have access to your Hosts file, try adding something like the following, where the IP is that of your site:

Code:

127.0.0.1   massmma.net

LauzonRSD · 11-02-2005, 09:26 AM

Okay. I do not have access to my hosts file, because I have no SSH access. I have emailed them asking for it, but I'm sure they will tell me no. Is there anything else I can try to do?

Charter · 11-02-2005, 10:18 AM

Try indexing the following, where LIMIT_TO_DIRECTORY is set to true in the config file:

Code:

http://www.winwebhosting.com/~massmma/

Does it index for you? If so, after it is done, edit the site_url entry in the sites table and the path entries in the spider table to reflect the direct links to your site.

LauzonRSD · 11-02-2005, 10:27 AM

I just did that. When it adds the site, its showing as http://www.winwebhosting.com, and it doesnt index.

Charter · 11-02-2005, 11:16 AM

The domain name could be changed in the table, but given it still won't index, then short of getting your host to do something or buying another hosting account, preferrably *nix and without load balancing, the next best thing might be to install Apache, MySQL, PHP, and PhpDig on the machine in front of you (if you are on Windows, check out easyphp.org) and index from there. Afterwards, make a MySQL dump of the PhpDig tables and install the dump server-side, and then FTP over the local PhpDig files to the server, changing your database information in the connect.php file, so you can do searches server-side.

LauzonRSD · 11-02-2005, 01:50 PM

I got my host to give me shell access, but only for 24 hours. I made the file /etc/hosts (there was not one there). Is this what it should include?

HTML Code:

127.0.0.1       localhost
127.0.0.1       massmma.net

Thanks... and in the mean time, I am trying what you said about indexing the site from another page. I would like to keep it all together and not have to go through indexing from somewhere else though.

Charter · 11-02-2005, 03:05 PM

The IP for massmma.net should be your site IP address:

Code:

127.0.0.1		localhost
xxx.xxx.xxx.xxx		massmma.net

Check here for some further instruction for various OS.

LauzonRSD · 11-03-2005, 07:10 AM

Okay.... and now is that their local private IP? (like 10.0.0.2) or their public IP (70.85.77.84)?

I believe it should be the private IP, but want to make sure. Its a major hassle... they gave me SSH access... but no write privileges, so Im fighting them on that now.

Charter · 11-05-2005, 12:46 PM

As the Hosts file is a mapping between names and IPs, it should probably be the IP such that http://massmma.net/ and http://xxx.xxx.xxx.xxx/ go to the same website.

11-02-2005, 07:58 AM	#1
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	Spider Issues I am trying to spider my site, http://massmma.net Latest stable version of phpDig is on the same domain. I added my site to the index list, http://massmma.net Now it says 1 host, but 0 pages. DataBase status Hosts : 1 Entries Pages : 0 Entries Index : 0 Entries Keywords : 0 Entries Temporary table : 0 Entries I asked if I was on a load balanced server, and was told I was not. Thanks

11-02-2005, 08:05 AM	#2
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	When I added my site to the index list, http://massmma.net, with a depth of 5, and Link Per of 0. When I tell it to dig, it gives the display Spidering in progress... [Stop spider] Optimizing tables... Indexing complete ! [Back] to admin interface. Also, I can get it to Spider Another site, subfighter.com. Last edited by LauzonRSD; 11-02-2005 at 08:09 AM.

11-02-2005, 08:50 AM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	A test index of your site gave the following: Code: Spidering in progress... [Stop spider] SITE : http://massmma.net/ Exclude paths : - @NONE@ 1:http://massmma.net/ (time : 00:00:08) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + level 1... 2:http://massmma.net/readMore.php?id=67 (time : 00:00:29) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 3:http://massmma.net/readMore.php?id=8 (time : 00:00:42) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + level 2... links found : 3 http://massmma.net/ http://massmma.net/readMore.php?id=67 http://massmma.net/readMore.php?id=8 Optimizing tables... Indexing complete ! [Back] to admin interface. It looks like your site is hosted by WinWeb, and it's a guess, but I'd say WinWeb is a reseller, and if so, it could be that whatever servers they offer do have load balancing on them. If I had to take a guess, I'd guess that WinWeb is reselling through HE (see similar DC pics here) and that HE does have load balancing on its servers (see dedicated server features here). If you have access to your Hosts file, try adding something like the following, where the IP is that of your site: Code: 127.0.0.1 massmma.net __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-02-2005, 10:18 AM	#5
Charter Head Mole Join Date: May 2003 Posts: 2,539	Try indexing the following, where LIMIT_TO_DIRECTORY is set to true in the config file: Code: http://www.winwebhosting.com/~massmma/ Does it index for you? If so, after it is done, edit the site_url entry in the sites table and the path entries in the spider table to reflect the direct links to your site. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-02-2005, 11:16 AM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	The domain name could be changed in the table, but given it still won't index, then short of getting your host to do something or buying another hosting account, preferrably *nix and without load balancing, the next best thing might be to install Apache, MySQL, PHP, and PhpDig on the machine in front of you (if you are on Windows, check out easyphp.org) and index from there. Afterwards, make a MySQL dump of the PhpDig tables and install the dump server-side, and then FTP over the local PhpDig files to the server, changing your database information in the connect.php file, so you can do searches server-side. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-02-2005, 09:26 AM	#4
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	Okay. I do not have access to my hosts file, because I have no SSH access. I have emailed them asking for it, but I'm sure they will tell me no. Is there anything else I can try to do?

11-02-2005, 10:27 AM	#6
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	I just did that. When it adds the site, its showing as http://www.winwebhosting.com, and it doesnt index.

11-02-2005, 01:50 PM	#8
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	I got my host to give me shell access, but only for 24 hours. I made the file /etc/hosts (there was not one there). Is this what it should include? HTML Code: 127.0.0.1 localhost 127.0.0.1 massmma.net Thanks... and in the mean time, I am trying what you said about indexing the site from another page. I would like to keep it all together and not have to go through indexing from somewhere else though.

11-02-2005, 03:05 PM	#9
Charter Head Mole Join Date: May 2003 Posts: 2,539	The IP for massmma.net should be your site IP address: Code: 127.0.0.1 localhost xxx.xxx.xxx.xxx massmma.net Check here for some further instruction for various OS. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-03-2005, 07:10 AM	#10
LauzonRSD Green Mole Join Date: Oct 2005 Posts: 6	Okay.... and now is that their local private IP? (like 10.0.0.2) or their public IP (70.85.77.84)? I believe it should be the private IP, but want to make sure. Its a major hassle... they gave me SSH access... but no write privileges, so Im fighting them on that now.

11-05-2005, 12:46 PM	#11
Charter Head Mole Join Date: May 2003 Posts: 2,539	As the Hosts file is a mapping between names and IPs, it should probably be the IP such that http://massmma.net/ and http://xxx.xxx.xxx.xxx/ go to the same website. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
explode & array issues	Graphicgreen	Coding & Tutorials	1	03-29-2007 12:35 PM
IE Issues with exclude and include tags	starpogi	Troubleshooting	0	03-24-2006 06:15 AM
Security issues	Niele	How-to Forum	1	04-25-2005 09:52 AM
Function_phpdig_form.php issues	sanfly	Troubleshooting	1	10-10-2004 09:14 AM
Authentication Issues	tdsongo	Troubleshooting	1	10-04-2004 11:40 AM