Some sites won't index - Page 2

Charter · 10-13-2003, 05:23 PM

Quote:

Was recently indexed

Did it index the first time? Are you trying to reindex?

vvvvv · 10-13-2003, 05:30 PM

>Did it index the first time?

Nope.

This is all I ever got and I've tried it on several different URLs.

SITE : http://www.mysite.com/
Exclude paths :
- @NONE@
No link in temporary table

mike221 · 10-13-2003, 05:42 PM

I had the same problem but it went away after performing the mods published Here .

My sever (Where phpdig is) : Apache/1.3.28 (Unix) mod_auth_passthrough/1.8 mod_gzip/1.3.26.1a mod_log_bytes/1.2 mod_bwlimited/1.0 PHP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.15 OpenSSL/0.9.7a on Linux.

I still have problems indexing a couple of servers running Netscape out of 265 servers with all kind of configurations.

Good Luck

vvvvv · 10-13-2003, 05:58 PM

OK I'll give that a spin. Thanks for the suggestion mike221

Looks like a late night cup of coffee for me.

vvvvv · 10-13-2003, 06:33 PM

OK I did the mods but still the same.

Any other ideas? Again much appreciate the help.

rayvd · 10-14-2003, 07:32 AM

You've probably already checked this... there's no robots.txt file on your server preventing the crawling is there?

Tanasja · 10-17-2003, 03:40 AM

Hi,

Think I have the same problem. The first time the indexing went fine. Then I changed some filenames. When reindexing, the old filenames were taken and the new ones skipped. Also when I index directly indexed the new filename the index couldn't find it.

I read the posts on this item and tried the following things:
- delete en reindex site (several ways)
- delete en reinstalling database
- empty dir text_content (not keepalive) and dir admin/temp (which stayed empty when reindexing)
- change config: LIMIT_DAYS=1 and PHPDIG_DEFAULT_INDEX=false
- run spider.php from a browser

I also see suggestions like:
- lynx from command line
- adjusting the routing table on the machine with the webserver

I don't understand these suggestions. Can anybody exlpain them? Or are there other options left? Maybe useful information: I host my sides at a provider.

Anybody can help?
Greetings from Amsterdam,
Tanasja

Charter · 10-17-2003, 03:52 PM

Hi vvvvv. Maybe this is a JavaScript issue? Does setting PHPDIG_DEFAULT_INDEX to false have any effect?

Hi Tanasja. Just to be sure, when you say "the old filenames were taken and the new ones skipped" are the new links in the files you are trying to index?

web newsroom · 05-19-2004, 12:35 PM

Im sure that its something quite simple.

I have been able to successfully spider certain sites and currently show the following stats.

Last Run : May 19, 2004
Pages : 5025 Entries
Index : 1397195 Entries
Keywords : 230416 Entries
Temporary : 110440 Entries

However, I still cannot seem to spider our own site.

A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped.

10-13-2003, 05:42 PM	#18
mike221 Green Mole Join Date: Oct 2003 Posts: 3	I had the same problem but it went away after performing the mods published Here . My sever (Where phpdig is) : Apache/1.3.28 (Unix) mod_auth_passthrough/1.8 mod_gzip/1.3.26.1a mod_log_bytes/1.2 mod_bwlimited/1.0 PHP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.15 OpenSSL/0.9.7a on Linux. I still have problems indexing a couple of servers running Netscape out of 265 servers with all kind of configurations. Good Luck Last edited by mike221; 10-13-2003 at 05:48 PM.

10-17-2003, 03:52 PM	#23
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi vvvvv. Maybe this is a JavaScript issue? Does setting PHPDIG_DEFAULT_INDEX to false have any effect? Hi Tanasja. Just to be sure, when you say "the old filenames were taken and the new ones skipped" are the new links in the files you are trying to index? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

05-19-2004, 12:35 PM	#24
web newsroom Green Mole Join Date: May 2004 Posts: 4	Same Problems Im sure that its something quite simple. I have been able to successfully spider certain sites and currently show the following stats. Last Run : May 19, 2004 Pages : 5025 Entries Index : 1397195 Entries Keywords : 230416 Entries Temporary : 110440 Entries However, I still cannot seem to spider our own site. A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped. __________________ http://www.thewebnewsroom.com

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Trying to index some dynamic sites	guillaume	Troubleshooting	2	08-08-2007 05:40 AM
PHPDig won't index most sites and only go down one level on all	confusion	Troubleshooting	1	10-14-2005 10:32 AM
I just want to index main sites	afesh	How-to Forum	1	08-26-2005 08:45 PM
"I don't want to index your sites!!!" - said PHPDig	#ASH	How-to Forum	1	04-06-2005 01:57 PM
index intershop-sites?	comko	Troubleshooting	4	03-30-2004 08:22 AM

10-13-2003, 05:30 PM	#17
vvvvv Green Mole Join Date: Oct 2003 Posts: 6	>Did it index the first time? Nope. This is all I ever got and I've tried it on several different URLs. SITE : http://www.mysite.com/ Exclude paths : - @NONE@ No link in temporary table

10-13-2003, 05:58 PM	#19
vvvvv Green Mole Join Date: Oct 2003 Posts: 6	OK I'll give that a spin. Thanks for the suggestion mike221 Looks like a late night cup of coffee for me.

10-13-2003, 06:33 PM	#20
vvvvv Green Mole Join Date: Oct 2003 Posts: 6	OK I did the mods but still the same. Any other ideas? Again much appreciate the help.

10-14-2003, 07:32 AM	#21
rayvd Green Mole Join Date: Oct 2003 Location: Mesa, AZ Posts: 15	You've probably already checked this... there's no robots.txt file on your server preventing the crawling is there?

10-17-2003, 03:40 AM	#22
Tanasja Green Mole Join Date: Oct 2003 Location: Amsterdam Posts: 9	Hi, Think I have the same problem. The first time the indexing went fine. Then I changed some filenames. When reindexing, the old filenames were taken and the new ones skipped. Also when I index directly indexed the new filename the index couldn't find it. I read the posts on this item and tried the following things: - delete en reindex site (several ways) - delete en reinstalling database - empty dir text_content (not keepalive) and dir admin/temp (which stayed empty when reindexing) - change config: LIMIT_DAYS=1 and PHPDIG_DEFAULT_INDEX=false - run spider.php from a browser I also see suggestions like: - lynx from command line - adjusting the routing table on the machine with the webserver I don't understand these suggestions. Can anybody exlpain them? Or are there other options left? Maybe useful information: I host my sides at a provider. Anybody can help? Greetings from Amsterdam, Tanasja