Spider site with links

dell_10 · 09-08-2004, 05:56 AM

Hi..
i need help ?????????????????????
1) I have 2 web sites and phpdig run on solaris machine when we indexing Server A (Windows 2000 Advances Server ) evry thing OK but whem im try index B (Windows 2003 Server) On this Server im only have folder includes MS word and PDF Files and only on HTML page this page include links to this documents and PDF Files ,,, i'm Try to index this page http://serverB/Folder/
got only fisrt page indexed and No Links Found

Im posting my COnfig File and wish any one can help me

2) i need to know if there is and hosting companies support phpdig so i can put phpdig there thanks

dell_10 · 09-10-2004, 10:17 PM

Whta ???????????
Any one can help me or what ?????

dell_10 · 09-11-2004, 04:33 AM

Hi..
i need help ?????????????????????
1) I have 2 web sites and phpdig run on solaris machine when we indexing Server A (Windows 2000 Advances Server ) evry thing OK but whem we try index Server B (Windows 2003 Server) [ On this Server we only have folder includes MS word and PDF Files and only on HTML page this page include links to this documents and PDF Files ,,, We Try to index this page http://serverB/Folder/page.htm
got only fisrt page indexed and No Links Found
Im posting our Config File and wish any one can help me

2) i need to know if there is any hosting company support phpdig so i can put phpdig there thanks

Charter · 09-11-2004, 06:01 AM

1) http://www.phpdig.net/forum/showthread.php?t=799

2) PhpDig can run on a lot of, but not all, remote servers

dell_10 · 09-11-2004, 06:49 AM

Dear ...
i don't have any problem with using external binaries my problem is when im trying to index another server (.NEt Server) include folder contain sub folders this sub folder contain files (word, xls, pdf ) and the main folder contain html page with links to this files when im start index this server and put the path to this html page i think phpdig will spider and follow links frm this page and index filse .. i try it on another server (2000 Advance server) and it's work but i can't make it done on this .NET Server the result only indexing of the page without any links
and when im try to index any file bu put the full path of the file phpdig can index it

asanad · 09-11-2004, 07:35 AM

Dear all:

I am a new commer to this forum, I really like PHPDig very much.
Still I have a problem in one of my servers, whenever I try to crawl that specific Server, PHPDig just crawls the first page then it stops going deeper in the levels. It simply gives me "No temporary table links" or something like that.
For example, I have a document called "doc1.txt" its under the folder "test" in "Server A". When I tell PHPDig to index "http://ServerA/test/doc1.txt" ot indexes the file. Now, i need to index all the documents, so I created a file called "index.html", this file caontains a link to all the documents under the folder "test" , for example, "doc1.txt, doc2.txt,doc3.htm,doc3.pdf, doc4.doc". PHPDig for some reason indexes only the "index.html", it does not go and crawl the links within the page.

Could you please give me a hint, why this is happening.
many thanks for your help and support.

Charter · 09-11-2004, 07:50 AM

If PhpDig can index the HTML page but won't follow the links and index the DOC, XLS, and PDF files, then that seems to indicate an external binary problem on the one server.

asanad · 09-11-2004, 08:05 AM

Thank you for your quick reponse,

I forgot to mention to you that PHPDig is residing on a Unix server, it is able to completely index many other servers sucessfully. This case is only for a specific server.

So, all the binaries and conversions work on the PHPDig Unix Server, it is just the issue that one of the destination servers does not crawl properly.

Regards

Charter · 09-11-2004, 08:23 AM

Okay, that changes my response.

Some thoughts: check the destination server request and error logs, and also check the MIME type mappings on the destination server.

dell_10 · 09-12-2004, 03:56 PM

Hi

My problem is:
i have 3 servers: Unix(Phpdig Server),2 windows Servers (A,B)
PhpDig crawl Server A completley But it's only crawl & index 7 Pages on Server B, This server contain more than 500 pages.
** No Problems with conversion tools like catdoc ...
Any idea please

asanad · 09-20-2004, 06:42 AM

I have checked the request and MIME type, it seems normal , does PHPDig have certain settings for the mime type on the server ?

09-11-2004, 06:01 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	1) http://www.phpdig.net/forum/showthread.php?t=799 2) PhpDig can run on a lot of, but not all, remote servers __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

09-11-2004, 07:35 AM	#6
asanad Green Mole Join Date: Sep 2004 Posts: 11	I have the same problem Dear all: I am a new commer to this forum, I really like PHPDig very much. Still I have a problem in one of my servers, whenever I try to crawl that specific Server, PHPDig just crawls the first page then it stops going deeper in the levels. It simply gives me "No temporary table links" or something like that. For example, I have a document called "doc1.txt" its under the folder "test" in "Server A". When I tell PHPDig to index "http://ServerA/test/doc1.txt" ot indexes the file. Now, i need to index all the documents, so I created a file called "index.html", this file caontains a link to all the documents under the folder "test" , for example, "doc1.txt, doc2.txt,doc3.htm,doc3.pdf, doc4.doc". PHPDig for some reason indexes only the "index.html", it does not go and crawl the links within the page. Could you please give me a hint, why this is happening. many thanks for your help and support.

09-11-2004, 07:50 AM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	If PhpDig can index the HTML page but won't follow the links and index the DOC, XLS, and PDF files, then that seems to indicate an external binary problem on the one server. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

09-11-2004, 08:05 AM	#8
asanad Green Mole Join Date: Sep 2004 Posts: 11	I forgot to mention this point Thank you for your quick reponse, I forgot to mention to you that PHPDig is residing on a Unix server, it is able to completely index many other servers sucessfully. This case is only for a specific server. So, all the binaries and conversions work on the PHPDig Unix Server, it is just the issue that one of the destination servers does not crawl properly. Regards

09-11-2004, 08:23 AM	#9
Charter Head Mole Join Date: May 2003 Posts: 2,539	Okay, that changes my response. Some thoughts: check the destination server request and error logs, and also check the MIME type mappings on the destination server. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

09-10-2004, 10:17 PM	#2
dell_10 Green Mole Join Date: Sep 2004 Posts: 13	Whta ??????????? Any one can help me or what ?????

09-11-2004, 06:49 AM	#5
dell_10 Green Mole Join Date: Sep 2004 Posts: 13	Dear ... i don't have any problem with using external binaries my problem is when im trying to index another server (.NEt Server) include folder contain sub folders this sub folder contain files (word, xls, pdf ) and the main folder contain html page with links to this files when im start index this server and put the path to this html page i think phpdig will spider and follow links frm this page and index filse .. i try it on another server (2000 Advance server) and it's work but i can't make it done on this .NET Server the result only indexing of the page without any links and when im try to index any file bu put the full path of the file phpdig can index it

09-12-2004, 03:56 PM	#10
dell_10 Green Mole Join Date: Sep 2004 Posts: 13	Only Crawl Some Files. Hi My problem is: i have 3 servers: Unix(Phpdig Server),2 windows Servers (A,B) PhpDig crawl Server A completley But it's only crawl & index 7 Pages on Server B, This server contain more than 500 pages. ** No Problems with conversion tools like catdoc ... Any idea please

09-20-2004, 06:42 AM	#11
asanad Green Mole Join Date: Sep 2004 Posts: 11	I have checked the request and MIME type, it seems normal , does PHPDig have certain settings for the mime type on the server ?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
spider ignores links	Maarten Wijnen	Troubleshooting	2	03-17-2005 02:23 PM
Spider External links to a depth of 1 (1.8.3)	kenazo	How-to Forum	0	10-20-2004 06:28 AM
specific links not spider	flanders	How-to Forum	1	10-06-2004 11:12 PM
no spider my file links	lolodev	Troubleshooting	21	07-16-2004 06:31 PM
cannot spider own site - others yes	web newsroom	Troubleshooting	1	05-23-2004 03:32 PM