PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 02-27-2008, 06:30 AM   #1
Alex-FBTech
Green Mole
 
Join Date: Feb 2008
Posts: 1
Slow indexing

Hello!

I just want Phpdig to index around 40 000 pages, and I saw that other people made this without problems. I cannot give the root of my website in the admin panel because it's only a search engine and without a list of links, it's not possible to crawl the entire site.

At the beginning I made an html file with all the links of my site, and I launched the spider on it with a 1 depth level but it seemed to had some problems to catch too much links in a page.

I tried to divide my links in 10 files of around 4000 links.
Now, Phpdig has no problems to just catch quickly the links in the temporary table, but it takes around 30/40 seconds per page when it begin to index! (~ 10 days to index the site)

I tried with the web interface, with shell commands and I had the same results. I set always with 1 depth level and 0 links per.

How can I do to speed up the spider?

Last edited by Alex-FBTech; 02-27-2008 at 06:34 AM.
Alex-FBTech is offline   Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Indexing are very slow, how can i speed up? ossi01 Troubleshooting 2 05-06-2005 04:26 AM
Indexing slow.... no, _really_ slow bluntman Troubleshooting 1 09-24-2004 02:23 PM
helpme indexing slow xorex Troubleshooting 3 07-27-2004 07:00 AM
Indexing very slow mdkn Troubleshooting 2 03-21-2004 02:01 PM
Very Slow Indexing airplay Troubleshooting 2 03-09-2004 03:20 PM


All times are GMT -8. The time now is 01:33 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.