|
02-27-2008, 05:30 AM | #1 |
Green Mole
Join Date: Feb 2008
Posts: 1
|
Slow indexing
Hello!
I just want Phpdig to index around 40 000 pages, and I saw that other people made this without problems. I cannot give the root of my website in the admin panel because it's only a search engine and without a list of links, it's not possible to crawl the entire site. At the beginning I made an html file with all the links of my site, and I launched the spider on it with a 1 depth level but it seemed to had some problems to catch too much links in a page. I tried to divide my links in 10 files of around 4000 links. Now, Phpdig has no problems to just catch quickly the links in the temporary table, but it takes around 30/40 seconds per page when it begin to index! (~ 10 days to index the site) I tried with the web interface, with shell commands and I had the same results. I set always with 1 depth level and 0 links per. How can I do to speed up the spider? Last edited by Alex-FBTech; 02-27-2008 at 05:34 AM. |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Indexing are very slow, how can i speed up? | ossi01 | Troubleshooting | 2 | 05-06-2005 03:26 AM |
Indexing slow.... no, _really_ slow | bluntman | Troubleshooting | 1 | 09-24-2004 01:23 PM |
helpme indexing slow | xorex | Troubleshooting | 3 | 07-27-2004 06:00 AM |
Indexing very slow | mdkn | Troubleshooting | 2 | 03-21-2004 01:01 PM |
Very Slow Indexing | airplay | Troubleshooting | 2 | 03-09-2004 02:20 PM |