Search times and speed

[Dave A]

Heaps of people are presently chatting about search times.
If your version of PHPDIG is running on a normal web host then it is sharing resources with other users include processor time and db access.
I am presently changing over to a dedicted server and the early tests show that search times are a whole heap quicker plus it gives me more scope for creating a larger index without speed issues.
The trouble can be that web hosting companies, get titchy when you start to consume a heap of bandwidth and start to hog the processor usage.

[jmitchell]

I'm running on a dedicated server myself (and love the control that I have) but I'm still getting slow response times.

It could be how many pages I have indexed (60,000), but if so, I'm not sure what I can do to fix that.

any ideas anyone?

JMitchell

[Dave A]

Yes I also have a heap of pages indexed, when it was on a hosted server and not a standalone, a search for say " Camp sites Auckland " could take up to 44 seconds, in some cases due to the processor usage and further pushed by Multiple searches at the same time.
Since then I found the best way around it was to set up the mysql database in the ram in the host (Cached) and I have set up a small Php code addition that stores previous searches for five days. So that searches are checked via the cache then run across the database.
If you try two searches that are identicle within twenty seconds the old search should remain in your hosts memory and so the time will be a heap shorter for the second search.
Since I am not the brightest bulb in the light house, it may well be worth someone having a look into storing the search results in other ways to see if we can all work out the problem of slow search speeds.
It could be a way of turning phpdig into something really special.
Everyone who uses Phpdig should be looking at ways of combining our knowledge and experiences to help each other, Charter and Vinyl Junky are brilliant and the support they offer, is really neat and we must bear in mind that support isn't guaranteed it's offered freely and must take a heap of time for them.
One thing that I had considered that might be useful would be a simple drop down chart/menu made of boxes with problems that answer yes or no too leading to the possible cure. It may save people time in searching the forums but it's just an idea.

I hope this helps you out, I have popped an email over to your email address found on your web site..

All the very best
Dave Andrews

[Andypale]

Wow! 60,000 pages in a website? Do you run amazon.com?

[jmitchell]

no, just trying to get a small search engine going.

Charter, have you thought of programming a search engine that can take several million results? it would probably have to be in something like C or C++.

JMitchell

[Dave A]

Well I have just upgraded my host to twinned 360gb drives (Mirrored) a gig of RAM and it's running twin 3.2 Gig processors and on a multiple word search (Five words searched) it manages to whiz through an index of seven million in less than a second which is qiute speed increase so it may be something to do with the make and access times of the drives and the memory which is in the host.
The system runs red hat so I am not sure if the operating system may have some influence on it.
But if we can compare notes we may be able to find out what makes some systems slow down.
All the best
Dave Andrews

[bloodjelly]

Have you guys seen this thread: http://www.phpdig.net/forum/showthread.php?t=978 ? It speeds up search queries quite a bit. With 1.8.7 though it doesn't seem to recognize the different types of queries, except for start and exact. Basically it gives the searching job to the MySQL DB, which can perform the functions faster than PHP. I'd be curious how much of a difference it would make for you, JM, since you have so many pages indexed