mutilple spider via fork()

[ragnoroc]

Well I read that one message about using the perl wrapper script to seperate spider tasks:
http://www.phpdig.net/showthread.php?s=&threadid=934


Then the one about the php wrapper:
http://www.phpdig.net/showthread.php?threadid=662

What I am thinking of trying is using pcntl_fork() to split the spidering and indexing processes.

Another thing I am going to take a look at is running the phpdig code through Zend Performance Studio and running it on a server with Zend optimizer to see how much more speed I can get out of it.

I really like phpdig and it suits all my needs. If I can get it to run faster all the better.

Anyone have any comments or experience info on trying any of the above mentioned.

[bloodjelly]

No experience with that forking function (bad joke there) but good luck! I have a feeling that leftover gunk in the temporary spider table might confuse multiple spiders, but we'll see I guess.