Submit crawler

Mindrot · 03-22-2004, 03:19 AM

Anyone know if there is a way to create a form for a webmaster to fill out on my phpdig site and then click submit and have him crawl his own site for my search engine?

I would like to add a feature for webmasters to add their site to my engine, and I think even if there is a small box with nothing more than URL:______ and a submit button it would rock.

There could be a message that says Please wait while we spider your site: (this could take a while)

And then it could do the "++++++++" thing for them, and on a locked error report back that their site is next in line for queue.

Is this possible and if so how? Like so many others I am not too keen on PHP but Im not a complete novice either. If this can be done can someone post a howto?

Thank you

Chris

jerrywin5 · 03-30-2004, 03:15 PM

There is a new mod that does something close to this. Rather than automatically indexing user submitted URLs, it waits for admin approval. See this thread for more info.

druken · 04-24-2004, 09:29 AM

mindrot: if your still looking for this, i have modded the spider.php on my site to allow just that. users can spider their own site even shows the links and everything it shows us. if your intrested in it. send me an email to druken@abss.ca

vinyl-junkie · 04-24-2004, 09:53 AM

Quote:

Originally posted by druken
mindrot: if your still looking for this, i have modded the spider.php on my site to allow just that. users can spider their own site even shows the links and everything it shows us. if your intrested in it. send me an email to druken@abss.ca

Why not post a zip file here on the forum and make it available for everyone to download? Just a thought.

snorkpants · 04-26-2004, 03:05 AM

Yes. I'd be very interested in having a look at that too

chrisoverly · 08-26-2004, 07:36 PM

i wouldnt mind having that script either.

03-22-2004, 03:19 AM	#1
Mindrot Awaiting Email Join Date: Mar 2004 Location: I live in the USA Posts: 2	Submit crawler Anyone know if there is a way to create a form for a webmaster to fill out on my phpdig site and then click submit and have him crawl his own site for my search engine? I would like to add a feature for webmasters to add their site to my engine, and I think even if there is a small box with nothing more than URL:______ and a submit button it would rock. There could be a message that says Please wait while we spider your site: (this could take a while) And then it could do the "++++++++" thing for them, and on a locked error report back that their site is next in line for queue. Is this possible and if so how? Like so many others I am not too keen on PHP but Im not a complete novice either. If this can be done can someone post a howto? Thank you Chris

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawler speed improvement (although affects limit)	marco	Mod Submissions	0	03-23-2007 07:14 AM
User submit link	noel	How-to Forum	4	11-19-2005 12:34 PM
Multiple crawler code error	noel	Troubleshooting	3	11-06-2005 11:10 AM
Split the search engine and the web crawler	nobrin	How-to Forum	1	08-15-2005 10:53 AM

03-30-2004, 03:15 PM	#2
jerrywin5 Orange Mole Join Date: Mar 2004 Posts: 48	There is a new mod that does something close to this. Rather than automatically indexing user submitted URLs, it waits for admin approval. See this thread for more info.

04-24-2004, 09:29 AM	#3
druken Green Mole Join Date: Apr 2004 Posts: 2	mindrot: if your still looking for this, i have modded the spider.php on my site to allow just that. users can spider their own site even shows the links and everything it shows us. if your intrested in it. send me an email to druken@abss.ca

04-26-2004, 03:05 AM	#5
snorkpants Green Mole Join Date: Apr 2004 Location: North Wales Posts: 11	Yes. I'd be very interested in having a look at that too

08-26-2004, 07:36 PM	#6
chrisoverly Green Mole Join Date: Aug 2004 Posts: 4	i wouldnt mind having that script either.