index intershop-sites?

comko · 03-23-2004, 11:28 AM

Hi folk,

i'm successfull installed phpdig on my linux server (LAMP) - great work, guys!!! thanks for it.
no probs with the most sites, i will search for.
but, only one won't: www.junfermann.de is the site, generated with INTERSHOP, and that's my problem:
all pages where contained, are WITHOUT any "pages". it's true, e.g. one page is : http://www.junfermann.de/cgi-bin/junfermann.storefront/DE/Catalog/1011/
Q: how can i index this site?

thanks a lot for help:
Ingo

Charter · 03-24-2004, 04:15 PM

Hi. First, download the ZIP file in this thread and replace robot_functions.php with the one in the ZIP file.

Next, in the new robot_functions.php file, search for "<frame" (without the quotes) and on this line add in a [[:blank:]]* so that:

PHP Code:


			
while (blah blah *content=['\"][0-9]+;url blah blah) {

becomes the following:

PHP Code:


			
while (blah blah *content=['\"][0-9]+;[[:blank:]]*url blah blah) {

Last, the server for that site does not return a content-type for certain pages. You would need to force the content-type for these pages so, in the new robot_functions.php file, search for:

PHP Code:


			
if (!eregi('[a-z0-9]+',$answer)) {

and right before that line add:

PHP Code:


			
// THIS CODE IS ONLY FOR WHEN CONTENT-TYPE IS NOT RETURNED

// IT IS NOT FOR GENERAL INCLUSION IN THE CORE PHPDIG CODE

elseif (!eregi("Content-Type: *([a-z]+)/([a-z.-]+)",$answer,$regs)) {

   $status = 'HTML'; // no content-type so set to html

}

Remember to remove any "word" wrapping in the above code.

comko · 03-25-2004, 12:33 AM

Yikes, it works!!
thanks for geat work!

Ingo

malieut · 03-30-2004, 07:09 AM

hello Charter,
I followed your suggestion,but got this screen as the following when i click the dig this button.

------------------------------
Spidering in progress...

------------------------------
Nothing happened else, the spidering is end.

Charter · 03-30-2004, 09:22 AM

Hi. Recheck the mods and make sure to refresh the admin/index.php page before indexing. Only apply the last bit of code if a content type is not returned, which is generally not the case.

03-23-2004, 11:28 AM	#1
comko Green Mole Join Date: Mar 2004 Posts: 2	index intershop-sites? Hi folk, i'm successfull installed phpdig on my linux server (LAMP) - great work, guys!!! thanks for it. no probs with the most sites, i will search for. but, only one won't: www.junfermann.de is the site, generated with INTERSHOP, and that's my problem: all pages where contained, are WITHOUT any "pages". it's true, e.g. one page is : http://www.junfermann.de/cgi-bin/junfermann.storefront/DE/Catalog/1011/ Q: how can i index this site? thanks a lot for help: Ingo __________________ -- my name is dump, core dump

03-24-2004, 04:15 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. First, download the ZIP file in this thread and replace robot_functions.php with the one in the ZIP file. Next, in the new robot_functions.php file, search for "<frame" (without the quotes) and on this line add in a [[:blank:]]* so that: PHP Code: `while (blah blah content=['\"][0-9]+;url blah blah) {` becomes the following: PHP Code: `while (blah blah content=['\"][0-9]+;[[:blank:]]url blah blah) {` Last, the server for that site does not return a content-type for certain pages. You would need to force the content-type for these pages so, in the new robot_functions.php file, search for: PHP Code: `if (!eregi('[a-z0-9]+',$answer)) {` and right before that line add: PHP Code: `// THIS CODE IS ONLY FOR WHEN CONTENT-TYPE IS NOT RETURNED // IT IS NOT FOR GENERAL INCLUSION IN THE CORE PHPDIG CODE elseif (!eregi("Content-Type: ([a-z]+)/([a-z.-]+)",$answer,$regs)) { $status = 'HTML'; // no content-type so set to html }` Remember to remove any "word" wrapping in the above code. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-25-2004, 12:33 AM	#3
comko Green Mole Join Date: Mar 2004 Posts: 2	Yikes, it works!! thanks for geat work! Ingo __________________ -- my name is dump, core dump

03-30-2004, 09:22 AM	#5
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Recheck the mods and make sure to refresh the admin/index.php page before indexing. Only apply the last bit of code if a content type is not returned, which is generally not the case. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Trying to index some dynamic sites	guillaume	Troubleshooting	2	08-08-2007 06:40 AM
PHPDig won't index most sites and only go down one level on all	confusion	Troubleshooting	1	10-14-2005 11:32 AM
I just want to index main sites	afesh	How-to Forum	1	08-26-2005 09:45 PM
"I don't want to index your sites!!!" - said PHPDig	#ASH	How-to Forum	1	04-06-2005 02:57 PM
Some sites won't index	jalerta	Troubleshooting	23	05-19-2004 01:35 PM

03-30-2004, 07:09 AM	#4
malieut Green Mole Join Date: Mar 2004 Posts: 1	hello Charter, I followed your suggestion,but got this screen as the following when i click the dig this button. ------------------------------ Spidering in progress... ------------------------------ Nothing happened else, the spidering is end.