Astachoth@xxxxxx wrote:
I just noticed that the Lycos search engine is "stuck in a loop". I
like having my list pages indexed in the search engines, but Lycos
is following the next links at the bottom of the list pages in my
rings. It now has over a hundred links to my same list pages,
because of the "offset=" extensions and different site ids, which
are in other members codes.
Is there a way to prevent this from happening without adding
"robots noindex" meta tags to the list pages?
It just struck me... What do you think about:
<meta name="robots" content="index,nofollow">
That would tell the spiders to index the page, but ignore links to
other pages. It wouldn't remove URLs that have already been indexed,
of course, but might prevent future redundant indexing.
/ Gunnar
at the homepage.