Astachoth@xxxxxx wrote:
I just noticed that the Lycos search engine is "stuck in a loop". I
like having my list pages indexed in the search engines, but Lycos
is following the next links at the bottom of the list pages in my
rings. It now has over a hundred links to my same list pages,
because of the "offset=" extensions and different site ids, which
are in other members codes.
Is there a way to prevent this from happening without adding
"robots noindex" meta tags to the list pages?
Hmm.. This was a new problem. Previous search engine related
discussions have been about the reverse situation, i.e. that some
search engines (aka Google) don't index complicated URLs at all.
I have no immediate solution that obviously would be better. Maybe
this is something we just have to live with. After all, if Lycos works
the way you describe, the Lycos users are probably used to see a lot
of redundant URLs on the search pages...
I'd appreciate comments from other list members on this one.
Also, wouldn't it be a good idea to cause a "404 not found" page to
appear when someone tries to access a deleted WebRing within a
WebRing system. Currently, if someone previously ran a WebRing,
(which is indexed in a search engine), on a particular system, it
will remain in the search engine even after deletion, because of
the error page. The search engine doesn't realise that the WebRing
has been deleted, because a valid HTML page still displays.
Eventually, the search engines will become filled with links to
deleted rings resulting in error pages.
I see your point, and I suppose it's possible to have Ringlink send
something else but "200 OK" even if a valid page is displayed. I'll
look into it.
I have noticed that webring.com has the same problem. Deleted rings
remain in the search listings.
A bigger problem with WebRing.com is that their definition of "delete"
differs from what most people mean with that term. When a ringmaster
"deletes" a webring at that system, the ring keeps existing, even if
the old ringmaster looses control over it...
/ Gunnar