Search Engines

Jun 6 18:41 UTC 2000

Currently Grex is configured in such a way to discourage search engines
from indexing our conferences.  It seems to be working - I've turned up
any Grex Backtalk pages on any search engine.

Theoretically this could be changed.  If I worked at it, I could
probably convince search engines like Google and Altavista to index the
Grex conferences.  It is likely that I could convince them to index only
certain conferences - I could possibly even get the control down to the
level of indexing only selected items.

What I'm wondering would be if this is a good idea?

If we allowed indexing, people would feel that these forums are "less
private" and would be more reluctant to post personal information.

But if we want to attract lots of people into our conferencing system,
having the search engines index our conferences would be a great way to
do it - search for some subject, find Grexers discussing it, join in.

I don't know where I stand on this.  I think making some conferences
searchable (like "coop") but not others might be worth discussing.

response 2 of 123:

Jun 6 22:54 UTC 2000

I like the idea of making conferences searchable.  Not only could it
attract new users, but it would be useful to established Grex users
such as myself in searching for old stuff.

response 4 of 123:

Jun 7 02:19 UTC 2000

I like this idea, especially when we have discussions regarding political
viewpoints or childbirth.  Some of this information is wonderfully
informative.

response 6 of 123:

Jun 7 02:34 UTC 2000

Ick.  

I think I may be against this.  While I supported the anonymous reading from
the web, I'm not sure I would support indexing anything unless posters knew
from the start that their response might become searchable on the web.  In
fact, part of my willingness to post on Grex has been premised on the idea
that no one could find my thoughts unless they found Grex first and also were
interested enough to join a conference.  

Perhaps only indexing future Agoras? Or newly-started conferences?  I'd be
willing to restart Small Business so that it could be indexed, but only if
there was a line on the entry screen that said that was being done.  

response 8 of 123:

Jun 7 03:52 UTC 2000

I'm with the last three responders.  I don't think this is a good thing to
do to people who entered stuff on the expectation that search engines can't
read it.

response 10 of 123:

Jun 7 12:54 UTC 2000

There is a reason I use a pseudo in some items...  That way nobody can
search by name or attribute it to me.

response 12 of 123:

Jun 7 16:14 UTC 2000

I'd REALLY be upset if the search could be by name or login.  I've got enough
stalking problems without that added breech.  

response 14 of 123:

Jun 7 17:47 UTC 2000

More likely, the indexing would be made by a stupid search engine. The
clever ones will heed instructions found in a robots.txt file, or a
noindex command embedded on a page.

response 16 of 123:

Jun 7 18:27 UTC 2000

Anyway, my point was that that there has never been a guarantee
that Grex conferences won't get indexed, just a reasonable
supposition that the major search engines won't do so directly
because they are known to follow certain rules.  I think it could
happen indirectly, though.  Someone could create a "rogue mirror
site" of Grex, modified in such a way the things that prevent
indexing by major search engines are not present.  Then the
mirror site would get indexed by the major search engines.
I realize that this might not be legal to do, but that does not
mean that it wouldn't happen.

So in discussing this issue, people should keep in mind that
the most Grex can do is influence the likelihood that people's
responses here will or will not become web-searchable; Grex
can't guarantee that they won't.

response 18 of 123:

Jun 7 19:16 UTC 2000

Yes, I'm aware that such indexing may happen.  I just don't think we should
do anything to encourage it.

response 20 of 123:

Jun 7 21:35 UTC 2000

The major search engines are well-behaved and obey our gentle requests
not to index the conferences.

But not all robots are that polite.  I've never seen any search engine's
spider go through the Grex conferences, but I have seem email address
collectors do so.  It does tend to be somewhat hard on the internet
connection when this happens, though the more legitimate spiders tend to
me more careful about this too.

I don't have a strong opinion either way.  I guess I agree that it would
be appropriate to open only new material to indexing, not old material.

Another possiblity would be to allow "item list" pages (which show the
titles of all the items in a conference) to be indexed in some
conference, but not the actual items themselves.

response 22 of 123:

Jun 8 02:58 UTC 2000

resp:5 -- Jan suggests that it would be possible to limit which items a
search engine would be encouraged to inedex.  Presumably that means that we
could continue discouraging them from indexing items entered before a certain
date, in order to protect people who entered stuff before the policy change.

That said, I still don't think that encouraging search engines would be a good
idea.

response 24 of 123:

Jun 8 14:44 UTC 2000

Re #21 - both my full name and cf pseudo are true pseudos.  If people know
my login, oh well.  They'll still see that "gypsi" belongs to "Groovy
Birdy".  =)  For a LONG time, I was using a completely different last name.
The only reason I changed it was because I know a lot of grexers in real
life now, and it gets confusing when they try to look me up or whatnot.