ImperialViolet

I don't usually comment on anything Google related here since I started, but I'll make an exception this time:

Once upon a time there was a website, a kind of proto wiki, and, at the bottom of each page, was a link titled "Delete this page". The Google crawler did its crawl and that was the end of that website.

(The reason it came to light is because the webmaster of that site emailed Google "asking for his website back" (or words to that effect) and I believe that we dug them out of the crawl data for him. But this is besides the point.)

The point is that no one does that sort of thing any more. GET links on a website must not be mutable or you can be sure that one of a number of crawlers will mutate it.

But people didn't really learn, they just retreated behind login pages and such and made all the same mistakes. Now 37signals, no less, a group which carried my respect (until today) is getting very upset that crawlers are getting behind the login pages all over again.

I won't even comment that they seem to think that GWA has been pulled because of their blog post, nor about the commentators who are making a big deal of the MUST NOT vs SHOULD NOT wording in the spec. Here's the end result

It's a bad idea - anywhere.

(I'm not on the GWA team. This is not from them and involves no non-public knowledge of that project)