... oh Google. How I want to love you, and how, sometimes, you fail so very hard.

This reminds me of that meta-Google search engine that you wrote about in the T-lands universe, though. ;)

And also reminds me of Chibi Jesus. XD Good times!

(Reply) (Thread)

Yeah, Google gets closer and closer to DWIM every day.

In a world with magitechnology, DWIM would be abused just as hard by spammers as Google is today. On the other hand, in a world with magitechnology, every time that spam arrived in a mage's mailbox, someone's server would catch on fire.

(Reply) (Parent) (Thread)

"...a firestorm engulfed the entire southern hemisphere today..."

(Reply) (Parent) (Thread)

Gah, spam - scourge of the universe!

(Reply) (Thread)

That led to some fascinating Google searches. I hope that we can find out more about the matter. What else have you checked to see how Google's behavior has changed - have they made any announcements that seem relevant?


Favorite techniques after searching: changing the code direction and the more effort-intensive but accessible mod_rewrite and PHP/JS trickery recommended by A List Apart. If I were a better programmer, I'd try to implement the latter in Python/Pylons, since I have a vague idea of how it could be done.

(Reply) (Thread)

I haven't seen anything from Google, and I'm not certain how to contact them about this sort of issue (it doesn't seem like the sort of thing to report to their security@ bounce), but now that the story has hit /. hopefully answers might be forthcoming.

Incidentally, as a counterpoint to the links you provided above, I followed a comment in one of those pages to http://jasonpriem.com/2009/05/stop-obfuscating-email/ . I disagree with its main argument, but its point about the poor security of email obfuscation is well taken.

(Reply) (Parent) (Thread)

I sent off a message linking to this here post to my webmaster and sysad. And I have a kitty here asking me to go to bed, and so I shall...

(Reply) (Thread)

It's in Google's interest to be able to index simple document.write-generated HTML, since it's so common. They're definitely not doing full-blown Javascript execution. I munge email addresses on my own website by swapping every pair of letters and the Google snippet doesn't show the results of that. This also suggests that it's not simply a time-bounded execution, since my decoder takes almost no time to run.

Project Honey Pot's suggestion that a harvester would need a full-blown Javascript engine seems a little ridiculous. I bet you can get a lot of the way with a bounded-time non-Turing complete subset of Javascript.

(Reply) (Thread)

Huh, that's actually even stranger: there's no substantial difference in our two routines except for yours calling a subfunction. I mean, ultimately they both boil down to taking a hardcoded string as input, tweaking with the input in various ways and document.write'ing it. And yet, you're right, on your site Google hasn't seemed to pick it up. (Though your address is compromised in a hundred other ways ...)

I suppose this could also be taken as evidence that Google doesn't (currently) interpret pages based on items included by reference, since you call a separate .js file for your functions. That theory would benefit from a more rigorous test.

(Reply) (Parent) (Thread)

I have a page I help maintain, and I also reference an external .js file that does the munging, and it also appears immune from Google's address harvesting.

(Reply) (Parent) (Thread)

This is worth investigating at a stopgap measure. Thank you both.

(Reply) (Parent) (Thread)

I came here from Slashdot. I can also confirm that on one of the sites I host, an email address which is decoded by an external script file is *not* picked up in Google's excerpts.

Very interesting stuff, thank you for this post.

(Reply) (Parent) (Thread)

In modern CMS frameworks, libraries and support utilities are included through a script tag to an external javascript, and page essential javascript is run inline (document.write, onload handlers to trigger the writes in the function, &c.)

They're probably just picking the low-hanging fruit that's easily rendered with a full javascript engine, but minimal time to process (ie: none of the special effects from external libraries, or heavy handed processing). Plenty of benefit for a large chunk of sites.

Emails are going to be like SSNs soon. Just don't hand them out. Make a form that submits an email through the server (it's own issues with spam then, but email address doesn't get out...) or use natural language and make people work it out with their head :P (though texting any one of those pay 1c for a real person to answer a question service is easy and cheap -- tis how spammers get past some nagging signup processes that can't be automated just yet)

(Reply) (Parent) (Thread)

The problem with forms is pretty much the same as the problem with munging: The full interface, along with everything that is needed for mail delivery, is presented to the client, and best practice for up-front (not filtering-based) spam prevention is to rely on the difference between programmed response and pattern matching. Solutions like CAPTCHA or exploiting the difference between human and bot clients are basically the state of the art for robust contact forms.

If we had more contact forms, there would be more advancements in form-breaking, for the same reasons you cite. So I don't think there's anything magical about forms as a solution here -- but if they provide better results, I'm willing to give them a try.

(Reply) (Parent) (Thread)

You might want to try retrieving the email address with an XHR next - I doubt google will permit such a request, and so they will be unable to decode it.

(Reply) (Thread)

You could use robots.txt to block Google's access to your external javascript file or XHR request URL.

(Reply) (Parent) (Thread)

That seems like the best immediate measure to take. I'll also try putting an inline filter-free address on a robots.txt'd page in an effort to see if spammers can crack it without Goog backing them up.

(Reply) (Parent) (Thread)

You might want to factor some very large numbers in your munging code there...

(Reply) (Thread)

While that does scratch the itch for spam vindictiveness, that seems to me to be doomed to failure. I do want the address to display to human visitors, and I would suspect that any delay large enough to deter automated processes would also serve as a source of annoyance or "This must be broken" for my meatspace visitors.

(Reply) (Parent) (Thread)

The only way the address is being displayed is if you search for the address itself. And if you already know the address it doesn't matter if it's plain text. Any other summary shown doesn't contain the address. I suspect something managed to harvest it, and you're just going after the easy target. Show me a google search for something that isn't the e-mail address that shows the address in plain text.

(Reply) (Thread)

If they are indeed running the Javascript then I would just change the code to only execute if it's run in a full browser (which, I assume, they are NOT doing.) Checking navigator.userAgent should make this relatively easy.

(Reply) (Parent) (Thread)

navigator.userAgent can be trivially faked.

(Reply) (Parent) (Thread)

Yes, but why would Google completely fake a user agent? They've always been good about identifying themselves as a bot somewhere in the user agent string.

(Reply) (Parent) (Thread)

Oh, good point.

I think what I was thinking at the time of my reply was that, if Google has started doing this, other companies (and less savoury individuals) are probably not far behind. A better long-term solution would be to focus on a method of munging still not broken, rather than try to exclude special cases based on a voluntary field like that.

(Reply) (Parent) (Thread)

> The only way the address is being displayed is if you search for the address itself.

Not entirely true.

While the address does not display if you try to search for it indirectly -- "chibi jesus" "please send here *" actually returns the noscript alternate text for the address, which blows my mind completely -- a spammer can still get the address trivially without knowing it to begin with. Google has powerful wildcarding.

Voila: "* tomorrowlands org" site:tomorrowlands.org chibi

All you need to do is search for things shaped like e-mail addresses, optionally on the same domain that you're searching for addresses on. I added the "chibi" in to push the CJ page to the top of the results list but I'm sure it's in the complete listing as well.

(Reply) (Parent) (Thread)

Google tries to block bots from access. The obvious end results of this is botnets to query Google- not to intentionally DDoS it, but for information harvesting, since Google's the best tool to do the heavy lifting in the background, apparently.

A brief experiment doesn't show Bing, Ask, Yahoo, or AltaVista doing this.

(Full disclosure, for those who don't know- hi, /. p33pz!- I'm a full-time employee of Microsoft. This comment doesn't reflect any position or information official to Microsoft, and no representation I make about a Microsoft product herein is official or checked for accuracy. I do not work on the Bing team, have no non-public information about Bing, and wouldn't disclose it if I did.)

(Reply) (Thread)

Posted this in IM to Bax this morning, but maybe someone else will find it interesting:


<html>
    <head>
        <style type="text/css">
            #mailme             { list-style-type: none; float: left; }
            #mailme li          { display: inline; float: right; }
        </style>
    </head>
    <body>
        <ul id="mailme">
            <li>m</li>
            <li>o</li>
            <li>c</li>
            <li>.</li>
            <li>n</li>
            <li>i</li>
            <li>a</li>
            <li>m</li>
            <li>o</li>
            <li>d</li>
            <li>@</li>
            <li>l</li>
            <li>i</li>
            <li>a</li>
            <li>m</li>
            <li>e</li>
        </ul>
    </body>
</html>


It's not easy to hilite, and it's a mess if you copy it to the clipboard, but it's an interesting non-JavaScript way to post an address that would require some computational feats for spambots to interpret correctly.

(Reply) (Thread)

This is much simpler and works just as well:

<div style="font-family: 'courier new', courier, monospace; line-height: 0px;">
z r y i h g a l c m<br>
&nbsp;c a f s @ m i . o </div>


Example:

z r y i h g a l c m
 c a f s @ m i . o

(Reply) (Parent) (Thread)

Nice, although I specifically avoided any methods which used a 0 line-height or font-size or similar, because it's too easy to filter that out. Google for example has been watching for that kind of stuff due to SEO word salad abuse.

(Reply) (Parent) (Thread)

Even better, try this replacing the '.' with a &+#+46; and the @ with a &+#+64; (remove the plus signs)

:)
Bud

(Reply) (Parent) (Thread)

Hi, I work at Google and help out in our Webmaster Help forums. In general, we work hard to find, index and make available content that we find on the web -- this includes innovations and experiments in recognizing information which is rendered via Flash or JavaScript, in PDF files and possibly in other documents that contain indexable content. If you wish to prevent the crawling and indexing of content, it would be best to use methods that explicitly disallow access to that content (for example through a robots.txt file or a robots meta tag on the page itself).

(Reply) (Thread)

Seems like you've been relying on an unreliable way to protect your email address. It's certainly foreseeable that some company might run the javascript. Probably the biggest reason for doing this is not to intentionally expose email addresses, but instead that since many pages have now turned to AJAX for basic text display, Google must process the javascript in the AJAX to correctly index and allow searching on the results.

(Reply) (Thread)