Jeff Clavier's Software Only

Technorati celebrates 1B links (and Wordpress)

Drove up to San Francisco tonight for the Technorati party organized by Niall Kennedy. Dave Sifry announced that over 1B links were now in their index, tracking 8.4M blogs. Scott Beale over at Laughing Squid has a good write-up.
I briefly caught up with Dave and Tantek, talking about the kind of spam that photo sites like us get, typically product placements inundating popular galleries, but also adult sites posting "certain types of photos" that we have to hunt down and remove.

As we are talking about links, check out new  PubSub LinkCounts (and Bob's post). Sounds a bit funny that the top 8 sites are photo storage related.

Jonas was in attendance, and obviously discussions circled around the "Wordpress incident", regarding which he posted a reasonably restrained note (for someone who had coded for 2 days straight). He clarified for us that Matt had removed the incriminated links, and would provide his view/version/explanation of the incident when he comes back from Italy. Jonas also mentioned that he was helping Matt on the setup of a foundation, not a company as it has been widely reported (though this is what I understood as well from this). And he is spending his time answering phone calls and emails from journalists (he got a call every 4 minutes or so when we were chatting). I suggest to also  read his subsequent posts.

My personal take on the whole story is that, as usual, it is exaggerated and Matt, who was praised just a few days ago, is now vilified. And yes, manipulating search engine results by using one's juice is "bad", but then get in line people, everyone is doing it. The only issue I see is one of transparency, if he needed income to cover his expenses, then he should have mentioned in openly and put these links up without hiding them. People contributing to the open source movement have to provide for their families and pay their mortgages at the end of the month, like everyone else. So let's chill out and wait for Matt's explanation (which has just been posted here).

Anyway, thanks to Technorati for organizing this party, and thanks for the T-Shirt (I won in the raffle).

Crap! Spam invades photoblogs as well!

I have written about spam in the past, and I must say, I was distressed when I read that the conversion rate of spam was 4%. C'mon people, we'll never get this to stop if spammers get that high a conversion.

This morning I was presenting Buzznet at a MediaCenter conference, replacing Caterina from Flickr (with a 12 hour notice). One of the themes I was presenting to publishers and media executives is how the platform Buzznet has built can help newspapers to provide a venue for citizens to post pictures of their local community. The poster child of that movement is the work Howard Owens has spearheaded for the Ventury County Star (see Susan and Dan's posts). I therefore did a quick demo of the site, showed my own photoblog, the Venture County Star's, and noticed something wrong with the gallery of one of the bands which is using Buzznet for fan-generated pictures: HIM.

A spammer had inundated the gallery with bitmaps of fake Rolex, that also showed on the Buzznet home page which displays latest uploads... Very un-cool for a demo.

I could only share with the audience the conclusion that Buzznet had gotten into the mainstream since spammers now started spending time bothering our users. This however means that the policy we have been using to let fans freely post to the site, using a self-regulation approach to remove inappropriate content, might have to be changed.

Thanks to Susan and Andrew for the opportunity to participate to this great media gathering. Great discussions and exchanges throughout the day.

A few pictures taken during the day:

JD Lasica just published a full account of the topics covered during the conference.

Why privacy matters: Challenges and Opportunities

Microsoft hosted today a Valley Speakers Series, on "Why privacy matters: Challenges and Opportunities". It was moderated by Moira Gunn, host of NPR’s "Tech Nation" (now podcasted on ITConversations), and featured Scott Shipman, privacy counsel at eBay Inc.; Barbara Lawler, chief privacy officer at HP; Peter Cullen, chief privacy strategist at Microsoft; and Fran Maier, executive director and president of TRUSTe.

I thought that, with such a set of panelists, we would be able to get quite a bit of perspective on what is being developed around both privacy, identity protection and federated identity. Unfortunately, and it might been a case of over-expectations, a lot of time was spent on current issues and current, limited, solutions. Interesting, but I had a sense of a missed opportunity.

I noted the following points:

G8 countries have decided to enforce data retention by ISPs in order to make sure that hacker intrusions, path, data changes,... could be traced through the multiple hops they follow. In effect, only a limited amount of data needs to be collected, but it has to cover all ISP users. This has raised privacy issues, and led privacy advocates to attack the directive. The reply from law enforcement is that such tracking is required in order to maintain the security of Internet users. This is an example of the common dilemna: Anonymity vs. Accountability, ie. how much privacy needs to be protected, vs. how much traceability needs to be put in place to prevent/track abuse.

Schools have to give the home phone number of parents/kids to military recruiters to avoid losing their federal funding. It is possible to opt out from this list but only within 3 weeks of the beginning of the school year. Opt-in was tried in California but led to a low % of acceptance, and was reverted to opt-out.

The decision as to whether opt-in or opt-out is to be used depends on the context, users' expectation and the scope of disclosure. Customers want choice and control as to what information is collected, and how it is used. The current approach seems to be a pre-checked opt-in, but there is no clear definition of what is acceptable and what is not. Ebay has studied that using opt-in or opt-out leads to the same % of action from users: 1%.

Identity theft requires a lot of education of the public in order to 1) be aware of the issue and how it may happen, and 2) the grave consequences ID theft may have. In fact, Citibank - which has partially built its consumer value proposition on covering its customers in case they are victim of ID theft. And recently, TV series like Law&Order have included ID theft in their stories. All these elements lead to the required awareness of everybody, because everybody is at risk.

Ebay is doing a lot to educate its customers as to how you can prevent/detect identity theft. Risk is for someone to get phished, and have their account used to sell bogus items, leveraging their reputation score. Ebay detects unusual patterns: selling different types of items, and from unusual IP addresses - in this case blocking potential transactions.

Unfortunately, education is a challenge, because of the level of sophistication phishers now used to lead their targets to enter critical data. It takes some time to decipher that an email that looks genuine actually contains one or two bogus URLs. Not mentioning malware that might install a keylogger on one's machine through a "drive-by download". Event experts or sophisticated users can be fooled, sometimes because they are not paying enough attention to what they are receiving, downloading or watching.

There is still a paradox: people are aware that their identity needs to be protected, companies spend serious dollars implementing dongles, secure ids, thumb scanners,... and at the same time, a lot of us drop our business card in a bowl to win a free meal...

Some level of regulation is required, but it is not clear how much can be achieved (example of the Can Spam regulation that led to... more spam). A baseline discussion has to take place at the national level about privacy and identity, treating both online and offline. Current state of laws is creating complexity and confusion.

There are a lot of point solutions, which are working reasonably well, like the Ebay toolbar that detects phishing sites. The industry has yet to come up with a set of standards that are broadly adopted, and swiftly implemented. But we are not there yet. In the meantime, this is a useful resource to consult regarding phishing.

So to summarize: the problem is complex, crooks are agile and come up with new ways of trying to steal things from us (phishing and malware are recent evolutions of previous deceptive techniques), and whilst the industry is trying to play catch up, "Users Beware" is in order.

Spam: I knew it was bad, but those numbers are stunning

IT Facts.biz (which is a must use reference for any IT-related statistic) provides some stunning numbers relative to spam volume.

1- Experts say businesses and governments around the world will spend $41 billion this year to defend their computer systems against electronic spam. In China spam account for half of all the e-mail. Spam is expected to account for about 67% of all e-mails worldwide this year, up several-fold from 8% in 2001, according to the United Nations.

2- IDC estimates that spam comprises about 38% of all e-mails sent in 2004 in North America, up 2% YTY. In 2002 spam comprised 24% of North American traffic. 31 billion e-mails are sent daily, according to IDC. The number of spam sent on an average day is 11 bln, compared with 4.5 bln in 2002. In comparison, the number of person-to-person emails currently sent each day and in 2002 is 12 bln and 10.5 bln, respectively.

Also, Future Now points to statistics collected by USA-based Postini, regarding spam and virus activity. Postini, one of several provider of email protection solutions, claims over 4000 customers worldwide. Quick excerpt:

85% of all email traffic Postini filters on behalf of its clients is spam related.

3% of all emails carry a virus

50% of smtp connections are attempts to validate e-mail addresses, in order to add them to spammers lists. You can learn about Directory Harvest Attacks (DHAs) here.

Geographic repartition of virus and spam activity:

An Ultra reliable DNS is key to your web infrastructure (and Amazon seems to agree)

It is with great personal satisfaction that I have spotted that Amazon.com's DNS entry was now pointing to UltraDNS servers (like softtechvc.com, linkedin.com, and 8000 other companies and 26 TLDs). The press release just came out, announcing that Amazon had selected UltraDNS to optimize its Web site infrastructure.

Why is DNS important ? Because, as most of you know, it is like the door knob to an Internet presence. If it does not work, you can't get in your web site (the famous error 404 that you get when a browser can't reach a page) or receive emails. Unfortunately this happens more often than one would like to think: DDOS attacks, operational errors, and simple mistakes lead to unavailability of Internet resources for minutes, hours or even days (in the worst case, it can take up to 48 hours to update one's DNS entry on all Internet name servers).