Human.io lends itself to small, simple tasks: Vote on an item, take a picture of a storefront, etc. It allows you to script with humans as easily as you would script with software. It also offers easy access to the sensors on the phone: GPS, camera, and so on.

The architecture borrows from Twilio: The developer sends UI widgets to the client, receives HTTP callbacks when their users perform actions, and then responds with more UI widgets. We can also use a long poll so you can easily run apps from behind a firewall, on your laptop, or otherwise without opening a port on the firewall.

We also built Photo Scavenger Hunt on the platform as a fun little demo. I'm going to give away an ELPH camera to whoever's winning by tomorrow.

Seven on Seven: Trailer

Seven on Seven: Monica Narula & Joshua Schachter

The data was gathered using the Mechanical Turk. I didn't really have enough time to talk about data, so here is actual dataset.

I've posted presentation as well.

There are a variety of reasons for this, but I'm generally pretty sensitive to my tools, and I haven't been thrilled with either what I am currently using or what I might use in the future. Do I want to use Wordpress on a virtual machine at some hosting provider? Do I want to write something custom on AppEngine? Or one of a dozen dozen other choices? It makes me want to lie down.

It occurs to me that the tools we have available each do a large variety of things, and that there's no good reason for these functions to be bound together into one application. For example, Maciej's recent article on why not to have a public of Wordpress (and more details) shows that serving the website and editing it can be very separate pieces. The original ancient Blogger software also used to push a copy up to your site via FTP.

There are a number of separable pieces in the system:

Authoring - the actual role of editing a blog entry is usually just a big text field, but several tools use FCKEditor and other nice javascript-based editors. Google Documents could fill this role too. Current weblog APIs allow this part to be decoupled, for the most part, usually for desktop or mobile clients.

Storage - a simple database would suffice. Not much metadata is required, nor is complicated indexing. Amazon S3 and Google Docs both fit the bill here.

Templating - The system that turns the raw blog posts from the storage engine into the pretty HTML version. There's nothing that really fits this bill in the current systems

Hosting - there's no need for the system that runs the blog authoring and storage software to serve the raw HTML pages. Amazon S3 would also suffice here, IF it dealt with directory index pages in a useful manner. (Currently, a url ending in a slash cannot map to a document on S3, so far as I know.) RSS/Atom feeds would also be served from the same system.

Feeds - as standards change over time, it would be nice to be able add the appropriate functionality. Feedburner already does some of this.

Comments - there are several solutions for hosting comments outside the blog applications: Disqus, Intense Debate, JSKit, and so on. I think moderation outsourcing and aggregating comment behavior will be increasingly necessary due to spam issues. Nor do I think that publishers should own comments, but that's a matter for another article.

I wonder if there is a way to define loose interfaces between these systems so that they could both work together but also not set APIs in concrete solid enough to stop innovation. Because the various pieces of the systems currently are all tightly bound together, it is very hard for the parts to move forward separately. For example, I've wanted to be able to specifically reply to comments in place in a visually differentiated way as the publisher, rather than just as another commenter. But this feature hasn't emerged, and if I hacked it into one platform via plugins, I'd be stuck with it forever.

It would also be nice if these systems could work together without all being client-side embeddable widgets. This usually slows down page loads tremendously.

What else have I left out?

Aside from the raw utility of allowing URLs to fit within a Twitter message, newer services add several interesting bits of functionality. The most important of these is that let the linker turn any link into THEIR link, and view metrics on how far it's spread and how many clicks it's gotten. Showing a user how popular his actions are is inevitably addictive. Shorteners are relatively easy and lightweight to set up. Adding a simple interstitial before the redirect provides an obvious way to monetize. And maybe someday all the link data will be worth something.

So there are clear benefits for both the service (low cost of entry, potentially easy profit) and the linker (the quick rush of popularity). But URL shorteners are bad for the rest of us.

The worst problem is that shortening services add another layer of indirection to an already creaky system. A regular hyperlink implicates a browser, its DNS resolver, the publisher's DNS server, and the publisher's website. With a shortening service, you're adding something that acts like a third DNS resolver, except one that is assembled out of unvetted PHP and MySQL, without the benevolent oversight of luminaries like Dan Kaminsky and St. Postel.

There are three other parties in the ecosystem of a link: the publisher (the site the link points to), the transit (places where that shortened link is used, such as Twitter or Typepad), and the clicker (the person who ultimately follows the shortened links). Each is harmed to some extent by URL shortening.

The transit's main problem with these systems is that a link that used to be transparent is now opaque and requires a lookup operation. From my past experience with Delicious, I know that a huge proportion of shortened links are just a disguise for spam, so examining the expanded URL is a necessary step. The transit has to hit every shortened link to get at the underlying link and hope that it doesn't get throttled. It also has to log and store every redirect it ever sees.

The publisher's problems are milder. It's possible that the redirection steps steals search juice — I don't know how search engines handle these kinds of redirects. It certainly makes it harder to track down links to the published site if the publisher ever needs to reach their authors. And the publisher may lose information about the source of its traffic.

But the biggest burden falls on the clicker, the person who follows the links. The extra layer of indirection slows down browsing with additional DNS lookups and server hits. A new and potentially unreliable middleman now sits between the link and its destination. And the long-term archivability of the hyperlink now depends on the health of a third party. The shortener may decide a link is a Terms Of Service violation and delete it. If the shortener accidentally erases a database, forgets to renew its domain, or just disappears, the link will break. If a top-level domain changes its policy on commercial use, the link will break. If the shortener gets hacked, every link becomes a potential phishing attack.

There are usability issues as well. The clicker can't even tell by hovering where a link will take them, which is bad form. Some sites offer link previews, but there's no way to make a preview preference stick globally across the many shortening services. And just like ad networks, link shortening services could track a user's behavior across many domains. That makes the paranoid among us uncomfortable. We hope the shortener never decides to add interstitials or otherwise "monetize" the link with ads, but we have no guarantee.

For these reasons, I feel that shorteners are bad for the ecosystem as a whole. But what can be done to improve the situation?

One important conclusion is that services providing transit (or at least require a shortening service) should at least log all redirects, in case the shortening services disappear. If the data is as important as everyone seems to think, they should own it. And websites that generate very long URLs, such as map sites, could provide their own shortening services. Or, better yet, take steps to keep the URLs from growing monstrous in the first place.

You could guarantee that the shortened link is the one that was originally shortened by using a cryptographic hash. But this causes URLs that aren't as short as is possible.

A variety of greasemonkey scripts resolve shortened URLs and replace them inline.

Finally, shortening services could provide archives of their entire database - but this raises all sorts of privacy concerns that I hesitate to even dig into.

The most likely, of course, is that we don't do anything and that the great linkrot apocalypse causes all of modern culture to dissapear in a puff of smoke. Hopefully.

With thanks to Maciej Ceglowski

Updates

• June 15th, 2009: cli.gs, the "4th most popular" shortener, gets hacked, redirecting a huge number of sites to a new location. 93% of hacked urls can be restored from backup
• August 9th, 2009: tr.im throws in the towel after being able to figure out how to monetize the site. There are zero interested buyers. The site will redirect links until "at least" the beginning of 2010, but no future is guaranteed.

1. Make sure you have the appropriate codecs installed. I generally use the Perian codec package. I additionally find that some FLVs require QTPro to be installed; it's not very expensive.
2. Download the video somehow. Some sites, like Google Video, let you download a copy. Others, like YouTube, do not allow this. However, most embedded flash video can be grabbed via the technique in the bottom video in the demo videos at Perian.
3. Open the video in QuickTime. The video is now happily outside the browser.
4. Go to Window → Show A/V Controls; change the playback speed in the relevant window. I find that 2.0x generally works pretty well; the video will be faster and the audio is a little clipped but nicely de-chipmunked.
5. Enjoy your new lecture! The glacial discussion now arrives at a rapid-fire pace. You'll be too busy trying to keep up to play Desktop Tower Defense, and you'll be done in a half hour.

]]> http://joshua.schachter.org/2008/11/overclocking-lecture.html http://joshua.schachter.org/2008/11/overclocking-lecture.html complaining Sun, 02 Nov 2008 00:14:21 -0800 Ever since seeing a presentation by Dolores Labs about Amazon's Mechanical Turk, I've been itching for an excuse to play with the system.

I recently saw a thread that highlights the distinction between expected value and utility. Would you take a more likely but lower payoff instead of a less likely but higher payoff? Similarly, the St. Petersburg Paradox takes the problem to its logical extreme. By constructing a game that has a series of increasingly rare payoffs of increasingly larger size, a game with infinite expected value is created.

So I constructed 21 versions of the questions, varying the size of the dollars as well as the rate of payoff for the second outcome.

For one cent apiece, I sent the questions to be answered by one hundred people each, and collated the results. 2100 questions, three hours, and thirty dollars later, I have my results.

Clearly, people (or at least these Turks) begin to cross over at larger values, reaching equilibrium at around $1,000.

While this isn't the most groundbreaking work, it is nice to be able to generate an experiment and gather the results in the course of an evening and then have the results be so pleasing.

The Mechanical Turk is presented as a way to solve problems that are easily explained to people but difficult to implement for computers, frequently described as "artificial artificial intelligence." However, I think some of the most intriguing uses yet will be to explore the edges of our own uniquely human behavior and self-understanding.

A publish/subscribe architecture is natural to other problem domains such as instant messaging and financial data systems (Tibco, Reuters, and so on).

Similarly, Brad Fitzpatrick implemented something similar as a never-ending Atom feed a few years ago for Livejournal (sans XMPP, which wasn't as conceptually prevalent then.)

One important point in the presentation is that, for example, a single application would poll Flickr approximately three million times in a day to fetch only several thousand updates. At Delicious we saw a similar level of polling activity, made somewhat worse by speculative querying (hitting the URL information pages to see if there was any data for arbitrary URLs, which was generally unlikely.)

One solution that ocurred to me at the time was to build a simple callback system over HTTP. This would fall comfortably between full polling and full persistent publish/subscribe. The clever acronym even writes itself: PIMP Is Mostly Push, although maybe PRSS (Push RSS) would be slightly more polite.

Simply described, instead of polling frequently, a client would send a normal HTTP request with the resource to be subscribed to and an endpoint to deliver updates to: http://your.app/subscribe?resource=/some/user&callback=http://my.app/endpoint

Presumably the endpoint would then receive RSS item fragments when and only when that resource updated. For security, the exchange should include some kind of token, borowing from the appropriate protocols. The subscription would lapse after, say, 24 hours, or that could be passed in as a parameter.

In some ways this is slightly more elegant than the XMPP solution as neither side has to maintain a dedicated long-running process. A simple server-side implementation would justfetch items from a work queue and send out HTTP messages. A simple implementation on the client side would be a plain old web page that could accept and process a POST request. There are a number of people on inexpensive service providers who have at best web scripting hosting and not much else. The case where Delicious/Twitter/Flickr pushes my own items (and not much else) up to my blog is an important one. Additionally, there would not need to be any persistent TCP connections, which is probably more efficient in server resources (but less efficient in network resources; for billions of messages the TCP overhead becomes significant).

Of course, callbacks are totally infeasible for a variety of other uses, especially for mobile or desktop applications (which are likely to be firewalled). ]]> http://joshua.schachter.org/2008/07/beyond-rest.html http://joshua.schachter.org/2008/07/beyond-rest.html ideas Thu, 24 Jul 2008 10:41:59 -0800 Rhizome 2008 Benefit in New York City later this week. There are still tickets available, so you have no excuses for not attending.

]]> http://joshua.schachter.org/2008/05/rhizome.html http://joshua.schachter.org/2008/05/rhizome.html Mon, 12 May 2008 12:14:15 -0800 ]]> http://joshua.schachter.org/2008/05/tag-mockery.html http://joshua.schachter.org/2008/05/tag-mockery.html complaining Thu, 01 May 2008 14:49:22 -0800 So when I realized that tomorrow's imminent arrival of Stupid Internet Joke Day (was: April Fool's Day) would require the avoidance of all unnecessary internet contact, it also occurred to me that I may as well point out some common Funny Anti-Patterns. But that's been done to death - thank you, internet hipsters.

I merely wish to remind you all that elaborate hoaxes (press releases involving small company acquiring a large one, switching stylesheets with someone, etc) are immediately and transparently stupid. Instead, try to actually do something surprising. The ha you save might just be your own. ]]> http://joshua.schachter.org/2008/03/stupid-internet-joke-day.html http://joshua.schachter.org/2008/03/stupid-internet-joke-day.html complaining Mon, 31 Mar 2008 12:57:34 -0800 However, a simple trick I learned early on is that even if you have only a single web server, a proxy in front can help out performance significantly. Through the simple expedient of buffering the communication with slow web clients, your potentially heavyweight (especially when mod_perl meant that each process was dozens or even a hundred megabytes apiece) and/or expensive Apache processes don't have to waste time serving every request for the entire length of time the client is connected. This allows you to run vastly fewer Apache processes.

In the past, I've used pound and perlbal. Pound is fast and lightweight, and allows routing based on the HTTP query; for example, everything under /img/ got routed to a high-speed thttpd instead of the Apache itself. Perlbal is much more configurable but slightly harder to get running, and the documentation was sparse.

These days, I'd also investigate nginx and varnish. Pen, a generalized TCP load-balancer with server affinity (connections will go to servers they've gone to recently in the past) is also quite interesting but will not help with the slow client problem. Finally, a second set of apache processes, configured to reverse-proxy via mod_proxy, will also do the trick. A ]]> http://joshua.schachter.org/2008/01/proxy.html http://joshua.schachter.org/2008/01/proxy.html lessons learned Tue, 22 Jan 2008 00:16:32 -0800 fed up with PR folks spamming him.

Me too.

These were sent to the email address listed on my blog; I use tear-off addresses for subscribing myself to things.]]> http://joshua.schachter.org/2007/10/unsubscribe.html http://joshua.schachter.org/2007/10/unsubscribe.html complaining Tue, 30 Oct 2007 20:54:13 -0800 I find myself lately re-entering everyone I know into the system every year or two; I remember Six Degrees, Friendster, Linkedin, (I skipped MySpace -- I'm too old,) Facebook, Dopplr, Flickr, and so on. Brad Fitzpatrick seems to agree that this is an annoying waste of time, and says so in his thoughts on the social graph.

Most social systems never forget anyone. Given that recent behavior appears to send friend requests to anyone you've ever met even briefly, I find my contacts list ends up filled with people I don't really know. In many systems, removing someone from your list is either buried or simply impossible. Further, since these systems make implicit relationship information explicit, deleting someone becomes a loud signal. In real life you would merely back off a bit, but the systems only allow you to express a binary sort of relationship.

Therefore, switching networks becomes a way to regularly cleanse your contact list. There is evidence that younger internet users regularly start new instant messaging IDs; this likely serves a similar purpose.

So perhaps frequent switching is less a function of fashion but instead a coping mechanism to deal with the mismatch between reality and software.