Академический Документы
Профессиональный Документы
Культура Документы
and the
Programmable World of Tomorrow
Jeff Lindsay
three years after coining web hooks, people are starting to get excited about them.
i’m going to share what they are, why they’re significant, and what’s going on
in the growing web hooks ecosystem
WEB HOOKS
and the
Programmable World of Tomorrow
Jeff Lindsay
i’ve given a version of this talk before, but this time i’d like to try and focus on this
“programmable” idea that helped inspire web hooks.
WARNING:
Emergent dynamics described ahead.
was a guy really excited hearing about web hooks, but was disappointed to see what they are.
i’m describing a game changer based on a mechanism that takes one line of code to describe.
seems so simple, people assume you can only do simple things. quite the contrary...
go. simple mechanisms can create rich dynamics.
easily teach somebody the rules, doesn’t mean they see the implications of those rules.
i want to share the implications of this simple mechanism.
in 1997, jon udell talks about websites as data sources that can be reused and remixed
today that idea isn’t very novel
“a new programming paradigm
that takes the whole Internet
as its platform”
he envisions the Internet as a programming paradigm
The Programmable Web
“The Web as Platform”
it starts to manifest as the programmable web and talk of “the web as a platform”
The Programmable Web
APIs and Mashups
john musser here starts to track apis and mashups in 2005. there are now over 1k apis and
increasingly more.
apis and mashups became the foundation of the programmable web...
Pro g ramm at ic
The Programmable Web
APIs and Mashups
but are they? after thinking about web hooks and what they can give us, i realized apis and mashups
don’t make a programmable web. they make a programmatic web
Pro g ramm at ic
The Programmable Web
Twitter API lets you use Twitter programmatically.
It does not let you program Twitter to do more.
they let you use web apps programmatically. they don’t let you program them to do more,
whereas web hooks can. so i like to argue that web hooks will bring about the *real* programmable
web.
Web Service
interwebz
User
so let’s get specific. here’s an example scenario showing the use of web hooks.
Web Service
Hi, I’m Twickr,
a new web service.
interwebz
User
“Twickr”
interwebz
User
“Twickr”
Okay, you’re all set.
Now listen...
interwebz
User
“Twickr”
I might find myself with some
data you might be interested in.
interwebz
User
“Twickr”
interwebz
User
“Twickr”
interwebz
User
interwebz
User
“Twickr”
...sure, why not.
Hmm, what is that URL?
interwebz
User
“Twickr”
interwebz
User
“Twickr”
What does it do?
interwebz
User
“Twickr”
interwebz
User
interwebz
EMPOWERED
User
but do you see what they did? they made the web service extensible and empowered the
user.
a friend of mine got me to explain web hooks to her. not a programmer, but has iphone.
she compared it to jailbreaking the iphone: letting users do what they want, customize, add
apps, etc
having web hooks is like jailbreaking your web apps. opens functional extensibility.
a pseudo code example of what the heart of implementing web hooks looks like.
SIMPLE: make an HTTP request to a user-specified URL on major events
paypal represents on the oldest web hook implementors
their instant payment notification is a web hook
“When a customer pays you, PayPal posts a notification to your server at a URL you specify.”
its framed as a notification, but that doesn’t properly imply the usage its intended for:
integration.
started thinking about this in 2006.
everything flashed before my eyes and was very confused why it wasn’t used more.
felt like i was taking crazy pills. today i know why...
compare to REST.
both have been around longer. rest is simpler..
in fact, it’s almost described as “using HTTP properly”
but not until it got a name could it be used in discourse to make it popular
REST
Hooks
rest apis and web hooks are two sides of the same coin
they complement each other in ways i’ll get to later
but i just want to give this pattern a name, and start associating some ideas with it
Push
Pipes
Plugins
the methods and protocol ended up in the webdav implemented for exchange
but then in 2000, rss 0.92 was released. five years later it gets an icon and widespread
adoption.
it started with blog feeds, then comment feeds...
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds
even feeds of feeds
it makes you think of feeds like in the telecom world
data coming directly to you
but we know that’s not how it works
?
of course, now we have other web applications consuming feeds and it doesn’t make sense.
even most of our feed readers have become web applications
SUP is a recent incremental solution to increase the efficiency of consuming large amounts of
feeds.
however, its based on polling and is essentially yet another feed,
so i couldn’t help...
SUP DAWG, WE HEARD YOU LIKE FEEDS
SO WE MADE YOU A FEED OF OUR FEEDS
evan and kellan gave a great talk a while back about this push issue, and their proposed solution:
xmpp
i have a condensed version of this talk. the slides speak for themselves
(aka XMPP)
they have a great point. polling sucks, and xmpp is a pretty good solution for data streams
but it’s kind of heavy weight. it does a lot and makes a decently complex little system.
luckily it’s not *that* hard to use with today’s library support
evan and kellan did point out there are extremes when it comes to data streams.
most data streams will probably fall somewhere in between,
but i do think xmpp is suited for the fast and furious end
joshua schachter of delicious responded to this talk in a blog post.
he basically suggests web hooks as an alternative
“One solution that occurred to me at the time was to build a simple callback system over HTTP.
This would fall comfortably between full polling and full persistent publish/subscribe.”
he says they fall comfortably between polling and xmpp. i agree, and think they can cover
most use-cases
others seem to agree....
this is a standard for discovering and subscribing to content changes.
they let you use web hooks OR xmpp, which is nice.
good idea, but (like GENA) standard specs alone don’t get very far
gnip is a service that some may have heard of (but don’t understand)
Polling
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
they’re basically an adaptor for data streams, letting you pick your own protocol and mechanism, no
matter what the feed publisher is providing.
Source Destination
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
however, they recently dropped support to consume via xmpp, making web hooks their primary push
mechanism for consuming data streams.
Source
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
as an example, if i wanted to get digg updates via web hooks, gnip will poll digg for me and invoke my
callback with the content as new updates come in.
Source
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
alternatively, i could even poll gnip for digg updates. although seemingly redundant, it helps ease the
load on digg and allows gnip to provide filtering functionality.
Protocol + Mechanism Protocol + Mechanism
Format Format
Web Hooks Web Hooks
RSS RSS
XMPP XMPP
Atom Atom
XML XML
REST REST
Publisher Consumer
but another example might be if twitter provides an xmpp stream, but i already have a polling setup...
gnip would let me poll instead of integrate xmpp.
microformats
in promoting web hooks, i sort of look up to the way microformats work in values and process.
very ground up, grassroots... take existing popular use-patterns and make it a convention.
microformats can be viewed as an alternative xml+rdf
xml+rdf vs microformats
so i told him about web hooks. “what are they?” “push over http”
“how are they diff than xmpp?” “they’re a lightweight alternative” lower the barrier to entry...
xmpp vs web hooks
he says...
so the moral here is that push is good and there’s finally a demand for it. we have some good
solutions waiting adoption...
XMPP is ideal when needed,
but Web Hooks generally do the job.
as far as xmpp vs web hooks, i think they both have their place.
web hooks are easier, so you might as well default to web hooks unless you really need
xmpp.
But push is not the point.
however, as i framed this talk, it’s really not about push. it’s a nice way to get the social
media kids into bed with web hooks, but hooks are really about more than pubsub and
notifications. otherwise i would have called them web notifications or something...
Pipes
instead of push, i was more drawn by the pipes metaphor. i wanted to integrate and
orchestrate the web applications i use, conceptually similar to unix pipelining.
so i thought about it. pipes were a really amazing feature that let you get more out of your commands
by letting you chain them together. it sort of let you program without programming, combining
commands in ways that weren’t necessarily intended by the author.
Input Output
Program
Program
STDERR
grep
wc
cat
so you had all these simple little programs, that might not even be useful alone
cat
xargs
wget
echo
mail
grep
wc
xargs
wget
echo
mail
wc
cat grep mail
xargs
wget
echo
wc
and you have something more useful than the sum of the parts
Write programs that do one thing and do it well.
this helped put forth the unix philosophy and encouraged building these small composable
commands
STDIN
Program
Web App
Web App
it’s not that they can’t, they just don’t. we just need to start putting hooks in so they can.
those roles are best played by mechanisms that use the protocol the web is built on: http
backhoe
+
front-loader + excavator
cat grep mail
Basecamp
Project finished
Todo completed
Milestone created
Contact added
File uploaded
Basecamp
My handler
http://example.com/handler
My handler
http://example.com/handler
Basecamp
for example, all these apps share data about todos. they each have respective specialized talents,
but all work with todos. by putting hooks on todo CRUD,
you can use their apis to keep them synced pretty well. magically. real-time.
Service integration
based on the idea that web urls can run code. and code can do anything.
when i first thought about this, cheap PHP hosting was all over, but it could be even simpler.
there are paste bins like this one. made to share formatted code with people over IRC or email
put in some code you want to share or ask a question about
http://pastie.org/84826
hit save and you get a URL you can share when you ask “where’s the data stored?”
i’m thinking of the same thing, but that you write handlers with
http://pastie.org/run/24576
and saving doesn’t just give a url to view, but to run. pass that url into an app with web hooks
Basecamp
Project finished
http://pastie.org/run/24576
for example, basecamp. now when you finish a project, everybody meets for shots in the break room.
fortunately, we have this. it’s called appjet by ex googlers.
just hit a button, write code, hit save, share the url. it’s javascript
obviously app engine, although it’s a little more involved than appjet for quick handlers.
but it is an option for python.
and there are ruby/rails hosts like heroku
one thing i’ve been working on is an extension to play with ways end-users might interact
with web hooks
Hey, there’s an
event hook here!
and write some code. hit save, it posts to AppJet (or wherever),
registers the handler (assuming a standard protocol), and done. all inline.
go back and change the code.
Real world examples
--Tobias Lütke
tobias used them in shopify. i’m told he’s revamping their api to have more hooks. they were
one of the earliest adopters and recently had their 1 yr anniversary using web hooks
google code recently caught up other code hosts by providing a post commit hook.
it was very well done, as you’d expect from google and in particular shows that
authentication can be done with hmac signatures
github was one of the first majorly popular site to use and promote web hooks.
they’ve been doing really well with their post-receive hook. users have used it to integrate
with mailing lists, chat, other project management tools, continuous integration, etc
they were so successful with the adhoc integration, they formalized it.
but in the best way! using their existing web hook infrastructure.
they just have modules running in a separate but local web service.
in fact, that lets them open source it. letting people fork, write new handlers, and push back.
this is probably going to be the standard model of service integration.
and a great example of services integrated with github, besides lighthouse, is runcoderun.
they run your regression tests for you. continuous integration in the sky. love it.
they sign you up automatically if you put their hook in github.
and of course i mentioned paypal. but i should mention, web hooks make so much sense
for paypal... they’re not so much about pushing content, but INTEGRATING. that’s what
web hooks are great for, even though they can be used for content push.
jott is another example of a web hook implementer that doesn’t know it.
they parse voice over the phone and do stuff with it, like post to twitter, etc
they do it with “Links”... which are just hooks. they post to a script that does something with
the parsed text. really cool for todos.
User jotts to The message is Message is sent via Jott reads back the User receives
a Jott Link converted HTTP Post to a response and sends information back
into text web page it via SMS & SMTP
logic and content. the read-write web gave us user contributed content...
i want user contributed (democratized) logic.
Function
Content
logic and content. the read-write web gave us user contributed content...
i want user contributed (democratized) logic.
Integrate
Function
Content
Function
Content
[General]
Name = Amazon Sales Rank
only they have an extra layer for meta data. but that’s a cool pattern.
“If you’ve used a UNIX-based operating system, you’re probably
familiar with the notion of pipes. The output of one program is
piped into the input of another, creating a filter chain. This is
conceptually the same as the way Dabble’s plugin IO works.
Nice and simple.”
of course, they compare it to pipes. the simplicity. the natural fit of it.
of course, i think they should have web hooks for all their standard CRUD events...
this way their database apps can integrate (like PayPal) with the rest of your workflow
in fact, all these “app platforms” like coghead and salesforce should have web hooks.
that would make them more useful, less silo’d off into just processing data in their world
IMified uses web hooks. sells the tech too: “allows anyone with basic web programming skills
to quickly and easily create ...”
General Systems
Theory
increase the ways to work together, the means of combination, increases the DoF
Increase possibility space
Thanks!
webhooks.org
“What is the equivalent of the pipe
in the age of the web?”
in 2000, just as rss 0.92 was being released, tim oreilly asks...