NB: this document is a work-in-progress. We'd like to encourage people to add comments without changing the text until we've got some kind of consensus; the right place to discuss this stuff and achieve consensus is on the rest-discuss mailing list. A bit of protocol: if you're editing this page on the Wiki, please be careful not to mess up somebody else's edits. If you're going to be editing the page for a while, please note this in big bold caps up here at the top. Be sure and look for that kind of indication yourself before diving in to a long edit. -- PP, JB    (MA)

Abstract    (MB)

This specification describes a mechanism called "HTTPEvents". These allow a publisher application to notify another server or an end-user when a particular web resource changes. All of these services can be provided through HTTP. This document describes some conventions to make the construction of generic event publisher and subscriber software feasible.    (MC)

For instance a user might want to know when a static web page changes or when a search engine result changes. A web resource might also be something more abstract such as an email mailbox. Then the subscriber would want to know when sub-resources (email messages) are added to the resource. One can also imagine a resource called "the price of stock XYZZY if and only if it is beyond $100." If that resource went from being empty to having a value, then obviously the stock had gone beyond the limit.    (MD)

This specification tries to build on the basic HTTP model by being very URI-centric. Rather than invent some event filtering language, we encourage you to use specialized URIs: just as Google did not invent a text querying language but instead uses query URIs. Instead of forcing every event engine to implement some kind of implicit message queue, we encourage queues to be represented as URIs that have links to other URIs (perhaps better termed "channels"). The primary benefit of making everything explicit in this manner is that a service can be off of the network for a while and easily get "caught up" on what messages are in the channel. Multiple subscribers can also share a channel.    (ME)

Terminology    (MF)

Publisher Application: Software that generates events of interest to the Subscriber Application.    (MG)

Subscriber Application: Software that receives events from the Publisher Application.    (MH)

Watched Resource: A resource (in the HTTP/URI sense) that can be watched according to the conventions described in this document.    (MI)

Subscriptions URI: A resource representing the subscriptions to the watched resource.    (MJ)

Subscription: A resource that represents a commitment by the Publisher Application to deliver notifications to a particular Subscriber.    (MK)

Server: An HTTP Server serving a watched resource.    (ML)

Client:An HTTP Client that represents the Subscriber Application.    (MM)

Publisher Event Source: An HTTP Client that generates notifications on behalf of the Publisher Application.    (MN)

Notification: An HTTP request from the Sever-side Event Source indicating that the watched resource has changed.    (MO)

Subscriber: An HTTP Server that collects resource change notifications on behalf of the Subscriber Application.    (MP)

Initial Negotiation    (MQ)

Initial Negotiation: An HTTP Server MAY indicate that a particular resource is "watchable" by returning a "Subscriptions" header to a GET or HEAD request. This should be a URI addressing a Subscriptions resource. This may be hosted on the same machine, but it need not be. It might also be implemented by a third-party that somehow knows when the resource changes (e.g. polling or email notifications).    (MR)

If a Subscriber Application knows a priori about an appropriate Subscriptions resource then it may choose to bypass this step. For instance a Subscriptions URI might be listed on a web page or it might be possible to generate them through URI concatenation of the form: "http://mysubscriptions.com/myaccount?uri_I_want_to_watch".    (MS)

Subscription Request: A Subscriber Application that wishes to subscribe should POST to the Subscriptions resource. This is called the Subscription Request. The message MUST contain a header called "Subscriber". This contains the subscriber URI. Subscriber Applications are encouraged to use a unique URI per subscription to make subscription management easier.    (MT)

Subscription Response: For interactive applications it would be polite for the server to return an HTML page acknowledging the subscription. Otherwise the structure of the body is application-defined. The response to the Subscription MUST include a header called "Location" which is a URI that represents the Subscription. If a properly authorized client does an HTTP DELETE on this resource, the subscription is removed. The Publisher Appication should cease sending notifications. A future version of this specification may describe the structure of these resources more formally.    (MU)

Content type: The subscriber and producer MAY use a form of content negotiation to determine what should be in the body of the HTTP message. It should in some sense be a representation of the resource, but some form of "difference" may suffice if the two negotiate it. (this needs to be elaborated further because the negotation is not the same as normal HTTP...it is for a future request/response pair...also idempotency issues must be considered)    (MV)

Notifications    (MW)

Notification: When the resource changes, the Server-side Event Source MUST send a notification message to the Subscriber URI. The HTTP model is explicitly flexible about what it means for a resource to change. For instance incrementing a page counter might not be considered a meaningful change by a particular Publisher Application. This specification's definition for "change" will be more formally unified with HTTP's at a later date.    (MX)

The HTTP method used for the notification is up to the application but the method must not be GET, HEAD or any other method that is defined to be safe (side-effect-free).    (MY)

Query parameters or other URI "tricks" can be used to generate logical resources with particular change characteristics.    (MZ)

A notification message MAY have the following headers:    (N0)

Watched-URI: (URI of the watched resource) 
             The Watched-URI may be any valid URI for the resource that generated the
             notification. It need not be the original URI but it must refer to the
             same resource. If the client wants to retrieve a representation of the
             resource it SHOULD use this URI, not the original one. This will allow
             publishers to do a simple form of load balancing.

Subscription-URI: (URI of the subscription representation)    (N1)

In order to improve performance, a future version of this specification may allow a user to ask for minimized messages without these headers.    (N2)

The body and other headers of a notification are application defined. But they MUST be redundant and idempotent.    (N3)

Redundant: It MUST be possible for the subscriber to build appropriate subscriber-side state merely by doing GETs on the resource and other resources referenced by it. If it misses a message (e.g. if it is offline) then it can (re-)build its state by doing so. For instance, if the body is some form of "diff" then the full resource should be available on the server site. In other words, the message body and headers are always just an optimization, never a replacement for proper resource modeling on the server.    (N4)

Idempotent: It MUST NOT be harmful for a Subscriber Application to receive the same notification twice. For instance, if the body is some form of "diff" then it must be explicit about the document version that it is a diff against so that a subscriber will not accidentally apply it twice. Merely embedding message IDs is usually sufficient for this purpose.    (N5)

Each notification response MUST include a "Continue-Subscription" header. The value associated with this header MUST be the string "true". If the header is not present, the publisher MUST remove the subscription.    (N6)

If the publisher fails to deliver the message or to recieve a legitimate HTTP response then it MAY retry an application-specific number of times after an application-specific amount of time has passed. Servers are encouraged to use some form of exponential back-off strategy.    (N7)

Mail binding    (N8)

If the Subscriber URI is a "mailto:" address then the Publisher Application MAY either agree to email notifications or report an error. The exact binding to SMTP may be elaborated by a future version of this specification. Any binding must have the following constraints: The server MUST NOT send more than one notification without a reply acknowledging that the subscriber wants to continue the subscription. The server MUST have some mechanism for associating the delayed response from the subscriber with the notification.    (N9)

Legacy application support    (NA)

Some legacy applications have trouble working with headers. In the future, this specification may define a convention for using query parameters to represent the same information that is now sent in headers. Ideally, however, query parameters would be reserved for the exclusive use of the application built on top of the notification service.    (NB)

Standardization    (NC)

If these conventions are found to be flexible and general enough, then their basic ideas could be reformulated into an HTTP extension (perhaps with an extension method) one day.    (ND)

Examples    (NE)

Web Page    (NF)

Consider the example of watching a web page. The subscriber might want a notification if the page changes. This interaction could proceed as follows.    (NG)

The end-user could type the page's URL into the browser. The server would return an "Subscription" URI indicating that it is willing to watch this resource for the subscriber. An HTML HTTP-equiv could also be used if it was more convenient. In addition, the subscription URI could be rendered as an ordinary HTML button. The only problem is that buttons typically cannot set up HTTP headers. This could be done either through Javascript or using the legacy application support described above.    (NH)

The browser could enable a menu item that would indicate that page is watchable. The user would click the menu item or button. The browser would ask the user whether the user would like the notifications to go to email (mailto: ...) or the current machine (microserver) or a store-and-forward service run by the browser manufacturer (ad-supported). The browser would issue an appropriate POST to the Subscription-URI.    (NI)

Watchlist    (NJ)

A good metaphor where notifications may be required, but also where one expresses more interest in some events/pages/changes than others, is the "watchlist" we all know and love from some wikis (like MediaWiki).    (NK)

Stock Quote    (NL)

Consider the example of watching a stock quote. The client may only want a notification if it goes below $10.00.    (NM)

The quote provider could design a web page that generates URIs that match these constraints:    (NN)

POST http://www.stockquote.com/quotewatcher/PAUL?floor=10.00
Subscriber: http//my.handheld.computer.com/watches?431452    (NO)

The response could be an HTML page thanking the user for the subscription. Just as on the Web of today, the bit following the question mark can be more or less a query language. If you wanted a notification every time it passed a dollar boundary, you could do "?increment=1.00". If content negotiation is unavailable for some reason, you could also specify the representation you would like.    (NP)

The client machine would have to have a (potentially tiny) HTTP server running on it. For end-users on devices that are intermittently connected, it would make more sense to set up a server that stores notifications and can deliver them in batch on request (not unlike the role HoTMaiL? plays for email).    (NQ)

Large Website    (NR)

What if you wanted to send out a notification when anything, anywhere on a large website changed? Well, you could have the web server maintain a "/changelog" resource which is updated when anything, anywhere changes. That changelog would have dates, times, pointers to the changed page and even a summary of the change (add, delete, diff, etc.).    (NS)

Users could watch that changelog and then download it when it changes to see what precisely had changed. Old events could "scroll off the top" into another file to keep the changelog of reasonable size. Query parameters could be used to filter particular types of changes. As an optimization, change log entries might be inlined into notifications.    (NT)

Multi-website Corpus    (NU)

A vast corpus like the [macro[GFDLTextCorpus]], accessed through multiple sites but with strong legal and cultural cohesion, may need a common watchlist for all of them at once. See http://wikinfo.org/wiki.phtml?title=GetWiki:InterWiki for some of the issues involved with this.    (NV)

Open Questions    (NW)

What subset of HTTP must the subscriber support?    (NX)

Could the subscriber's HTTP server be a stealth server that only accepts UDP packets, not TCP?    (NY)

What does an event queuing service look like? Should/can it be combined with an event generation service? Ideas here: NotificationQueuing    (NZ)

Perhaps mailto: URIs should be merely disallowed and a separate proxying service should be defined.    (O0)

Is it a waste to continually send back the watched URI and subscription URI?    (O1)

Ideas that have not been fully fleshed out yet (and may not be)    (O2)

Events    (O3)

Notifications could have an Event header:    (O4)

Event: (CHANGED|EXTENDED|DELETED|ADDED|(or application defined))    (O5)

The types of Events correspond in semantics to HTTP methods: CHANGED is like PUT over an existing resource, EXTENDED is like POST, DELETED is like DELETE. ADDED is like an "original" PUT. Over time we could come up with guidelines for when it is appropriate to choose application-defined (if ever).    (O6)

MikeDierken is the primary champion of something like this (but I would rather see the notification indicate the operation that occured (or didn't occur) by reflecting the HTTP methods, since they are the only way to interact with the resources). See for example this message and this one.    (O7)

Etags    (O8)

There could be some strategy so that clients know what precise version of a document they are getting a notification for. This is primarily an optimization as discussed here.    (O9)

Jeff Bone is the primary champion of something like this.    (OA)

Contributions    (OB)

Editors: Paul Prescod, Jeff Bone    (OC)

Contributors: Lucas Gonze, Mike Dierken, Mark Baker, Jim Ancona    (OD)

This makes a lot of sense. One minor comment: my sense is that if you think that an implementation should do something, it probably must, because you needlessly complicate the state logic of the client if it must fork based on a server implementation. In my reading of this, the only thing that really arguably should be optional is the "polite response" to a subscription.    (OE)

-Bill Hofmann (wdh@acm.org)    (OF)

For more context on this stuff, it might be good to look at draft-khare-notify-scenarios-01.txt. It is quite old but it encapsulates a lot of good thought on the issues. We need to carefully mine it for both requirements and warning signs.    (OG)

- PaulPrescod    (OH)

This looks like a good proposal. My only niggle is that the garbage collection on unused subscriptions seems a little weak. Presumably a publisher will expire subscriptions, and it would be nice to make the expiration time explicit so that subscribers can re-subscribe (sort of like leases in jini).    (OI)

A possible mechanism would be to rely on HTTP caching, and use the 'Cache-Control' header in the subscription request. This would be preferable to using an 'Expires' header because the Expires header does not actually imply that the resource has expired. The use of a time delta rather than absolute timestamp is probably less prone to error as well.    (OJ)

For example, the subscriber might post    (OK)

POST http://.....
Subscriber: http://me
Cache-Control: min-fresh=100    (OL)

indicating that it wants a subscription of at least 100 seconds. The publisher might respond with    (OM)

blah
Cache-Control: max-age=60    (ON)

indicating that it is only willing to give out a subscription for 60 seconds. The client is advised that the subscription may be garbage collected after this time. There is a slight inconsistency here in that the subscriber is sending a min-fresh value that the publisher is not really honouring. Instead, the semantics are the same as jini leases, where the subscriber requests a particular lease period, but the publisher grants only what it's willing to honour (and is explicit about the length of the lease given).    (OO)

I suppose the default of having no Cache-Control header brings us back to the base state of having an indefinite lease.    (OP)

The 'Continue-Subscription' header in the notification is still a nice optimisation for well behaved clients that are nice enough to cancel subscriptions.    (OQ)

Well, it's a wee bit heavier, but probably cleaner. What do people think?    (OR)

- JamesUther    (OS)

KnowNow.com has built something like this - a microserver installed on the client. However, the thought of putting a web server in every browser, something I built years ago with an ActiveX? control, opens up such a massive security hole that the concept will never be adopted. Period. -dave cline    (OT)

I agree with Dave, too many firewalls are going to block the callback even if people want to use this. If you can't push at something then you have to convince it to pull, which requires polling. Polling sucks but we could use it as a fallback. The event source would have to implement both, but the event sink need only handle one. Gnutella uses something similar IIRC to this to deal with firewall issues. -- JonHanna    (OU)

A comparible service could be provided for polling: having the notification service queue notifications and have the client poll a single queue resource to receive notifications. Not uncoincidentally, this is just like RSS, RSS, and Atom but without the full resource content in the notification. Atom folks have discussed this as SuperSimpleFeedFormat but I don't believe many people are grokking the usage. -- KenMacLeod    (OV)

Some prior work has been done in this area, notably GENA. I have also written several blog entries on the subject, including This one discussing the relationship between HTTP subscription, polling, proxies, and the tiers of the internet. I also lay out some of the design goals I think HTTP subscription should meet. I recently hit the squid-cache-users list and the dialogue is pretty much summarised in this email.    (5DW)

Update: I have started my own specification, HttpSubscription.    (5H0)

To discuss this specific proposal I have a few comments.    (5DX)

You may also find my critique on GENA useful.    (5E9)

-- Benjamin Carlyle    (5EA)

I have implemented an open-source HttpEvents library, using the definition as it stands above. The intended use is to create anti-spam networks. Anyone interested in helping out with the project is most welcome. -- ChrisPurcell    (5PU)