What is HTTP Delta Encoding? | 2

Delta encoding would only be useful if resources change relatively often, In fact, several studies have confirmed that many Web resources do change frequently. For example, Douglis et al. [4] reported on a large trace in which 16.5% of the resources changed every time they were accessed, and about 15% of the responses reflected changes in resources that had been accessed previously.

Delta encoding also makes sense only if the differences between instances are much smaller than the full content of a resource. Again, studies have shown that this happens fairly often. The potential savings vary by Content-type, because some Web content, such as images and continuous media, tends to change more radically than HTML text. The savings also depend on the efficiency of the format used to encode the differences (see 6). One trace-based study [12] showed potential overall bandwidth savings of 8.5% and latency savings of 5.6%, over and above the savings of a traditional cache. For HTML content, the study showed significantly greater improvements, reducing bandwidth requirements by 19%-24%.

Since basic delta encoding can only pay off if the resource being referenced has already been cached before (many URLs are never referenced more than once), and isn't of interest if the resource hasn't been modified (in which case traditional caching works fine), only a modest fraction of requests (about 10% of all requests, and 25% of requests for HTML files) are eligible for delta encoding. For such "delta-eligible" requests, the average savings are a lot higher: about 56% bandwidth savings overall, and 62%-65% savings for HTML files. In other words, when delta encoding can be used at all, it usually pays off quite handsomely.

3. The connection between delta encoding and caching

There is an interesting connection between delta encoding and caching. In traditional, full-reuse caching, the bits in a cache entry are either all useful (i.e., with a cache hit) or not useful at all (i.e., for a cache miss). With a successful delta encoding, however, while not all of the bits in a cache entry are useful, most of them are. In effect, delta encoding lets us extract utility out of more of the bits that are kept in a cache.

4. The connection between delta encoding and compression

While the term delta compression is sometimes used instead of delta encoding, it is somewhat misleading. Delta encoding and compression are related, but delta encoding is not really a form of compression analogous to traditional compression algorithms (such as gzip). Most traditional compression formats are self-contained; that is, you can convert between X and the compressed form of X without any other information. Delta encoding, in contrast, is not self-contained; both the sender and receiver need access to something like X but different from it.

Next Page: Delta encoding and rsync compared

Comments are welcome
Written by Jeffrey C. Mogul and