web acceleration, web performance, website acceleration, CDN acceleration, bandwidth management

Data compression is one of the cornerstones of the traditional WAN Optimization technologies. By reducing the amount of data that have to be transmitted over the WAN/Internet, it increases the effective throughput of the WAN link by a large factor.

The existing WAN compression products in the market are primarily built upon proprietary protocols for the convenience of the designs and implementations. The drawback, however, is the complication in the deployment phase. Proprietary protocols require the firewalls to be reconfigured to allow such "unknown" traffic in and out. Since most data traffic is encapsulated in the proprietary tunnel, the traffic auditing and logging is also disrupted on the WAN side. To make it worse, since the compression algorithms have to work in the symmetric environment, the error-prone reconfigurations have to be performed on both parties of the deployment. The equipment upgrade after the deployment on any side could easily break the setup and cause service interruptions.

Besides all the advanced features to be elaborated below, AppEx's HyperCompression has been designed with lowering TCO in mind. Once the compression feature is enabled, it automatically detects if the peer is also an AppEx device capable of performing a whole set or subset of the compression/caching functions. If so, they automatically negotiate the methods to use and all the detection, negotiation, and further data compression messages are sent inline the flows that the users initiates. This ensures maximum firewall penetration and virtually zero-reconfiguration of the existing network infrastructure.

AppEx HyperCompression technology offers the following advantages:

1. Maximum compression ratio via deep compression (Byte Caching with Compression): The data transmitted is first looked up and indexed in a dictionary to find a match of the history data. If a match is found, the entire data block can be shrunken into a 2-byte ~ 5-byte token. The matching is performed in a sliding-window fashion so that data blocks differ only by offsets can also be picked up. This history match works on non-compressible data equally well, which means the repetitive transmission of the same video or encrypted files is able to achieve very high compression ratio. After the tokenization with the history data, the resultant token sequences are once again compressed by the LZ-algorithm. This ensures nearly zero redundancy in the final data to be transmitted.

2. Super low latency: For better compression ratio we need larger history storage, therefore Hard Drives are used to store the history data. However, disk access is orders of magnitude slower than memory access, causing huge delays. AppEx's intelligent 2-Level Caching Dictionary is designed to solve this dilemma. It works like a large-scale, intelligent OS VM pager that dynamically calculates the most probable "pages" to be accessed at the moment and prefetches them into the memory so the data will be looked up at full speed. At the same time it monitors the processing delay and skips caching to perform LZ-compression when necessary to limit the latency.

3. Bidirectional and concurrent caching: To maximize the effectiveness and efficiency of the caching, AppEx HyperCompression is able to use the data received from the peers as history, as well as those from the ongoing flows. Therefore it is able to use all the currently available data to promote the compression ratio.

4. Auto-detection, auto-negotiation and inline-messaging: as explained above, this dramatically lowers the deployment/maintenance cost and improves the firewall penetration.

HyperCompression Framework

For more details, please refer to AppEx whitepaper "HyperCompression".