ZMODEM

ZMODEM
Communication protocol
Purpose	file transfer protocol
Developer(s)	Chuck Forsberg
Introduction	1986; 38 years ago
Port(s)	None
Hardware	modems

ZMODEM is an inline file transfer protocol developed by Chuck Forsberg in 1986, in a project funded by Telenet in order to improve file transfers on their X.25 network. In addition to dramatically improved performance compared to older protocols, ZMODEM offered restartable transfers, auto-start by the sender, an expanded 32-bit CRC, and control character quoting supporting 8-bit clean transfers, allowing it to be used on networks that would not pass control characters.

In contrast to most transfer protocols developed for bulletin board systems (BBSs), ZMODEM was not directly based on, nor compatible with, the seminal XMODEM. Many variants of XMODEM had been developed in order to address one or more of its shortcomings, and most remained backward compatible and would successfully complete transfers with "classic" XMODEM implementations. This list includes Forsberg's own YMODEM.

ZMODEM eschewed backward compatibility in favor of producing a radically improved protocol. It performed at least as well as any of the high-performance varieties of XMODEM, did so over links that previously did not work at all, like X.25, or had poor performance, like Telebit modems, and included useful features found in few other protocols. ZMODEM became extremely popular on bulletin board systems (BBS) in the early 1990s, becoming a standard as widespread as XMODEM had been before it.

Improvements

Streaming

Generally, file transfer protocols break down a file into a series of packets, and then send them one-at-a-time to the receiver. The main portion of the packet, the payload, is a certain number of bytes from the file being sent. After the payload comes a checksum or cyclic redundancy check (CRC) that can be used to determine if the payload was received correctly. If the packet is received correctly, the receiver sends an ACK message and the sender then starts sending the next packet.

The telephone system introduces a small delay known as latency that interferes with this process. Even if the receiver sends the ACK immediately, the delay in the phone lines means there will always be some time before the sender receives it and sends the next packet. As modem speeds increase, this delay represents a larger and larger number of packets that could have been sent during the delay, decreasing the channel efficiency.

XMODEM used 128-byte payloads with a three-byte header and one-byte checksum for a total of 132 bytes per packet. In the era of 300 bit/s modems, a packet took about four seconds to send, and typical latencies were on the order of 1⁄10 of a second, so the performance overhead was not significant. As speeds increase the problem becomes more problematic; at 2400 bit/s a packet takes about 0.55 seconds to send, so about 1⁄5 of the available bandwidth is wasted waiting for ACKs. At 9600 bit/s a packet requires only 0.13 seconds to send, so about 1⁄2 of the bandwidth is wasted.

One solution to this problem is the use of a sliding window. These protocols address latency by allowing the sender to continue sending a number of packets without waiting for an ACK. The number of packets that it allows to continue is the "window", which was typically between two and sixteen packets in most implementations. A number of new versions of XMODEM with sliding window support appeared in the early 1980s.

Sliding windows are useful for latencies on the order of several packet lengths, which is the case for XMODEM on conventional phone lines. However, it is not enough to address longer latencies found on overseas phone calls, satellite connections, or X.25 services such as PC Pursuit, where the latencies are on the order of a second or longer. In other cases, where the reverse channel was much slower than the sending one, as was the case for Telebit or US Robotics modems, even the small number of ACKs might overwhelm the return channel and cause the transfer to pause.

ZMODEM addressed these problems by removing the need for ACKs at all, allowing the sender to send data continually as long as the receiver detected no errors. Only NAKs had to be sent, if and only if there was a problem. Since ZMODEM was often used on links with built-in error correction, like X.25, the receiver would often not send a single message back to the sender. As a result, the system would send the entire file in a continual stream, and ZMODEM referred to itself as a "streaming protocol".

ZMODEM's performance was so improved over previous common protocols that it generally replaced even special protocols such as YMODEM-g, which included no error correction at all and instead relied on error-free links maintained by the modems. Although YMODEM-g was faster (and thus popular among "power users"), the lack of other features such as restartable transfers made it less appealing.

Restart

XMODEM, and most protocols based on it, managed packet order by prefixing the data with a packet number from 1 to 255. Windowed versions used this packet number to indicate which packets had been received properly, or specify one that had not. Since the packets were 128 bytes long, this meant the maximum amount of data that could be transferred before the packet numbers rolled over was 32 KB.

ZMODEM replaced the packet number with the actual location in the file, indicated by a 32-bit number. This allowed it to send NAK messages that re-wound the transfer to the point of failure, regardless of how long the file might be. This same feature was also used to re-start transfers if they failed or were deliberately interrupted. In this case, the receiver would look to see how much data had been previously received and then send a NAK with that location, automatically triggering the sender to start from that point.

Auto-start

Auto-starting simplified management by allowing the sending machine to start the transfer. Previously the user had to first request the file from the sender, placing it into a "waiting" state, then return to their local program and invoke a command to start the transfer. With auto-transfer, they simply requested the file, the sender would then automatically trigger the transfer in the user's program.

Variations

A number of modified versions of ZMODEM appeared. ZedZap was a variant of ZMODEM with 8 KB blocks for better performance on high-speed modems. LeechZmodem was a mischievous ZMODEM variant (among similar XMODEM and YMODEM derivatives) that cheated BBS download quotas. A backwards-compatible extension of ZMODEM with 32 KB and 64 KB block lengths was created by ADONTEC in 2002 and 2007 to increase performance on high-speed error free connections like ISDN or TCP/IP networks.

The most notable ZMODEM implementations were from Chuck Forsberg's Omen Technology, Inc. These included DSZ (DOS Send ZMODEM), GSZ (Graphical Send ZMODEM), and the ubiquitous (l)rzsz for Unix variants.

In more current times, the developers of Synchronet have created a modern X/Y/ZMODEM implementation named SEXYZ, loosely based on the zmtx/zmrx package, which runs natively on Windows and Unix variants, supports long filenames and faster, more reliable data transfers. The ZMODEM implementation from SEXYZ has also been incorporated into the SyncTERM project. Synchronet, SEXYZ, and SyncTERM are all open-source, cross-platform, BBS-centric projects.

Forsberg himself collected a number of improvements into ZMODEM-90. The first of these is MobyTurbo, which removed control quoting to further improve performance, about 15%. Even on networks that "eat" control characters, ZMODEM-90 can be tailored to quote only those characters the network actually eats, as opposed to every possible one. A similar improvement allows ZMODEM-90 to work on 7-bit networks, whereas earlier protocols (with the notable exception of Kermit) had all demanded 8-bits to one degree or another. Finally, ZMODEM-90 includes a basic run-length encoding compression system to further improve performance on uncompressed files.

Limitations

Some of the ZMODEM packets (e.g. ZACK, ZRPOS) embed a byte-offset within the transferred file as a 32-bit unsigned integer. This design limits the feasibility of ZMODEM to only reliably transfer files that are under 4GB in size.
Even though the protocol could permit it, the reference (l)rzsz implementation cannot encode arbitrary non-control characters (e.g. '~') which are often used by TCP/IP connection programs like telnet and ssh as client-side "terminal escape" characters. Users must disable the terminal escape feature to achieve reliable transfers over these kinds of links, e.g. ssh -e none user@hostname.

References

External links

Evolution and Selection of File Transfer Protocols