Network Working Group A. Bhushan
Request for Comments: 310 MIT-MAC
NIC: 9261 April 3, 1972
Another Look At Data And File Transfer Protocols
Our experience with ad hoc techniques of data and file transfer over
the ARPANET together with a better knowledge of terminal IMP (TIP)
capabilities and Datacomputer requirements has indicated to us that
the Data Transfer Protocol (DTP) (see ref 1) and the File Transfer
Protocol (FTP) (see ref 2) could undergo revision. Our effort in
implementing DTP and FTP has revealed areas in which the protocols
could be simplified without degrading their usefulness.
This paper suggests some specific changes in DTP and FTP that should
make them more useful and/or simplify implementation. The attempt
here is to stimulate thinking so that we may come up with a better
protocol at the forthcoming Data and File Transfer Workshop (see ref
3).
Experience to Date
A number of ad hoc techniques of transmitting data and files across
the ARPANET already exist. Perhaps, the most versatile of these
existing methods is the TENEX "CPYNET" system. The "CPYNET" system
uses an ad hoc or interim file transfer protocol developed by Ray
Tomlinson and others at BBN to transmit files among the TENEX systems
on the ARPANET. [Private Communication with Bill Crowther, BBN.]
In CPYNET, the using process goes through the Initial Connection
Protocol (ICP) to server socket 7, establishing a full-duplex
connection with an 8-bit byte size. Control information, including
user name, password, command (read, write, or append), file name, and
byte size for the data connection is transmitted from the using
process to the serving process. The original full-duplex connection
is then closed, and a new full-duplex connection is established using
the original socket numbers but with possibly a different byte size.
The file is now transmitted on this newly established connection.
The end-of-file is indicated by closing the connection (the mode of
transfer is thus similar to DTP "indefinite bit-stream").
CPYNET has been used quite extensively for transfer of TENEX system
files. Because data is not reformatted, and because the optimum
connection byte size may be used for data transfer, CPYNET is quite
efficient. The PDP-10 (and there are quite a lot in the ARPANET)
works more efficiently with a 36 bit byte size which minimizes
packing and unpacking of data, and increases effective I/O speed
Bhushan [Page 1]
RFC 310 Another Look At Data And FTP April 1972
(bit rate is 36 times the I/O word transfer rate instead of 8 times).
The closing and reopening of connections does increase overhead but
this is small in TENEX when compared with inefficiency introduced in
data transfer using an inappropriate byte size.
Data and file transfer has been achieved at other sites by a simple
modification of the user TELNET to enable the transfer of ASCII files
as terminal I/O data streams within the constraints of the TELNET
protocol. An example of this approach is the use of the "send.file"
and "script" features within the MIT-DMCG user-TELNET. Send.file
enables the PDP-10 (DMCG) user to transmit his local ASCII files to a
receiving process such as an editor at the remote host via a TELNET
connection. The program allows for a variable buffer size for
transmission, and measures the transfer rate of information bits.
Script enables a user to receive an ASCII file from a remote host by
essentially printing it out (the terminal output stream is directed
to a local file).
Our initial experience with the use of send.file program has affirmed
the almost linear relationship between buffer size and transmission
rate (inverse relationship to processing cost) until the limits
imposed by allocates, NCP sending buffers, the IMP message size, or
the receiving process speed, are reached. Our experiments have
indicated that TELNET processes in which the receiving process
"looks" at each character are slow and expensive. The transfer rate
to most TELNET receiving processes ranges between 200 and 2,000 bits
per second. The NCP-to-NCP transmission rate however is an order-
of-magnitude higher (2,000 to 20,000 bits per second).
A variation of the above method which avoids the character-by-
character processing of TELNET, is transmitting files via auxiliary
connections (other than the TELNET connections) with or without the
use of DTP. We are collecting data on response times, connect times
and transfer speeds employing different transfer and buffering
strategies.
TIP Capabilities and TIP Users
It appears now that TIPs will not support DTP in its present form.
The more elaborate TIPs with magnetic tape units will however,
support the DTP block mode (descriptor and counts) [Private
Communication with Bill Crowther, BBN.] It is inconvenient, at the
very least, for a TIP user to use services based on DTP (such as
remote job service, file transfer, mail, and Datacomputer). The TIP
philosophy is that "the computational load and storage should be in
the hosts or in the terminals and not in the terminal processor."
(See ref 4.) To be consistent with this philosophy the protocols
should be simple and convenient to use from the user viewpoint.
Bhushan [Page 2]
RFC 310 Another Look At Data And FTP April 1972
Ideally, TIP users would like to connect (using the initial
connection protocol) to the advertised service socket (including
logger socket1) in the remote host and type their commands in a
uniform, easy to use, format. Allowing the use of ASCII within DTP
would facilitate this. (An alternate approach is extending TELNET to
include DTP modes, particularly the indefinite bit-stream mode.)
Another step would be to use printable ASCII strings instead of
numeric codes for commands and arguments in user-level protocols.
Use of standard file system commands (with uniform interpretation and
format) will lead towards the existence of a Network Virtual File
System, much in the same line as Network Virtual Terminal defined in
TELNET protocol.
The transparent mode in DTP was specifically included to allow
convenient use by TIPs. Since the TIPs will not support transparent
mode, it makes sense to do away with it. This change would lead to a
simplier DTP which allows transfer in Block mode, and the indefinite
bit-stream mode. (The suggested default which would be acceptable to
all including the TIPs, as it involves no overhead.). We can then
make optional or do away with the now mandatory modes available
handshake. The using process can indicate if it also accepts the
block mode for transfer. (Either by modes available transaction, or
by an argument in the command string). The server should accept
input in DTP mode as well as ASCII. These fundamental changes in DTP
will make communication with TIPs a lot easier.
TIP users who do not have a mediating user-FTP process and a file
system in their TIP, would probably want to transfer files from input
devices or to output devices such as line printer, card reader or
punch, or magnetic tape. These devices "listen" on specific "ports"
or sockets on a TIP. It would be desirable to modify FTP to allow
sending data to a specified socket in a specified mode and type. TIP
users would then find it convenient to obtain listing of their files
on a high-speed line printer, input their files from a card reader,
and keep back-up on cards or magnetic tapes.
Datacomputer Requirements
We have been having a continuing dialogue with CCA personnel (Dick
Winter in particular), regarding CCA's plans for data and file
transfer on the Datacomputer, and their specific requirements. Dick
Bhushan [Page 3]
RFC 310 Another Look At Data And FTP April 1972
Winter will be speaking on this subject at the Data and File Transfer
Workshop. This is an attempt to summarize the main points of our
discussion, and their implication for data and file transfer.
First, CCA appears quite flexible at this stage regarding the manner
in which Datacomputer is to be used. Even the Datalanguage (see ref
5) is flexible and can undergo change. However, CCA would like some
changes in the current file transfer protocol and its envisioned use.
Ideally, CCA would like to see a single full-duplex connection for
transfer of all control information which is in Datalanguage. This
information is generated by a process, which may be a user at a
console, or a user program. Ability to inter-mix data and control
information would be definite advantage. The Datacomputer would
probably support DTP and allow use of TELNET-ASCII.
Data may alternatively be sent to or received from a separate user
defined port (which may be a socket). It would be advantageous if a
user could instruct the Datacomputer to transfer data to or from a
file in remote system via FTP (assuming a server-FTP in remote
system). CCA is currently not committed to this idea, but is
considering it.
In the CCA view, the Datacomputer represents a data management
facility with Datalanguage as its command language. From the
viewpoint of Datacomputer as an FTP server, FTP commands be a subset
of the Datalanguage. It is therefore desirable that FTP commands be
printable ASCII strings instead of numeric codes.
Remote Job Service Requirements
Initially two separate protocols were proposed for Remote Job Service
(RJS). One was the NETRJS protocol (see ref 6) for remote job
service from large Hosts and the other was the NETRJT Protocol (see
ref 7) for remote job service from TIPs (and other mini-Hosts). The
current thinking however, is to move towards a single RJS with "as
much overlap as possible between the methods of dealing with these
two user populations." (See ref 8.) Perhaps inclusion of ASCII
within DTP would make this feasible.
The existing proposals for DTP and FTP have been considered somewhat
less than optimal for RJS needs. Specific drawbacks of DTP and FTP
have been pointed out in the handling of data structures and data
types. Most of these problems seem relatively easy to resolve. It
would involve making Network ASCII the default data type (acceptable
to all hosts) and providing a way in FTP for proposing and rejecting
alternative data types and data structures.
Bhushan [Page 4]
RFC 310 Another Look At Data And FTP April 1972
Another inadequacy of FTP (which pertains to other applications as
well) is in the area of error recovery. Currently there is no way to
"restart" transmission if an element in the transmission path fails.
One solution suggested has involved the use of sequence number (see
ref 9). A number of other solutions exist to the problem. These are
discussed later in the section 'FTP Reconsidered'.
DTP Reconsidered
The aspiration for DTP was that it would provide a uniform mechanism
for separating information into its logical structure (records,
files, and control), and rudimentary error control. The evaluation
of DTP and its modes should be on the basis of speed (real-time),
efficiency (processing cost), reliability (error control and
recovery), and the ease of its use.
It is now clear that unless DTP was significantly revised, the TIP
and other mini-Host user would find it difficult to use services
based on use of DTP. Allowing the use of ASCII within DTP, and using
defaults instead of the "modes available" handshake, would alleviate
this problem, but compromise the DTP error control function. Whether
error control belongs at the DTP level or at a higher level needs
further discussion.
DTP, in its present form, does not provide sufficient error control
and recovery procedures. To make DTP more useful, either it should
be simplified (at least from a user viewpoint), or it should be
extended to include better error control with built in error
recovery, and possible handling of data types and data structures.
In the simplified version, DTP would only be a format procedure in
which data could be transmitted as ASCII (no format) with escape to
an 8-bit transparent (indefinite bit-stream) mode or in data blocks
(descriptor and count mode). The choice of which mode to use, and
all error control, error recovery, and aborts would be handled by the
higher-level protocol.
The utility of the block mode in data transfer has been questioned by
many who suggest that it puts a large overhead for providing the
simple function of indicating end-of-file, and separating data and
control information. The alternative data transfer strategy is to
use separate connections for control and data information and/or
close and reopen connections. This causes an overhead of a different
sort, but has the advantage that the byte size for connection may be
chosen to optimize data transfer.
Bhushan [Page 5]
RFC 310 Another Look At Data And FTP April 1972
A drawback of present DTP is that it is geared to transfer of 8-bit
bytes. Perhaps a good strategy for data transfer would be to allow
sending data in an agreed upon transfer mode. The transfer mode
chosen should determine the byte size for connection, the data type
chosen, and any data structure information. This mode may be chosen
at the DTP level, or at the using protocol level.
FTP Reconsidered
The aspiration for FTP was that it would facilitate file management
and file transfer in the ARPANET Virtual File System. FTP success
should be evaluated by the extent of its use, convenience and
efficiency in its use, and its suitability for other applications
such as Datacomputer, RJS, and Mail.
Wide use of FTP would be possible if a user could use an FTP-server
directly without the help of a mediating DTP/FTP-User process. This
would require that commands be ASCII strings instead of numeric
codes, and that ASCII characters be an acceptable input. Such a
change in FTP would greatly increase its acceptance at the cost of
making the server-implementation more complex. Combined
implementation, however, would be simplified as the mediating FTP-
user process (if used at all) would be simplified.
Efficiency of transfer is an important factor affecting the
usefulness of FTP. File transfer may be very expensive (in terms of
CPU time) and slow (in real-time) if an inappropriate transfer
strategy is used (e.g., inappropriate byte size). Every attempt
should be made to optimize transfer of data. A good strategy may be
to allow transfer of files over a separate connection or close and
reopen connections (using perhaps a different byte size). A problem
with indicating end-of-file by closing connection is that is
uncertain if the connection was closed because end-of-file was
reached, or because of a failure or error condition. Perhaps "NCP
interrupts" could be used in addition to a "close" to indicate
definite end-of-file condition.
A drawback in the present FTP strategy is that it has no restart
procedure. One proposal for restart has involved the use of the
sequence numbers used in DTP block mode. Our feeling is that perhaps
restart may best be accomplished by incorporating a command in FTP
that allows a user to specify the place in file where to begin
retransmission. A possible solution is to use the "SPF" command
implemented in the UCSB Simple-Minded File System (see ref 10).
Another solution may be to have optional arguments for retrieve and
store commands that allow selective retrieval and replacement
(specified by bits, character, words, lines, pages or segments).
Bhushan [Page 6]
RFC 310 Another Look At Data And FTP April 1972
Another useful addition to FTP would be a protocol procedure between
user and server to agree to data type, data structure, and mode for
file transfer. This would enable the user and server to reach the
optimum file transfer strategy acceptable to both.
Concluding Remarks
We have discussed in this paper what we see as the major problem
areas in the present DTP and FTP specifications. We hope this
discussion will stimulate thinking, so that we can arrive at revised
specifications for DTP and FTP that satisfy all the diverse needs in
an elegant manner.
REFERENCES
1. The Data Transfer Protocol, Bhushan, et al, NWG/RFC #264, NIC
#7212.
2. The File Transfer Protocol, Bhushan, et al, NWG/RFC #265, NIC
#7213.
3. Data and File Transfer Workshop Announcement, A. Bhushan,
NWG/RFC #309, NIC #9260.
4. The Terminal IMP for the ARPA Compuer Network, Ornstein, et al,
SJCC, 1972, NIC #8218.
5. Datalanguage, Computer Operation of America, Datacomputer
Project, Working Paper No.3, October 29, 1971, NIC #8208.
6. Interim NETRJS Specifications, R. T. Braden, NWG/RFC #189, NIC
#7133.
7. NETRJT - - Remote Job Service Protocol for TIPs, R. T. Braden,
NWG/RFC #283, NIC #8165.
8. RJS Protocol Meeting Notes, 25 February 1972, A. McKenzie
(limited distribution).
9. A Suggested Addition to File Transfer Protocol, A. McKenzie,
NWG/RFC #281, NIC #8163.
10. Network Specifications for UCSB's Simple-Minded Files System,
James E. White, NWG/RFC #122, NIC #5834
[This RFC was put into machine readable form for entry]