 |
The ASC Real-Time Data Acquisition Network (RTDAN): (NOTE: the RTDAN concept was developed by ASC for Sypris Data Systems - formerly Metrum Data-Tape Inc; Please Contact Sypris Data Systems for Sales, Applications, Systems Configurations, & More Information on the RTDAN & PADNI: Sypris Data Systems ) For some time there has existed a need for a powerful, flexible real-time data acquisition and storage architecture and implementation capable of scaling over a wide range of individual and aggregate data bandwidths, number of input channels, and total storage capacity, while preserving precise reconstruction (i.e. playback), accurate channel-to-channel time coherency, and ease of configuration, monitoring, and control. Prior approaches, based on TDM (time division multiplexers) feeding a single output stream into a suitable tape or disc recorder, can not scale to the massive aggregate data rates now being contemplated; only cell- or packet-based switched fabrics, i.e. capable of supporting multiple simultaneous wide-bandwidth communications links to multiple output storage devices can provide the requisite aggregate bandwidth, capacity, redundancy, and storage and network utilization efficiency that will be needed in the next-generation of real-time data acquisition systems. Existing Storage Area Networks (SANs) offer high-speed file-based storage that can scale flexibly over both aggregate data rates and capacities, with convenient centralized OAM, but do not address the special needs of real-time data acquisition including precision time-tagging, and clock measurement and reconstruction, and channel-to-channel coherency. ATM switches provide extremely high-performance, low-latency space-time routing of data. ASC ‘s RTDAN builds on the familiar SAN and ATM network, but with specially-designed packetizer/depacketizer network elements optimized for real-time data acquisition, storage, and reconstruction, and centralized, high-performance provisioning, configuration, monitoring, and control to provide breakthrough performance, flexibility, and scaleability in real-time multiplexing, data-acquisition, and storage systems. RTDAN Architectures – File-based/ATM-cell-basedIndependent of aggregate data rates, number of input channels, or storage capacities, two variations of the RTDAN are distinguishable, depending by whether data is ATM-cell-based multiplexed or file-based multiplexed. For storage devices that do not provide random-access capabilities, such as tape, the ATM-cell-based architecture is nearly always preferred over the file-based architecture because it is usually more cost effective than the multiple storage devices required to support multiple (streaming) files; however for storage devices (including RAIDs and SS memory) that do efficiently support random (i.e. file-based) access, or for those applications where the multiplexed files can be distributed over separate devices to avoid unnecessary head seeks, file-based multiplexing, because it builds on existing, widely-utilized and cost-effective SANs, can be suitable for all but the very highest-performance networks, and are generally more cost-effective. The ATM-cell multiplexed architecture offers the maximum flexibility and the highest available thruput where file-multiplexed architectures are impossible (tape), or impractical (slow head seeks on a shared device), but frequently at higher cost. Both architectures share very many similarities, and in fact both architectures are supported by ASC’s specialized packetizer/depacketizer/Network Interface (PAD/NI) network element. ASCs RTDAN PAD/NI (packet assembler- disassembler/network interface) is the specialized network element that interconnects the raw input/output data stream or TDM-input/output data stream to/from the Real-Time Data Acquisition Network, that is, it serves as a specialized network interface card. The card provides numerous real-time data acquisition-specific functions including high-speed (170 or 340 Mbytes/s) LVDS i/o, high-resolution (0.1/1 microsecond) deterministic time tagging of the input data, recording precision clock measurement on input and precision clock-regeneration on playback, inputs/outputs for IRIG a,b,g time code and auxiliary data/voice track, etc., in addition to dual network interfaces of either 1Gps/2Gbps FibreChannel (AL, P-P, or switched fabric), or Sonet/SDH at 622 Mbs/2.5 Gbps. Crucially, the PAD/NI supports striping of input data across multiple networked storage devices, and synchronization of inputs/outputs across both multiple storage devices and multiple PAD/NIs. Consistent with the prior definitions of file-based and ATM-cell based RTDAN solutions, the same RTDAN PAD/NI can operate either in SCSI-3 over FC (FCP) or ATM over FC; alternate RTDAN PAD/NIs operate in ATM over SONET/SDH. The PAD handles all inband OAM (Operation, Administration, and Maintenance) signaling as well; that is, the PAD/NI itself can be remotely configured, provisioned, monitored, and controlled as a full network element and part of a centralized, system-wide OAM. A PAD/NI serial control interface also allows the PAD/NI to configure, monitor and control associated TDMs or other "front-end" peripherals, making possible implementation of complete end-to-end system-wide OAM through the PAD/NI; e.g, sample rates of individual channel a/d converters, etc. That is, system-wide OAM can configure and control both the PAD/NI and the associated peripherals connected to the PAD/NI. The current PAD/NI is designed to support combined input/output (user) data rates of as much as 170 Mbytes/s. Next-generation (-400) PAD/NIs are planned to accommodate as much as 340 Mbytes/s for dual 2 Gbps FC and SONET/SDH. In addition to high-speed LVDS i/o, the PAD/NI also incorporates 2 compact PCI slots suitable for a wide range of COTS and custom data acquisition (multiplexer/demultiplexer)/signal conditioning cards, along with very considerable processing power on-board (1600 - 8800 MIPs) suitable for a variety of ASC-customizable functions including CCSDS formatting/deformatting, lossy/lossless compression, encryption/decryption, filtering/smoothing, decimation, detection, parameter estimation, etc. The combination of two compact PCI slots and massive on-board signal processing provide at once a compact, efficient, cost-effective, and highly-scaleable networked solution to many real-time data acquisition requirements. A high-level block diagram of the RTDAN network PAD/NI interface adapter is illustrated in Fig 1. A detailed functional specification of the FC PAD/NI-200 is included as Table 1. I.1 File-based RTDAN configurations Note that when used in the file-based solution each RTDAN PAD/NI itself serves as its own SCSI controller; that is the PAD itself directly controls the file-based device (RAID or RAIDs) associated with its own file(s) storage/retrieval, issuing commands and data to/retrieving data from files on the indicated device through the network. In networks with multiple PADS each PAD issues SCSI commands and data through the network to the destination device for the PAD’s file or files. This is a true SAN architecture: There is NO central server bottleneck controlling access to/from all the data storage resources; once provisioned as to file name and destination device, each PAD controls its own storage resources, in parallel with all other PADs. The FC network itself can be a point-to-point, Arbitrated Loop, or switched fabric. This SCSI over FC PAD/NI and network design offers an exceptionally flexible and scaleable architecture that is extremely cost-effective from a single PAD/NI and RAID to huge networks of PAD/NIs, RAIDs, and large switch fabrics. For example, an extremely compact minimum configuration is illustrated in Fig 2a. which shows a single ASC PAD/NI interfaced to a single RAID using a FC point-to-point connection. The illustrated "RAID data recorder" can support as much as 70 Mhz analog/170 Mbyte/s thruput with an appropriate RAID. No additional servers or controllers are required. The PAD/NI is provisioned/configured (device and file names) from an associated PC over serial link. An example of multiple PAD/NIs interfaced using a low-cost arbitrated loop fabric is illustrated in Fig 2b (and fully-redundant dual AL in Fig 2c) where as many as three PAD/NIs are interfaced to a single or multiple RAIDs for an aggregate 180 Mbytes/s throughput (dual-loop). Here the PAD/NIs may be provisioned as above, or a central OAM network element (a PC) can be added for system-wide provisioning, OAM, and control. Note that the PAD/NI has provision for synchronizing reference and TOD clocks across multiple PAD/NIs and synchronizing playback from both multiple RAIDs feeding a PAD/NI and across multiple PAD/NIs. Remotely located PAD/NIs can be synchronized using low-cost GPS receivers. For even higher performance, switched fabrics can be employed, alone or in combination with arbitrated loops. An example of such a configuration is illustrated in Fig 2d. which shows multiple arbitrated loops and point-to-point links connected to a switched fabric. Using commonly available RAIDs which can support sustained 90 Mbytes/s, the network illustrated can support 360 Mbytes/s with four RAIDs and a low-cost 8-port FC switch, or 720 Mbytes/s with 8 RAIDs and a 16-port FC switch; even larger and faster networks are easily constructed. Suitable 8 to 64 port FC switches are available from a number of manufacturers, including Brocade. Summary: The file-based RTDAN can support an extremely wide range of data rates and capacities and is easily and flexibly scaled depending on requirements, made possible by a specialized PAD which provides the required application-level support for real-time data acquisition combined with complete SCSI over FC built-in. It offers an exceptionally cost effective approach with random-access devices including RAIDs and SS memores. Its standard interfaces (SCSI and FC) facilitate interoperation with a wide variety of third-party vendors of storage devices, host bus adapters, FC switches, etc. Provisioning, and OAM can be flexibly scaled as well, depending on requirements, from at the local individual PAD/NI level or on a system-wide basis by a separate OAM network element. ASC will offer a system-wide OAM console application for Windows for those users wishing to implement centralized network OAM on a PC. The main disadvantage of such file-based RTDAN is the throughput effect of file-based multiplexing on the RAID, due to the necessary head-seeks when multiplexing from file to file. This can be mitigated by writing large numbers of contiguous blocks per disk (very large numbers of contiguous blocks per file), thus minimizing seek overhead on throughput, but requires very substantial data buffering in the RAID or elsewhere for the channels (files) not currently being written; in general, RAID throughput will suffer due to file-based multiplexing which may require additional RAIDs (and possibly larger FC switch in switched fabrics). Increasingly, RAIDS with highly-intelligent control algorithms perform block allocation when writing multiple files so as to minimize head seeks; however, to achieve the maximum design throughput from the disk devices data should be written/read as a stream of contiguous blocks (i.e. a single logical file). I.2. ATM-cell based configurations In the case of the ATM solution, multiplexed data is stored as a single stream of ATM cells in a single logical file on the storage device rather than across multiple logical files; this facilitates the use of devices without random access (such as tape), as well as maximizes the throughput of disk-based systems by avoiding unnecessary head seeks. The specialized PAD/NI outputs ATM cells with appropriate VPI/VCI in either ATM over FC (AAL5) or ATM over SONET/SDH line interfaces. Ordinarily, for real time data collection applications, these will be "Permanent Virtual Circuits" provisioned at system configuration; however, it shall also be possible to establish and tear-down virtual circuits on an as-needed basis throughout the data collection mission, ordinarily under several milliseconds, as data acquisition requirements dictate. The adapter also terminates all ATM as well as line interface OAM traffic. The ATM switch itself implements the space-time division multiplexing/demultiplexing function, routing the ATM cells to the appropriate destination port. Following the switch fabric, the ATM cell stream is directed to a processor which serves to perform the "ATM-to –SCSI adaptation", that is, packages the ATM data, including channel identification information appended by the processor (in effect, the complete ATM cell itself including ATM cell header with VPI/VCI) for demultiplexing on playback, along with commands in appropriate SCSI format for the storage devices (RAIDs); this same network element also ordinarily serves as the centralized system-wide OAM node. In other words, the ATM switch performs multiplexing/demultiplexing function at the ATM cell level whereas the SCSI processor terminates the ATM communications and repackages the stream of ATM cell data along with the necessary channel identification and SCSI commands for storage of the data on the RAID or SS memory. Typically, the SCSI processor is a workstation-class computer utilizing one or more host bus adapters on the ATM side and separate HBAs for the SCSI storage side; depending on the aggregate BW requirements, more than one such ATM-to-SCSI processor may be required. Also, frequently, the SCSI processor will segregate the "metadata" (directory and FAT structures required to control the RAIDs) on separate (local) drives to maximize RAID thruput. An ATM-cell based configuration utilizing FC for all (ATM as well as SCSI) communications is illustrated in Fig. 3a. An even larger configuration, which uses an additional FC switch to interconnect the RAIDs to the SCSI processor is shown in Fig 3b. Note that the PAD/NI itself performs time tagging and stores clock and other application specific information within the payload of ATM cells, transparent to both the ATM switch and SCSI processor. Note as well that from the SCSI processor through the RAIDs the system looks just like a SAN, and in fact can be managed as thus using standard SAN management software from companies such as Veritas and Tivoli (SANergy), etc. In general, the SAN management software defines one server as owner of the system metadata (i.e. the file allocation information, file security/access and directory services); this metadata is made available to all SAN-connected servers (generally over separate ethernet network) as a network volume; all actual file data is then transferred over the FC-connected SAN using the metadata obtained from metadata server. And, as in large SANs, robotic archival tape storage (such as the SONY Petasite) will be provided for backup, data transfer, and archiving. In general, the ATM-cell based configuration offers the most flexibility as to communications links, and provides the highest communications performance, featuring low latency, lowest delay jitter, and highest bandwidth utilization effectivness; it also offers the highest multiplexing/demultiplexing bandwidths for real-time applications; and it can extract the best performance from disk data storage (presuming only that the ATM-to-SCSI adaptation processors are appropriately sized so as not to throttle the storage devices) as well. This comes however at the additional cost and complexity of a high-i/o-bandwidth computer (typically a workstation) which serves to process the ATM data stream into/from SCSI format suitable for data storage/retrieval, and one or more ATM switches (however, this same processing node can conveniently serve as the system-wide OAM node, which does cut down somewhat on overall system cost). For those users that already have a suitable SAN feeding one or more high-performance servers, the cost increment can be quite small (some HBAs and ATM switch). And of course, once in ATM format, the data is easily encapsulated into IP, Frame-relay, ethernet, or other formats for transmission as required. In general, a given level of real-time data acquisition performance (aggregate data rates, numberof i/o channels, channel coherency, etc. but generally NOT latency) can be equally achieved with either network architecture using RAIDs or SS memory (but NOT tape); the potential performance penalty in using file-based SCSI-over-FC multiplexing can frequently be overcome by using a larger number of RAIDs combined with multi-block buffering and larger FC switch and comes at the expense of storage latency. At some point, the cost impact of additional RAID hardware, RAID buffer, and/or FC switches, i.e. the cost/performance penalty of file-based multiplexing and storage, or the latency to storage, may overwhelm the benefits. For other users, the convenience and flexibility of transporting or redistributing ATM data over various local and wide-area networks, as well as more cost-effective use of the RAID hardware, the higher-bandwidth links available for ATM, and the lower-latency/higher network utilization efficiencies available in such networks, will justify the ATM-switched solution.
Network Interface
w 2 x 1 Gbps FC links w/optical interface User Interface
w LVDS data interface (approx 170 Mbytes/s
sustained, 266 Mbytes/s peak in either direction)
w High-resolution (0.1/1.0 microsecond)
deterministic Time Stamping of LVDS data (non-deterministic stamping
of PCI/Mezzanine PCI data) Applications
w 70 Mhz analog (170 Mbytes/s = 1.36 Gbps)
record/replay using single PAD/NI-200
|
