Synchronous optical networks (SONET) currently form the backbone of long haul and MAN scale telecommunication networks. However, inherently asynchronous Ethernet systems have recently penetrated into metropolitan area network design as a technological alternative to traditional SONET-based infrastructures. Ethernet is a commonly used name for a network access protocol referred to as carrier sense multiple access with collision detection (CSMA/CD). The CSMA/CD protocol is defined in ANSI/IEEE Standard 802.3, 2002 edition, published by the Institute of Electrical and Electronics Engineers, Inc., 345 East 45th Street, New York, N.Y. 10017. Advances in switching and transmission technologies combined with the introduction of 10 gigabits per second (Gb/s) Ethernet systems, have enabled carriers to offer telecommunications services at significantly lower cost compared with SONET.
SONET technology is a link-layer protocol designed for aggregation and transport of constant bit-rate (CBR) traffic over a high-bandwidth optical fiber. Further background information is found in J. Babcock, “Sonet: A Practical Perspective,” Business Communication Review, vol. 20, no. 9, pp. 59-63, September 1990, which is incorporated by reference herein in its entirety. When SONET was designed, much emphasis was put into the ability to transmit data in a synchronous manner, as the dominant use for SONET was in providing links for time division multiplexing (TDM) networks, which require that one segment of a network be able to slave its clock to another.
Voice telecommunication preferentially employs TDM in order to ensure perceived continuity at the receiving end of the transmission. While appropriate for voice traffic, the use of SONET in non-voice data traffic results in poor resource utilization and high equipment and operation costs, as such continuity in delivery generally is not required. Further background information is found in A. Chapman and H. T. Kung, “Enhancing Transport Networks with Internet Protocols,” IEEE Communications Magazine, vol. 36, no. 5, pages 100 and 101, May 1998, which is incorporated by reference herein in its entirety. Increasing numbers of non-voice data applications along with growing bandwidth demands and increased competition have forced carriers to challenge the traditional “dumb-pipe” MAN model and to seek lower-cost solutions.
In its evolution, Ethernet has recently reached the stage where it can be expanded from local enterprise to carrier-class networks. Three factors have enabled this expanded utilization of Ethernet technology. First, the introduction of long-haul optics has enabled Ethernet systems to reach distances of 40 kilometers (km) and higher. Second, advanced switching techniques, including multiprotocol label switching (MPLS), virtual local area networks (VLANs) and per-flow queuing, allow carriers to provide services bundled with proper security, reliability and quality of service (QoS). The latter is a measure of prioritization of telecommunications to the most time-sensitive transmissions, for example, voice communications. Finally, the introduction of 10 Gb/s Ethernet systems has provided the necessary bandwidth capacity.
From a clocking perspective, Ethernet networks are considered asynchronous, in the sense that each line card in an Ethernet switch is clocked from an independent and typically low-accuracy clock source. This clocking approach is in contrast to SONET, where all line cards are synchronized to a single master clock source, usually a port on the switch configured to be a master, and an entire network is synchronized to a primary reference source (PRS), for example an extremely accurate atomic clock. When SONET is used as a transport infrastructure for interconnecting two circuit-switched networks over a TDM link such as a T1 or T3 line, it is possible to establish a clock distribution hierarchy so that all elements in the network are synchronized to one primary clock source. The synchronization requirement comes from the telephone switch design. Specifically, the buffers implemented in telephone circuit switches are typically small and are used only to make up for the phase difference and small variations in clock frequency, creating a requirement for accurate timing synchronization across the network. Since SONET was originally designed as a transport technology for circuit switched networks, much attention in SONET design was put into addressing the synchronization issues.
On the other hand, conventional Ethernet switches are designed for non-voice data traffic, which is asynchronous by nature and eliminates the need for preserving the timing across the different line cards and switches. Indeed, the lack of synchronization requirements is one of the major factors that makes Ethernet switching equipment cheaper and simpler than SONET equipment. In view of the transition of network applications from voice to non-voice as demands for information transmission bandwidth steadily expand, as well as the ever-present market pressure to reduce operating costs, carriers have started implementing MANs based on Ethernet technology.
However, as new Ethernet-based infrastructures proliferate at the MAN carrier level, problems can arise with supporting legacy applications, such as interconnecting two TDM-based networks using an Ethernet MAN. For example, Ethernet-based carriers will desire to offer Ethernet transport services to individual enterprise customers. It should be possible to deliver such services at significantly lower cost than a traditional SONET-based carrier can offer. However, one important application for such a customer is likely to be an existing private branch exchange (PBX) telephone switch. PBX systems require a circuit-switched service such as a T1 connection, with the ability to slave its clock to the public network. Transferring that customer's business from SONET to an Ethernet system will practically require either facilitating continued operation of the PBX switch over the public network by making it compatible with Ethernet technology, or replacing the switch. The costs of meeting such requirements illustrate an important motivation for designing approaches to provide a clear-channel circuit emulation over an Ethernet-based packet switched network, with clock frequency synchronization being the critical feature of such a service. Published work and the efforts in relevant standards organizations and consortiums, such as the Internet engineering task force (IETF) and metro Ethernet forum (MEF), have specified the synchronization problem, but have offered very little or no insight into its solution. Much of the previous work has been focused on packetization techniques.
Timing synchronization has a wide range of applications, and the related concepts have been researched for several decades. Prior work that has been done includes synchronization in circuit switched networks, circuit emulation over asynchronous transfer mode (ATM), circuit emulation over Internet protocol (IP), and computer time synchronization.
Circuit switched networks evolved from a network of analog switches, which required no synchronization, to networks composed of synchronous digital switches interconnected by a synchronous digital hierarchy (SDH) of links. Further background information is found in S. Bregni, “A Historical Perspective on Telecommunications Network Synchronization,” IEEE Communications Magazine, vol. 36, no. 6, pp. 158-166, June 1998, which is incorporated by reference herein in its entirety. The need for synchronization arose with the introduction of digital switches, which required that the frames arriving from different links be aligned in time in order to perform slot interchanging. With increased volume of data traffic, frequent slips in slot positioning became intolerable, imposing strict requirements on clock synchronization.
Today, all public circuit switched networks implement an approach employing a small number of highly accurate PRSs to which an entire network is synchronized through a hierarchical clock distribution topology. Besides the hierarchical master-slave distribution, it is also possible to implement mutual synchronization and a hybrid of the two, but due to their complexity these approaches have not been used except in certain special applications, such as military networks. Further background information is found in: J. E. Abate, E. W. Butterline, R. A. Carley, P. Greendyk, A. M. Montenegro, C. D. Near, S. H. Richman, and G. P. Zampetti, “AT&T's New Approach to the Synchronization of Telecommunication Networks,” IEEE Communications Magazine, vol. 27, no. 4, pp. 35-45, April 1989; and D. Mitra, “Network Synchronization: Analysis of Hybrid of Master-Slave and Mutual Synchronization,” IEEE Transactions on Communications, vol. 28, no. 8, pp. 1245-1259, August 1980; both of which are incorporated by reference herein in their entirety.
In a central office, the master clock is extracted directly from one of the links that connect the office to the outside network, which is usually a feed from a source on a higher clock stratum level. The Stratum 3 standard, for example, requires that during the first 24 hours the clock may not drift by more than 3.7×10−7 relative to the last synchronized frequency, while any deviation for any reason may not be higher than 4.6 parts per million (ppm) of the nominal frequency. Further background information is found in American National Standard Institute, Synchronization Interface Standard, February 1994, ANSI T1.101-1994, which is incorporated by reference herein in its entirety.
Circuit emulation over ATM is an example of using a packet switched network to transport TDM traffic. TDM frames are packetized and transmitted over an ATM network using the asynchronous adaptation layer 1 (AAL-1). Further background information is found in International Telecommunications Union, B-ISDN ATM Adaptation Layer Specification: Type 1 AAL, August 1996, ITU-T I.363, which is incorporated by reference herein in its entirety. Circuit emulation in this environment differs from typical Ethernet systems in two fundamental respects. First, the ATM normally runs over SONET, which can be used as the basis for equipment synchronization. Second, ATM uses short, fixed-size cells, which make network jitter more controllable than in Ethernet networks, allowing for simple clock recovery systems.
The standardized clock recovery method in AAL-1 is the synchronous residual time stamps (SRTS) method, which relies on measuring the length of a time period defined by an equipment clock, using the network clock as a reference. Further background information is found in R. C. Lau and P. E. Fleischer, “Synchronous Techniques for Timing Recovery in BISDN,” IEEE Transactions on Communications, 1995, which is incorporated by reference herein in its entirety. If the equipment clock were synchronized with the network clock, a nominal measurement would result. In the SRTS method, the difference between the nominal and actual measurement is transmitted to the slave. Assuming that the slave has access to the network clock and knowledge about the nominal period, the residual time stamp is sufficient to recover the master clock. This method is highly accurate, but assumes that ATM cells are transmitted over a synchronous network and that both the master and the slave have access to the network clock. Since ATM normally assumes the use of SONET for a physical layer, this assumption is valid for ATM, but not for Ethernet.
Adaptive clocking in AAL-1 is an alternative to the SRTS method, which does not rely on the SONET clock for synchronization. Instead, the timing is recovered by averaging the cell inter-arrival time over a long period of time, either by directly measuring the inter-arrival time or by adapting the local clock to maintain a constant receiving buffer level. This method can be used as a starting point for deriving a synchronization system for circuit emulation over Ethernet, except that the problem is much easier to handle in the ATM case, as the jitter in cell-switched networks is more controllable than in packet-switched networks. Because of the availability of the more accurate and predominantly used SRTS method, ATM equipment vendors never had the motivation to implement an accurate adaptive clocking system.
The problem of circuit emulation over IP or Ethernet is unique in the sense that there is no alternative path for receiving the clock information. The underlying transport network is fully asynchronous, so it is not possible to rely on it for clock distribution as the ATM is able to rely on SONET links. Compared to ATM, adaptive clocking is the only option here and the problem is even harder because of the variable size packets.
There are many real-time applications and protocols implemented over IP, such as the real time protocol (RTP) and various application-specific protocols for streaming multimedia applications. Further background information is found in H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: A Transport Protocol for Real-Time Applications,” IETF RFC1889, January 1996, which is incorporated by reference herein in its entirety. However, there is a fundamental difference between them and the circuit emulation required for Ethernet MAN applications. Streaming protocols reconstruct the data stream by doing the playout with reference to the receiver local clock, which is typically a low-cost free-running clock whose accuracy is not very high. The playout using a local clock works well for human perception of a real-time stream, such as, for example, human perception of audio or video, but it is an inadequate solution if the receiver is a piece of equipment, such as a PBX for example, that requires accurate tracking of the clock on the remote side. Even in applications that connect to the synchronous TDM network, such as voice over IP (VoIP) gateways, the clock indications are not transported over the asynchronous network. Instead, both gateways are synchronized to the TDM network side assuming that the TDM subnets are not fully isolated and a common clock distribution path still exists.
Some early results on providing TDM services over IP are found in R. Noro, M. Hamdi, and J. P. Hubaux, “Circuit Emulation over IP Networks,” in Proceedings of Protocols for High-Speed Networks VI, Salem, Mass., August 1999, pp. 187-201, which is incorporated by reference herein in its entirety. This article describes the use of time stamps embedded in the packets to transmit clock time indications from a master to the slave. If the clocks were perfectly synchronized and if the network delay were constant, the slave timestamps would be a linear function of master time stamps. Using this observation as a foundation, a clock recovery system estimates parameters of a linear function by minimizing the mean square error over some predefined time window and directly adjusts the slave clock using the estimated linear dependency. After some number of iterations, the slave clock time converges to the master. The results of these early findings by Noro, et al. are of background relevance, but the performance of the systems presented by Noro is insufficient to be successfully used in practical applications.
The time to converge, computational complexity and residual error all are important performance indicators. Standards compliance criteria must also be considered as an important indicator of practical value of the system. Further background information is found in: International Telecommunication Union (ITU), Telecommunication Standardization Sector, The Control of Jitter and Wonder within Digital Networks which are Based on the 1544 kbits/s Hierarchy, March 2000, ITU-T Recommendation G.824; and International Telecommunication Union (ITU), Telecommunication Standardization Sector, The Control of Jitter and Wander Within Digital Networks Which are Based on the Synchronous Digital Hierarchy (SDH), March 2000, ITU-T Recommendation G.825; both of which are incorporated by reference herein in their entirety.
Using time stamped packets for synchronization is not limited to circuit emulation. Many distributed computing applications require that the time, and in some cases the frequency, be synchronized among the nodes. The network time protocol (NTP) has been established as a standard mechanism for computer time synchronization. Further background information is found in D. L. Mills, “Improved Algorithms for Synchronizing Computer Network Clocks,” IEEE/ACM Transactions on Networking, vol. 3, no. 3, pp. 245-254, June 1995 and D. L. Mills, “Adaptive Hybrid Clock Discipline Algorithm for the Network Time Protocol,” IEEE/ACM Transactions on Networking, vol. 6, no. 5, pp. 505-514, October 1998, both of which are incorporated by reference herein in their entirety. In the NTP protocol, the slave initiates a request by sending a timestamped packet to the master. The master responds with a packet that carries the original slave timestamp and two additional timestamps representing the time of arrival of the request and the time when the response was sent. A fourth timestamp is the time when the response arrives back at the slave. The four timestamps are then used to adjust the slave time, with a very low computational burden.
The NTP protocol has been conventionally used to establish computer times, and many of its features have been designed to support that function. In conventional NTP used to synchronize computer time, three timestamps are recorded into each packet. Conventional NTP also employs a form of adaptive loop filter. Further, conventional NTP employs clock filtering, in other words, a system that tries to evaluate multiple clock sources and select those that are most trustworthy from the standpoint of their accuracy. For example, if one clock differs too much from the others, the NTP protocol can drop it from the list of sources used to calculate the slave clock frequency.
While efficient in synchronizing computer wall clock time to a few milliseconds accuracy, a weakness of conventional NTP is its slow convergence and its sensitivity to the propagation time. Further background information is found in D. L. Mills, “On the Accuracy and Stability of Clocks Synchronized by the Network Time Protocol in the Internet System,” ACM Computer Communications Review, vol. 20, no. 1, pp. 65-75, January 1990, which is incorporated by reference herein in its entirety. The timestamps must be exchanged over a long time relative to the reference clock period, so that the actual phase and frequency error can become significantly larger than the noise introduced by side effects.