DWDM in Metropolitan Area Networks
DWDM in Metropolitan Area Networks
DWDM in Metropolitan Area Networks
SONET/SDH
SONET/SDH has been the foundation for MANs over the last decade, serving as the fundamental transport layer for both TDM-based circuit switched network and most overlay data networks. While SONET/SDH has evolved into a very resilient technology, it remains fairly expensive to implement. Inherent inefficiencies in adapting data services to the voice-optimized hierarchy and an inflexible multiplexing hierarchy remain problematic. More importantly, capacity scaling limitationsOC-768 may be the practical limit of SONET/SDHand unresponsiveness to bursty IP traffic make any TDM-based technology a poor choice for the future.
ATM
Many service providers favor ATM because it can encapsulate different protocols and traffic types into a common format for transmission over a SONET infrastructure. Meanwhile the data networking world, which is overwhelmingly IP-oriented, favors packet over SONET (POS), which obviates the costly ATM
3-1
intermediate layer. Advancements in IP, combined with the scaling capacity of gigabit and multigigabit routers, make it possible to envisage an IP-based network that is well suited for carrying primarily data traffic, and secondarily voice. Nevertheless ATM remains strong in the metropolitan area. It can accommodate higher speed line interfaces and provide managed virtual circuit services while offering traffic management capabilities. Thus ATM edge devices are commonly used to terminate traffic, including VoIP, DSL, and Frame Relay.
Gigabit Ethernet
Gigabit Ethernet (GE) is a proven technology for easy migration from and integration into traditional Ethernet. It is relatively inexpensive compared to other technologies that offer the same transmission rate, but does not provide quality of service (QoS) or fault tolerance on its own. When confined to point-to-point topologies, collisions and carrier sense multiple access (CSMA) are not of concern, resulting in more effective use of the full bandwidth. Because the optical physical layer can support much longer distances than traditional Category 5 cable, Gigabit Ethernet over fiber (1000BASE-LX, for example) can be extended into the wide-area realm using DWDM. The latest advancement in Ethernet technology, 10 Gigabit Ethernet, is being driven by a need to interconnect Ethernet LANs operating at 10, 100, or 1000 Mbps. Ten Gigabit Ethernet can be used for aggregating slower access links, in the backbone networks, and for WAN access. Using 1550-nm serial lasers, distances of 40 to 80 km (25 to 50 mi) are possible with 10 Gigabit Ethernet over standard SM fiber. With such technology, service providers can build simple Ethernet networks over dark fiber without SONET or ATM and provision high-speed 10/100/1000 Mbps services at very low cost. In addition, a very short reach (VSR) OC-192 interface can be used to connect 10 Gigabit Ethernet to DWDM equipment over MM fiber. Ethernet offers the technical advantages of a proven, adaptable, reliable, and uncomplicated technology. Implementations are standard and interoperable, and cost is much less than SONET or ATM. Architecturally, Ethernets advantage is its emerging potential to serve as a scalable, end-to-end solution. Network management can also be improved by using Ethernet across the MAN and WAN.
IP
Clearly, as traditional circuit-switched services migrate to IP networks and data grows, networks must evolve to accommodate the traffic. However, IP may need to become as complex as ATM to replace its functionality. Thus, both ATM and IP are candidates for transport directly over DWDM. In either case, the result is simplified network infrastructure, lower cost due to fewer network elements and less fiber, open interfaces, increased flexibility, and stability. The question is, in which format will IP travel over an optical network: IP over ATM over SONET, IP over SONET (as POS), or IP over Gigabit Ethernet or 10 Gigabit Ethernet? (See Figure 3-1.)
3-2
OL-0884-01
Chapter 3
Figure 3-1
GE/10GE
IP ATM SONET
FC
49368
Fibre Channel
Fibre Channel is the predominant data link technology used in storage area networks (SANs). See the Storage Area Networks section on page 3-5. Fibre Channel is an economical replacement for the Small Computer System Interface (SCSI) protocol as a high-speed interface for applications such as data backup, recovery, and mirroring. Fibre Channel interfaces are available at 100 MBps today; 200 MBps interfaces should be available in the near future, and 400 MBps interfaces are in testing.
Note
By convention, transfer rates for storage are specified in megabytes per second (MBps). Fibre Channel comes without the very short distance limitations of SCSI; it also avoids the termination restrictions of SCSI because each node acts as an optical repeater. Fibre Channel can be implemented in a point-to-point, arbitrated loop, or mesh topology using a switch. As shown in Figure 3-1, Fibre Channel, like other protocols, can be carried directly over the optical layer using DWDM.
3-3
Figure 3-2
Inner ring
49369
FDDI
FDDI is at this point a legacy technology. Having served a need at one time, it has been replaced by more advanced technologies. Although FDDI is capable of scaling to the metropolitan area, it is also a shared media technology with a relatively low capacity by current standards. This limitation, along with falling availability of FDDI interfaces on network equipment, is causing FDDI to be replaced by Gigabit Ethernet, or ATM. Nevertheless, it is also a protocol that can be transparently transported over the optical layer using DWDM.
3-4
OL-0884-01
Chapter 3
FC
FC
Tape
49370
Disk array
3-5
A number of types of interfaces have been used to connect servers to devices in a SAN. The most prevalent is IBMs Enterprise System Connection (ESCON), a 17-MBps half-duplex protocol over fiber. Fibre Channel, on which IBMs FICON is based, is also frequently employed in SANs and has a much higher capacity than ESCON (see the Fibre Channel section on page 3-3.) Both technologies, however, have significant distance limitations. For example, the standard maximum distance without repeaters is around 3 km (1.9 mi) for half duplex ESCON and around 10 km (6.2 mi) for full duplex 100-MBps Fibre Channel. There is performance degradation as distances increase beyond these numbers. This distance limitation can be overcome by transporting data between one or more enterprise locations and one or more SANs over the optical layer using DWDM. In Figure 3-4, for example, the distance separating the enterprise location and the SAN sites can be greatly extended. Access to the ring is by way of satellite OADMs with Fibre Channel or ESCON interfaces at each SAN location (one of these could provide data mirroring). These interfaces can also support Sysplex Timer and Sysplex Coupling Link interfaces, used in IBM environments for distributing loads across the members of a server complex.
Figure 3-4 SAN Access over the Optical Layer
Enterprise
OADM
OADM
New York
New Jersey
In addition to overcoming distance limitations, DWDM can also reduce fiber requirements in SANs. Both ESCON and FICON require a pair of fibers for every channel. By multiplexing these channels over DWDM transport, significant savings can be realized.
3-6
49371
Primary SAN
Standby SAN
OL-0884-01
Chapter 3
Replace or upgrade the equipment; for example, from OC-48 to OC-192. Install a new ring on new or existing fiber. Install one or more new rings by deploying DWDM.
Migrating the SONET Ring to DWDMBefore
Figure 3-5
SONET
ADM
SONET
ADM
SONET
ADM
ADM
49372
SONET
The third option is shown in Figure 3-6. By using DWDM to increase the capacity of the existing ring, one fiber can essentially act as many.
3-7
Figure 3-6
SONET
ADM
SONET
ADM
SONET
ADM
ADM
ADM
SONET
SONET
In a second type of scenario, DWDM can be used to remove an entire class of equipment, the SONET ADMs. This change, which might constitute a second phase of SONET migration, allows routers and other devices to bypass SONET equipment and interface directly to DWDM, while simplifying traffic from IP/ATM/SONET to POS to eventually IP directly over the optical layer (see Figure 3-7).
3-8
49373
OL-0884-01
Chapter 3
DWDM in Metropolitan Area Networks Topologies and Protection Schemes for DWDM
Figure 3-7
OC-48
OC-12 OC-48
OC-12 OC-48
In this phase of migration, end user sites are served by OADMs rather than SONET ADMs. In this way DWDM rings and mesh networks can eliminate the increased cost and complexity of introducing more SONET elements into the network to meet demand. The advantage here for carriers is the ability to offer bit-rate-independent services, freeing them from the DS1/DS3/OC-n framework. Such a scheme would also allow enterprise LAN access to be extended across the MAN or WAN without a SONET infrastructure. A further advantage in migrating from SONET to the optical layer is that protection and restoration becomes less susceptible to failure of electronic components; a common survivability platform for all network services is created, including those without built-in protection.
49374
3-9
Point-to-Point Topologies
Point-to-point topologies can be implemented with or without OADM. These networks are characterized by ultra-high channel speeds (10 to 40 Gbps), high signal integrity and reliability, and fast path restoration. In long-haul networks, the distance between transmitter and receiver can be several hundred kilometers, and the number of amplifiers required between endpoints is typically less than 10. In the MAN, amplifiers are often not needed. Protection in point-to-point topologies can be provided in a couple of ways. In first generation equipment, redundancy is at the system level. Parallel links connect redundant systems at either end. Switchover in case of failure is the responsibility of the client equipment (a switch or router, for example), while the DWDM systems themselves just provide capacity. In second generation equipment, redundancy is at the card level. Parallel links connect single systems at either end that contain redundant transponders, multiplexers, and CPUs. Here protection has migrated to the DWDM equipment, with switching decisions under local control. One type of implementation, for example, uses a 1 + 1 protection scheme based on SONET Automatic Protection Switching (APS). See Figure 3-8.
Figure 3-8 Point-to-Point Architecture
APS
APS
OADM
APS
APS
49375
Ring Topologies
Rings are the most common architecture found in metropolitan areas and span a few tens of kilometers. The fiber ring might contain as few as four wavelength channels, and typically fewer nodes than channels. Bit rate is in the range of 622 Mbps to 10 Gbps per channel. Ring configurations can be deployed with one or more DWDM systems, supporting any-to-any traffic, or they can have a hub station and one or more OADM nodes, or satellites (see Figure 3-9). At the hub node traffic originates, is terminated and managed, and connectivity with other networks is established. At the OADM nodes, selected wavelengths are dropped and added, while the others pass through transparently (express channels). In this way, ring architectures allow nodes on the ring to provide access to network elements such as routers, switches, or servers by adding or dropping wavelength channels in the optical domain. With increase in number of OADMs, however, the signal is subject to loss and amplification can be required.
3-10
OL-0884-01
Chapter 3
DWDM in Metropolitan Area Networks Topologies and Protection Schemes for DWDM
Figure 3-9
1, 2,....n
1, 2,....n
OADM
OADM
9-12 1, 2,....n
1, 2,....n
OADM
1-4 1, 2,....n
5-8
Candidate networks for DWDM application in the metropolitan area are often already based on SONET ring structures with 1 + 1 fiber protection. Thus schemes such as Unidirectional Path Switched Ring (UPSR) or Bidirectional Line Switched Ring (BLSR) can be reused for DWDM implementations. Figure 3-10 shows a UPSR scheme with two fibers. Here, hub and nodes send on two counter-rotating rings, but the same fiber is normally being used by all equipment to receive the signal; hence the name unidirectional. If the working ring should fail, the receiving equipment switches to the other pair. Although this provides full redundancy to the path, no bandwidth reuse is possible, as the redundant fiber must always be ready to carry the working traffic. This scheme is most commonly used in access networks.
Figure 3-10 UPSR Protection on a DWDM Ring
Hub
UPSR
OADM
OADM
Other schemes, such as Bidirectional Line Switched Ring (BLSR), allow traffic to travel from the sending to the receiving node by the most direct route. Because of this, BLSR is considered preferable for core SONET networks, especially when implemented with four fibers, which offers complete redundancy.
49377
49376
3-11
Mesh Topologies
Mesh architectures are the future of optical networks. As networks evolve, rings and point-to-point architectures will still have a place, but mesh promises to be the most robust topology. This development will be enabled by the introduction of configurable optical cross-connects and switches that will in some cases replace and in other cases supplement fixed DWDM devices. From a design standpoint, there is a graceful evolutionary path available from point-to-point to mesh topologies. By beginning with point-to-point links, equipped with OADM nodes at the outset for flexibility, and subsequently interconnecting them, the network can evolve into a mesh without a complete redesign. Additionally, mesh and ring topologies can be joined by point-to-point links (see Figure 3-11).
Figure 3-11 Mesh, Point-to-Point, and Ring Architectures
DWDM mesh networks, consisting of interconnected all-optical nodes, will require the next generation of protection. Where previous protection schemes relied upon redundancy at the system, card, or fiber level, redundancy will now migrate to the wavelength level. This means, among other things, that a data channel might change wavelengths as it makes its way through the network, due either to routing or to a switch in wavelength because of a fault. The situation is analogous to that of a virtual circuit through an ATM cloud, which can experience changes in its virtual path identifier (VPI)/virtual channel identifier (VCI) values at switching points. In optical networks, this concept is sometimes called a light path. Mesh networks will therefore require a high degree of intelligence to perform the functions of protection and bandwidth management, including fiber and wavelength switching. The benefits in flexibility and efficiency, however, are potentially great. Fiber usage, which can be low in ring solutions because of the requirement for protection fibers on each ring, can be improved in a mesh design. Protection and restoration can be based on shared paths, thereby requiring fewer fiber pairs for the same amount of traffic and not wasting unused wavelengths. Finally, mesh networks will be highly dependent upon software for management. A protocol based on Multiprotocol Label Switching (MPLS) is under development to support routed paths through an all-optical network. In addition, network management will require an as-yet unstandardized channel to carry messages among the network elements.
3-12
OL-0884-01
49378
Chapter 3
Is the DWDM system compatible with existing fiber plant? As discussed in the Optical Fibers section on page 2-5, some types of older fiber are not suitable for DWDM use, while newer types, such as NZ-DSF, are optimized for DWDM. Standard SM fiber (G. 652), which currently accounts for the majority of installed fiber, can support DWDM in the metropolitan area. If new fiber must be laid, a type should be chosen that will allow for future growth, particularly as DWDM systems expand into new wavelength regions and higher bit rates.
What is my migration and provisioning strategy? Because DWDM is capable of supporting massive growth in bandwidth demands over time without forklift upgrades, it represents a long-term investment. As discussed in the Mesh Topologies section on page 3-12, both point-to-point and ring topologies can serve as foundations for future growth into mesh topologies. Planning should allow for flexible additions of nodes, such as OADMs, to meet the changing demands of customers and usage.
What network management tools can I use? A comprehensive network management tool will be needed for provisioning, performance monitoring, fault identification and isolation, and remedial action. Such a tool should be standards-based (SNMP, for example) and be able to interoperate with the existing operating system.
What is my strategy for protection and restoration? Designing a protection strategy is a complex process that must take into account many variables. There are hard failures (equipment failures, such as laser or photodetector, and fiber breaks) and soft failures such as signal degradation (for example, unacceptable BER). The former must be addressed through redundancy at the device, component, or fiber level. The latter must be addressed by the system through intelligent wavelength monitoring and management. Protection and survivability strategies depend upon service type, system, and network architectures. In many networks, they also depend on the transport protocol.
Two additional and important considerations are calculation of the optical power budget and interoperability.
Length of the fiber Attenuation in dB of the whole link, as well as the attenuation of separate sections of the fiber (if any)
3-13
Attenuation characteristics of the fiber itself Locations of connectors, joints, and faults in the cable
The goal in calculating optical loss is to ensure that the total loss does not exceed the span budget. The following are typical values for the main elements in a span:
Connector splice lossThis is 0.2 dB if the connectors are modern single-mode connectors from the same vendor. If the manufacturers of the two connectors (halves of the connection) are different, then the average loss is 0.35 dB. Fiber lossThis is about 0.25 dB/km due to attenuation. Fiber agingThis is about 2 dB over the life of the system.
Because optical power loss (or gain) is measured in a logarithmic value, decibels (dBs), the combined effect of all contributing elements can be calculated using simple addition. Assume a span budget of 25 dBm (a dBm is the signal power level in relation to one milliwatt), we can make the following addition: total system loss + (fiber length *.25) + fiber aging margin + connector/splice losses If the sum is less than 25, then we are within the span budget. If not, then some changes must be made. This might include adding an amplifier or reducing the number of loss-inducing elements on the span. Fiber conditioning, which includes resplicing fiber, connector cleaning, and so on, may also be required to reduce loss. It is also important to ensure that the client side or tributary equipment does not overdrive the local receive laser of the DWDM equipment. This means that the client or tributary equipment must operate within the specifications of the DWDM client interface. Although it is generally not an issue at the distances used in the MAN, remember that optical amplifiers boost the entire input, including noise. Thus over time the signal-to-noise ratio becomes so high that a clear signal can no longer be detected at the receiving end. At this point regenerators must be used to perform the 3R functions.
Interoperability Issues
Because DWDM uses specific wavelengths for transmission, the wavelengths used must be the same on either end of any given connection. Toward this end, the ITU has standardized on a grid with spacings of 100 GHz (see Table 2-1 on page 2-14). However, vendors may use wider spacing, sometimes at 200 GHz, or narrower. In addition, different vendors who do use the same grid may not use the same lambda numbering scheme. That is, lambda 1 on vendor As equipment may be assigned a different wavelength from lambda 1 on vendor Bs equipment. Hence, it is important to be aware of the potential interoperability problems posed by different grid alignments. Other interoperability issues include power levels, inter- and intra-channel isolation, PMD tolerances, and fiber types. All these contribute to the challenges of transmission between different systems at Layer 1.
Future of DWDM
DWDM will continue to provide the bandwidth for large amounts of data. In fact, the capacity of systems will grow as technologies advance that allow closer spacing, and therefore higher numbers, of wavelengths. But DWDM is also moving beyond transport to become the basis of all-optical networking
3-14
OL-0884-01
Chapter 3
with wavelength provisioning and mesh-based protection. Switching at the photonic layer will enable this evolution, as will the routing protocols that allow light paths to traverse the network in much the same way as virtual circuits do today. These and other advances are converging such that an all-optical infrastructure can be envisioned. Figure 3-12 shows an example of such an infrastructure, using mesh, ring, and point-to-point topologies at the optical layer to support the needs of enterprise, metropolitan access, and metropolitan core networks.
Figure 3-12 Next Generation Metropolitan Optical Network
POP Enterprise Residential access Data center To/from national backbone
DWDM rings
DWDM mesh
DWDM bus
49379
3-15
3-16
OL-0884-01