1. Introduction

Emergence of mega data centers in near future imposes a big challenge on traditional electrical data center networks (DCNs). A mega data center [1] will contain more than 10⁵ server racks [2] distributed over 10⁵ square feet [3]. Sometimes, each pair of racks may generate a lot of traffic needed to be transmitted [4], though the bulk of communication typically changes very slowly [5]. To support nonblocking interconnection for so many racks, different kinds of electrical DCNs, such as folded Clos [6] and FatTrees [7], have been deployed. These new designs can provide full bisection bandwidth for each pair of racks, but would require a large number of long cables and switches, which remarkably increase the network cost and power consumption, and thus become the development obstacle of data centers. Hence, to meet the requirement of mega data centers, it is very important to design a scalable DCN architecture that can provide nonblocking interconnection with a low complexity.

An interesting effort in recent years is so-called hybrid electrical/optical DCN architecture, e.g., Helios [8] and c-Through [9], which employs the nonblocking electrical packet switching network together with a nonblocking optical circuit switching (OCS) network. Compared to the electrical network, the optical network can provide much higher bandwidth, but requires much longer reconfiguration time. Thus, the electrical network is used to deal with fine-granularity but delay sensitive traffic, while the optical network is employed to clean up the hotspot traffic which is typically delay insensitive and slowly varying. Because of the employment of high-speed optical network, the network complexity, in terms of number of needed switches, number of long cables, and power consumption, can be remarkably reduced. The simulation results in [8] confirm that a factor of 3 reduction in cost, a factor of 6.5 reduction in number of cables, and a factor of 9 reduction in power consumption can be achieved by the introduction of optical networks.

However, the scalability of the nonblocking OCS network in the hybrid DCN architecture is still a big challenge. In [8] and [9], the micro-electro-mechanical-system (MEMS) based crossbar was used to perform rack-to-rack non-blocking optical circuit switching. Constrained by the port count of the most updated MEMS that is 320 in practice or 1000 in Labs [10], such architecture is not applicable in the scenario of mega data centers which contains more than 10⁵ server racks. What’s more, if the number of racks is large, a large number of long fibers are still needed to connect the aggregation top-of-rack (ToR) switches to the MEMS. In [11], Porter et al. proposed a reconfigurable optical add-drop multiplexers (ROADM) ring based switch, in which a collection of optical couplers (OCs) and wavelength-selective switches (WSSs) are combined alternatively to form ROADM stations, connecting together by a fiber ring. In this architecture, each input is associated with a specific wavelength, and each output can connect to that input if it is tuned to the wavelength that is associated with that input. Though this design looks interesting, the port count of the ring-based crossbar is limited by the number of available non-interfering wavelengths. To improve the scalability, Chen et al. in [12] proposed a WSS-based optical network called WaveCube, of which the topology is an n-dimensional cube. WaveCube can support tens of thousands of ToR switches, and has a cabling complexity 2-3 orders smaller than that of a Fat-tree. However, it could only provide 70% - 85% bisection bandwidth of a non-blocking network with a complex route and wavelength assignment algorithm. Furthermore, WaveCube needs hop-by-hop electrical relays, which increase not only queuing delay of the traffic but also the switching overhead of electrical ToR switches. Another kind of optical network called OvS network was constructed in [2,13 ], based on flattened butterfly topology [14]. This design possesses the properties similar to the WaveCube, except that end-to-end communications require at most one electrical relay. Kitayama et al. in [15] proposed to improve the scalability using optical packet switching technology, which however will not be commercially available in the near future. Thus, the design of scalable optical networks to provide network-level nonblocking interconnections for all the ToR switches in the mega data centers remains an open issue.

To deal with the slow-varying bulk traffic in mega data centers, this paper studies multiple ROADM rings based OCS networks, which are essentially the combination of MEMS crossbars and ROADM rings proposed in [11,16 ]. This type of optical network is designed according to the idea of classical Clos networks [17], such that the size of each switching element can be small though the number of server racks is large. Also, this kind of network possesses another important advantage that the number of long optical cables can be further reduced since the ring-based crossbars can be arbitrarily stretched to pass through the areas that ToR clusters are installed such that each rack could connect to the OCS network through short fiber cables.

Two network structures are considered: ring-MEMS-ring (RMR) network and MEMS-ring-MEMS (MRM) network. In the RMR network, the ring-based crossbars constitutes the input stage and output stage and the MEMS crossbars associated with different wavelengths form the central stage. In the MRM network, the ring-based crossbars in the central stage are sandwiched by the MEMS crossbars in the input stage and output stage. We show that the MRM network is better than the RMR network, since the MRM network does not need tunable lasers to achieve nonblocking routing.

2. Preliminary

In this section, we introduce structure and features of a ring-based switch proposed in [11], since it is the building block of our designs. We show that the switching network directly constructed from the ring-based switches is not a good idea to build a large scale switching network, which actually motivates our work in Section 3 and 4.

2.1 Ring-based switches

Figure 1 displays two ring-based switches including a legacy ROADM ring in Fig. 1(a), an input-output ring in Fig. 1(b). The legacy ring in total contains n ROADM nodes comprising a 1 × 2 OC and a 1 × 2 WSS to add, drop or pass the signals simultaneously. Herein, the WSS has much longer reconfiguration time (on the order of ms) than that in Ref [16], because the OCS networks proposed in this paper are specifically designed for the slow-varying bulk traffic [18] in datacenter networks.

 

Fig. 1 The ring-based switches: (a) legacy ROADM ring, (b) input-output ring.

Download Full Size | PDF

Such add/drop form is not favored when illustrating the input to output relation. To clarify this, we propose the input-output ring by rearranging all the add ports on one side and all the drop ports on the opposite side. In the as-proposed ring, this modification mainly means adding ports become inputs, and dropping ports become outputs correspondingly.

For both above-mentioned rings, we label the ToR switch by a pair (i, x) if it connects to the xth input of the ith OC, where $i=1,2,\cdots ,n$ is the cluster address, and x = 1 is the local address. OCs and WSSs can be replaced with larger ports for less configuration but here we only use 2 ports as an example. These ring-based architectures build the foundation of our designs.

The connection relations of these ring-based switches are equivalent. They follow the same principle that each wavelength used by one input will not be used by any other inputs. Thus, the ring operates as a $n\times m$ cross-bar in the optical domain, and it connects one of the inputs to one of the outputs at a time. Though the ring can perform more switching functions such as optical multicast and broadcast [11], we only consider the circuit unicast in this paper.

The legacy ring has an attractive advantage that it can be arbitrarily stretched to cover the large areas that the ToR switches can connect to the core switch using short optical fibers. However, a single ring-based switch is not directly applicable to mega DCN. The port count of the ring-based switch is mainly determined by the number wavelengths in the optical fiber. The C + L band of the optical fiber only contains 160 wavelength channels, which is much smaller than the number of ToR switches in the mega DCN. Thus, it is necessary to explore multi-stage switching networks based on the ring-based switches.

2.2 RRR switching networks

In this part, we consider a three-stage network, which is completely constructed from the ROADM rings. In this network, there are n $n\times m$ rings in the input stage, in the central stage, and in the output stage, respectively. There is exactly one link between the rings in two adjacent stages. We refer to such network as RRR network.

Compared with a Clos network, the RRR switching network has the same topology but different nonblocking condition. In the Clos network, the route assignment is nonblocking if two connections that share the same input or output module do not pass through the same central module. In the RRR network, there is an additional wavelength constraint that two connections that share the same ring cannot use the same wavelength. It follows that the routing algorithm of the Clos network cannot be directly applied to the RRR network. To achieve nonblocking condition, the RRR network may have to pay more cost than the Clos network.

Motivated by this observation, we will construct the Clos networks by a set of ring-based switches and MEMS-based switches in Section 3 and 4. Our idea is to combine the advantages of these two kinds of switching modules, such that the proposed networks not only have no wavelength contention but also can avoid the usage of a number of long fibers.

3. Ring-MEMS-Ring (RMR) switching network

In this section, we consider a hybrid switching network, which consists of ring-based switching modules and MEMS-based switching modules. The major challenge of the network design is to avoid the wavelength contention that may be induced by ring-based switching modules.

To achieve this goal, we construct an RMR network by replacing the ring-based switching modules in the central stage of the RRR network by the MEMS-based switching modules, each of which is associated with a wavelength. We show that the one-to-one correspondence between the MEMSs and the wavelengths by nature eliminates wavelength contentions in the ring-based switching modules in the input and output stages.

In section 3.1, the logical topology of the RMR network is introduced. In section 3.2, we demonstrate the equivalence between the RMR network and a Clos network, and investigate the nonblocking condition. In section 3.3, the network implementation is considered, then we explain how RMR network can avoid the usage of a large number of long fibers.

3.1 Logical topology

A symmetric $N\times N$ RMR network is plotted in Fig. 2 In this network, there are r $n\times m$ ring-based switching modules in the input stage, m $r\times r$ MEMS switching modules in the central stage, and r $m\times n$ ring-based switching modules in the output stage, which means $N=rn$. We denote this hybrid network as $H_1\left(n,r,m\right)$.

 

Fig. 2 Logical Topology of an $N\times N$ RMR switching network $H_1\left(n,r,m\right)$.

Download Full Size | PDF

In the input stage, each ring contains s $\left(t+1\right)\times 1$ OCs and a $1\times (m+1)$ WSS, and is associated with the wavelength set $\text{Λ}=\left\{{\lambda }_1,{\lambda }_2,\cdots , {\lambda }_m\right\}$, where $st=n$. Each ring carries n ToR switches via the input ports of the OCs. We label the ToR switch by a triple $\left(a,i,x\right)$ if it connects to the xth input of the ith OC of the ath ring, where $a=1,2,\cdots ,r$ is the module address, $i=1,2,\cdots ,s$ is the cluster address, and $x=1,2,\cdots ,t$ is the local address. The WSS is fixed such that its kth output is associated with wavelength ${\lambda }_k$, though a WSS itself is a reconfigurable device.

In the output stage, each ring contains s $1\times (t+1)$ WSSs and a $\left(m+1\right)\times 1$ OC, and is associated with a wavelength set $\text{Λ}=\left\{{\lambda }_1,{\lambda }_2,\cdots , {\lambda }_m\right\}$. Each ring carries n ToR switches via the outputs of the WSSs. In particular, the yth port of the jth WSS of the bth ring connects to ToR switch $\left(b,j,y\right)$, where $b=1,2,\cdots ,r$, $j=1,2,\cdots ,s$, and $y=1,2,\cdots ,t$. Unlike that in the input stage, each WSS in the output stage is tunable, such that its each output can receive any one of the wavelengths in the set $\text{Λ}$.

There is exactly one link between two adjacent stages. In particular, the output k of the ath input module connects to input a of the k th central module, and output b of the k th central module connects to input k of the bth output module, where $k=1,2,\cdots ,m$. It follows the strategy that the k th central module is associated with a specific wavelength ${\lambda }_k$, which means all the connections at wavelength ${\lambda }_k$ will be switched by the k th central module. This does not incur wavelength contentions in the central modules since the MEMS is wavelength insensitive.

3.2 Non-blocking condition

Due to the one-to-one correspondence between the central modules and the wavelengths, the RMR switching network has the nonblocking condition similar to that of the Clos network. The constraint that two connections originated from the same input ring module or destined for the same output ring module cannot use the same wavelength is equivalent to the constraint that they cannot go through the same central MEMS module. In other words, if the route assignment is nonblocking, the network will be wavelength contention-free. An example is illustrated in Fig. 3(a) . Since the connections $c_1,c_2,c_3,c_4$ share the first input ring module, they should use different wavelengths ${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4$, and thus pass through different central modules. Similarly, the four central modules forward the first output ring module four connections $c_1,c_2,c_{11},c_8$, which are of course at different wavelengths.

 

Fig. 3 A $12\times 12$ RMR switching network $H_1\left(4,3,4\right)$, where $\text{Λ}=\left\{{\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4\right\}$.

Download Full Size | PDF

Thus, the wavelength and route assignment in the RMR network can be modelled as the coloring of a bipartite graph $G\left(V\cup U, E\right)$, where each vertex in V represents an input module, each vertex in U is corresponding to an output module, and each edge stands for a connection request from an input module to an output module. The fact that two connections in contention cannot share the same central module or the same wavelength is corresponding to the constraint that two edges share the same vertex cannot be colored by the same color. This correspondence is illustrated in Fig. 3(b). It follows from the theory of Clos networks [17] that the nonblocking condition of the RMR network $H_1$ is $m\ge n$.

Once the wavelength or the central module is selected according to the coloring of G, the path from the source ToR switch to the destination ToR switch is determined. For example, Fig. 2 shows the route of the connection from ToR switch $\left(a,i,x\right)$ to ToR switch $\left(b,j,y\right)$, if the wavelength ${\lambda }_k$ or the central module k is assigned to this connection. It is clear that each ToR switch should equip with a tunable laser, in order to perform the wavelength assignment or the central module assignment.

3.3 Implementations

In practice, the RMR network can be implemented in a folded form to reduce the cabling complexity. In the following, we take the network in Fig. 3 as an example, and illustrate how it can be reconfigured to the network in Fig. 4 , such that the usage of long cables can be avoided.

 

Fig. 4 Implementations of an $N\times N$ RMR switching network.

Download Full Size | PDF

In general, an input ring and an output ring will carry the same set of ToR switches, if they have the same module number. For example, in Fig. 3, input ring 1 and output ring 1 carry the ToR switches labelled by $\left(1,1,1\right),\left(1,1,2\right),\left(1,2,1\right),\left(1,2,2\right)$. Hence, we merge the input ring with the corresponding output ring together as follows. For the ath input ring and the a th output ring, we combine the ith $(t+1)\times 1$ OC of the input ring with the ith $1\times (t+1)$ WSS of the output ring to form a station, labeled as $T_a\left(i\right)$, that is attached by a cluster of t ToR switches, and combine the $1\times (m+1)$ WSS of the input ring with the $(m+1)\times 1$ OC of the output ring to form a switching point, from which the merged ring connects with m MEMSs. As an instance, the folded version of the network in Fig. 3 is plotted in Fig. 4(a). In this example, the connection $c_7$ at wavelength ${\lambda }_3$ originates from the second merged ring is switched by the third MEMS to the third merged ring.

Also, it is possible to improve the RMR network in terms of cost and optical loss. Firstly, each $1\times (m+1)$ WSS can be replaced by a $1\times m$ arrayed waveguide grating (AWG) because the $1\times (m+1)$ WSS is fixed to wavelengths and functionally equivalent to a $1\times m$ AWG. Such replacement can cut down the network cost. Similarly, each $(m+1)\times 1$ OC can be replaced by an $m\times 1$ AWG, to reduce the optical loss. Also, the optical fiber between each $1\times (m+1)$ WSS and $(m+1)\times 1$ OC actually does not carry any signal, since all the signals have been dropped by the $1\times m$ WSS and fed to the MEMSs. Thus, this optical fiber can be removed. After these steps, we obtain the network in Fig. 4(b).

In general, to provide nonblocking connection for N ToRs, the RMR network illustrated in Fig. 4(b) should equip with n r × r MEMSs, r 1 × n AWGs, r 1 × n WSSs and N tunable lasers, where N = nr. Furthermore, to compensate the optical loss induced by the coupler and WSS, optical amplifiers should be installed along the ring to amplify the signals of all the wavelength channels at the same time. It follows that the cost introduced by optical amplifiers is not high since it can be amortized over all the wavelength channels. The entire RMR network needs N/t optical amplifiers in addition, which is only 1/t of that of other WSS based networks [2,13 ].

The advantage of the RMR network is it can remarkably reduce the number of long cables in the network, which can be explained as follows. In the RMR network illustrated in Fig. 4(b), all the MEMSs can be centralized in the same place, while different rings can be stretched to cover different areas, such that the ToR switches installed in different buildings can attach to the rings via short optical fibers. According to the description in Section 3.1, each ring in the RMR network has n/t + 1 segments, and thus the number of long cables in the entire network is O(N/t), which is only 1/t of that of Clos networks. Currently, the port count of the commercial MEMSs r is 320 [10], and each ring can carry n = 160 wavelengths [2]. Thus, the proposed RMR network can support N = 51200 racks at most.

The main drawback of this design is that each ToR switch should be equipped with a tunable laser, of which the tuning range should be equal to $\left|\text{Λ}\right|=n$. Therefore, the cost of the RMR switching network is high. To reduce the cost, we propose another kind of network in the next section.

4. MEMS-Ring-MEMS (MRM) switching network

In this section, our goal is to design a hybrid switching network to avoid using the tunable lasers. To achieve this goal, we use the MEMS modules in the input and output stages, and the ring modules in the central stage, which yields an MRM switching network. In this network, the dependency between the wavelength assignment and the central module assignment is removed since the switching function of MEMS modules in the input and output stages does not depend on the wavelength assignment. As a result, the tunable lasers are not required in ToR switches.

4.1 Logical topology

Figure 5 plots a symmetric $N\times N$ MRM network, in which there are r $n\times m$ MEMS-based switching modules in the input stage, m $r\times r$ ring-based switching modules in the central stage, and r $m\times n$ MEMS-based switching modules in the output stage, which means $N=rn$. We denote this network as $H_2\left(n,r,m\right)$.

 

Fig. 5 Logical Topology of an $N\times N$ MRM switching network $H_2\left(n,r,m\right)$.

Download Full Size | PDF

In the central stage, the rings are numbered by $1,\cdots ,k,\cdots ,m$ from top to down. Each ring contains s $\left(t+1\right)\times 1$ OCs and s $1\times (t+1)$ WSSs, and is associated with a wavelength set $\text{Λ}=\left\{{\lambda }_1,\cdots , {\lambda }_r\right\}$, where $st=r$. Each ring connects to an input module or an output module via a link. We label an input module as $I\left(a,i\right)$ if its kth output is connected to input i of the ath OC of the k th ring, and denote an output module as $O\left(b,j\right)$ if its k th input is connected to output j of the bth WSS of the k th ring, where $a,b=1,2,\cdots ,s$, $i,j=1,2,\cdots ,t$ and $k=1,2,\cdots ,m$.

In the input stage, each input module carries n ToR switches and is associated with a wavelength. We label an ToR switch as $\left(a,i,x\right)$ if it connects to the xth input of input module $I\left(a,i\right)$, where $x=1,2,\cdots ,n$. Each ToR switch attached to input module $I\left(a,i\right)$ is equipped with a laser fixed at the wavelength ${\lambda }_{\left(a-1\right)t+i}$.

In the output stage, each output module carries n ToR switches. The yth output of output module $O\left(b,j\right)$ connects to ToR switch $\left(b,j,y\right)$, where $y=1,2,\cdots ,n$. Each ToR switch is equipped with a tunable receiver such that it can receive the optical signal at any wavelength in the set $\text{Λ}$.

4.2 Non-blocking condition

It is easy to show that the network will be wavelength contention-free, if the route assignment is nonblocking. To ensure nonblocking, the strategy here is a one-to-one correspondence between the wavelengths and the input modules. If the route assignment is nonblocking, two connections that share the same central module must originate from different input modules, and thus they are at different wavelengths, which indicates that there is no wavelength contention in the central stage. Also, there is no wavelength contention in both the input and output stages, because the MEMS is wavelength insensitive.

Therefore, the MRM network has the same nonblocking condition with a Clos network. It follows that the MRM network $H_2\left(n,r,m\right)$ is nonblocking if and only if $m\ge n$, and the nonblocking route assignment can be obtained by coloring a bipartite graph as mentioned in the last section. As illustrated in Fig. 5, once the central ring module is selected, the input MEMS module and the output MEMS module can configure their switching states such that the source ToR switch and the destination ToR switch can be connected to the central module.

4.3 Implementation

Similarly, the MRM network could also be implemented in a folded form, as illustrated by the network $H_2\left(3,4,3\right)$ in Fig. 6(a) .

 

Fig. 6 Implementation of a $12\times 12$ MRM network $H_2\left(3,4,3\right)$.

Download Full Size | PDF

In general, we combine the ath OC with the a th WSS of the kth ring as a station, labeled as $T_k\left(a\right)$. From the logical topology, it is easy to find that the stations $T_1\left(a\right),T_2\left(a\right),\cdots ,T_s\left(a\right)$ connect to the same set of input modules $I\left(a,1\right),I\left(a,2\right),\cdots ,I\left(a,t\right)$ and the same set of output modules $O\left(a,1\right),O\left(a,2\right),\cdots ,O\left(a,t\right)$. In practical application, these input modules and output modules might be installed together to carry a cluster of ToR switches. Figure 6(b) plots an example, in which the first OC and the first WSS of the first ring constitutes a station $T_1\left(1\right)$, and those of the second ring form another station $T_2\left(1\right)$. Both of two stations connect with input modules $I\left(1,1\right)$ and $I\left(1,2\right)$ and output modules $O\left(1,1\right)$ and $O\left(1,2\right)$.

To support N ToRs, the MRM network illustrated in Fig. 6(b) should contain 2r n × n MEMSs and n r × r ROADM rings, each of which possesses s 1 × (t + 1) WSSs. Similar to the RMR network, the entire MRM network needs N/t optical amplifiers in addition, which is only 1/t of that of other WSS based networks [2,13 ]. The advantage of the MRM network is it can remarkably reduce the number of long cables in the network, which can be explained as follows. To avoid the employment of a lot of long fibers, the input and output modules should be placed near the ToR switches, and the m rings can be installed as a bundle and be stretched to cover different locations, in each of which the rings are equipped with a set of s stations such that the input and output modules can connect to the rings, i.e., central modules, via short fibers. It is easy to show that the number of long cables of the MRM network illustrated in Fig. 6(b) is O(N/t), which is only 1/t of that of Clos networks. The above discussion indicates that, similar to the RMR network, the MRM network can support 51200 racks at most.

5. Conclusion

To provide scalable rack-to-rack nonblocking interconnection for mega data centers, this paper proposes multi-ring based OCS networks, following the idea of Clos networks. In particular, we consider two network structures: ring-MEMS-ring (RMR) network and MEMS-ring-MEMS (MRM) network. Both of the RMR network and the MRM networks possess the following three advantages. Firstly, they are highly scalable and can provide full bisection bandwidth. Secondly, they can remarkably reduce the number of long optical fibers. Compared with the RMR network, the MRM network is more cost effective, since tunable lasers are not required in the MRM network to achieve nonblocking routing.