Abstract

Every day, customers across the globe connect to cloud service provider servers with requests for diverse types of data, requiring instantaneous response times and seamless availability. The physical infrastructure which underpins those services is based on optics and optical networks, with the focus of this paper being on Microsoft’s approach to the optical network. Maintaining a global optical networking infrastructure which meets these customer needs means Microsoft must utilize solutions which are highly tailored and optimized for the application space which they address, with appropriately streamlined solutions for metropolitan data center interconnect and long-haul portions of the network. This paper presents Microsoft’s approach for tackling these challenges at cloud scale, highlighting the low-margin solutions which are employed. We provide a survey of Microsoft’s regional network design and corresponding optical network architectures, and present volumes of real-time polled metrics from the thousands of lines systems and tens of thousands of transceivers deployed today. We close by describing our approach to a unified software-defined networking toolset which ultimately enables the velocity and scale with which we can grow and operate this critical network infrastructure.

© 2019 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Elastic Optical Networking in the Microsoft Cloud [Invited]

Mark Filer, Jamie Gaudette, Monia Ghobadi, Ratul Mahajan, Tom Issenhuth, Buddy Klinkers, and Jeff Cox
J. Opt. Commun. Netw. 8(7) A45-A54 (2016)

Rapid Data Evacuation for Large-Scale Disasters in Optical Cloud Networks [Invited]

Sifat Ferdousi, Massimo Tornatore, M. Farhan Habib, and Biswanath Mukherjee
J. Opt. Commun. Netw. 7(12) B163-B172 (2015)

Design considerations for low-margin elastic optical networks in the nonlinear regime [Invited]

Seb J. Savory, Robert J. Vincent, and David J. Ives
J. Opt. Commun. Netw. 11(10) C76-C85 (2019)

References

  • View by:
  • |
  • |
  • |

  1. “What is DCI?” https://www.ciena.com/insights/what-is/What-is-DCI.html .
  2. IEEE ComSoc, “Data center interconnect,” http://techblog.comsoc.org/category/data-center-interconnect/ .
  3. AWS re:Invent, “Enterprise fundamentals: design your account and VPC architecture for enterprise operating models,” 2016, https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-enterprise-fundamentals-design-your-account-and-vpc-architecture-for-enterprise-operating-models-ent203 .
  4. “Inphi offers 100G PAM4 QSFP28 for 80-km data center interconnect,” https://www.lightwaveonline.com/articles/2016/03/inphi-offers-100g-wavelength-pam4-qsfp28-for-80-km-data-center-interconnect.html .
  5. M. Filer, J. Gaudette, M. Ghobadi, R. Mahajan, T. Issenhuth, B. Klinkers, and J. Cox, “Elastic optical networking in the Microsoft cloud [Invited],” J. Opt. Commun. Netw. 8, A45–A54 (2016).
    [Crossref]
  6. M. Filer, H. Chaouch, and X. Wu, “Toward transport ecosystem interoperability enabled by vendor-diverse coherent optical sources over an open line system,” J. Opt. Commun. Netw. 10, A216–A224 (2018).
    [Crossref]
  7. M. Filer and H. Chaouch, “Transmission performance of layer-2/3 modular switch with mQAM coherent ASIC and CFP2-ACOs over flex-grid OLS with 104 channels spaced 37.5  GHz,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.2.
  8. “How Microsoft builds its fast and reliable global network,” https://azure.microsoft.com/en-us/blog/how-microsoft-builds-its-fast-and-reliable-global-network/ .
  9. Packet Design, “BGP in the data center: part two,” https://www.packetdesign.com/blog/clos-architecture-in-the-data-center/ .
  10. A. Greenberg, “SDN for the cloud,” keynote talk at SigComm,Aug.2015.
  11. “Overview of availability zones in Azure,” https://docs.microsoft.com/en-us/azure/availability-zones/az-overview .
  12. “AWS global infrastructure,” https://aws.amazon.com/about-aws/global-infrastructure/ .
  13. F. Dikbiyik, M. Tornatore, and B. Mukherjee, “Minimizing the risk from disaster failures in optical backbone networks,” J. Lightwave Technol. 32, 3175–3183 (2014).
    [Crossref]
  14. S. Ferdousi, F. Dikbiyik, M. F. Habib, M. Tornatore, and B. Mukherjee, “Disaster-aware datacenter placement and dynamic content management in cloud networks,” J. Opt. Commun. Netw. 7, 681–694 (2015).
    [Crossref]
  15. AGC Research, “Inphi’s ColorZ-Lite Technology: offering an innovative solution for Nx100  G campus connectivity,” https://www.acgcc.com/inphis-colorz-lite-technology-offering-an-innovative-solution-for-nx100-g-campus-connectivity/ .
  16. R. Nagarajan, M. Filer, Y. Fu, M. Kato, T. Rope, and J. Stewart, “Silicon photonics-based 100  Gbit/s, PAM4, DWDM data center interconnects,” J. Opt. Commun. Netw. 10, B25–B36 (2018).
    [Crossref]
  17. M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.
  18. M. Filer, M. Cantono, A. Ferrari, G. Grammel, G. Galimberti, and V. Curri, “Multi-vendor experimental validation of an open source QoT estimator for optical networks,” J. Lightwave Technol. 36, 3073–3082 (2018).
    [Crossref]
  19. V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.
  20. V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.
  21. H. Chaouch, M. Filer, and A. Bechtolsheim, “Lessons learned from CFP2-ACO system integrations, interoperability testing and deployments,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.4.
  22. Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.
  23. P. Poggiolini, “The GN model of non-linear propagation in uncompensated coherent optical systems,” J. Lightwave Technol. 30, 3857–3879 (2012).
    [Crossref]
  24. A. Carena, G. Bosco, V. Curri, Y. Jiang, P. Poggiolini, and F. Forghieri, “EGN model of non-linear fiber propagation,” Opt. Express 22, 16335–16362 (2014).
    [Crossref]
  25. D. J. Elson, G. Saavedra, K. Shi, D. Semrau, L. Galdino, R. Killey, B. C. Thomsen, and P. Bayvel, “Investigation of bandwidth loading in optical fibre transmission using amplified spontaneous emission noise,” Opt. Express 25, 19529–19537 (2017).
    [Crossref]
  26. T. Richter, J. Pan, and S. Tibuleac, “Comparison of WDM bandwidth loading using individual transponders, shaped, and flat ASE noise,” in Optical Fiber Communications Conference and Exposition (OFC), San Diego, California, 2018, pp. 1–3.
  27. M. Cantono, A. Ferrari, D. Pilori, E. Virgillito, J. L. Augé, and V. Curri, “Physical layer performance of multi-band optical line systems using Raman amplification,” J. Opt. Commun. Netw. 11, A103–A110 (2019).
    [Crossref]
  28. E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.
  29. PSM4 MSA, http://psm4.org/ .
  30. CWDM4 MSA, http://www.cwdm4-msa.org/ .
  31. D. Rafique, T. Szyrkowiec, H. Grießer, A. Autenrieth, and J.-P. Elbers, “Cognitive assurance architecture for optical network fault management,” J. Lightwave Technol. 36, 1443–1450 (2018).
    [Crossref]
  32. S. Shahkarami, F. Musumeci, F. Cugini, and M. Tornatore, “Machine-learning-based soft-failure detection and identification in optical networks,” in Optical Fiber Communication Conference (Optical Society of America, 2018), paper M3A.5.
  33. A. P. Vela, M. Ruiz, F. Fresi, N. Sambo, F. Cugini, G. Meloni, L. Potì, L. Velasco, and P. Castoldi, “BER degradation detection and failure identification in elastic optical networks,” J. Lightwave Technol. 35, 4595–4604 (2017).
    [Crossref]
  34. Y. Xiong, Y. Li, B. Zhou, R. Wang, and G. N. Rouskas, “SDN enabled restoration with triggered precomputation in elastic optical inter-datacenter networks,” J. Opt. Commun. Netw. 10, 24–34 (2018).
    [Crossref]
  35. M. Dzanko, M. Furdek, G. Zervas, and D. Simeonidou, “Evaluating availability of optical networks based on self-healing network function programmable ROADMs,” J. Opt. Commun. Netw. 6, 974–987 (2014).
    [Crossref]
  36. R. Casellas, R. Martínez, R. Vilalta, and R. Muñoz, “Control, management, and orchestration of optical networks: evolution, trends, and challenges,” J. Lightwave Technol. 36, 1390–1402 (2018).
    [Crossref]
  37. E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.
  38. SNIA, “SFF specifications,” https://www.snia.org/technology-communities/sff/specifications .
  39. GitHub, “What is SONiC?” https://azure.github.io/SONiC/ .
  40. OpenConfig, http://www.openconfig.net/ .
  41. E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.
  42. Swagger, https://swagger.io/ .
  43. “OIF 400ZR,” https://www.oiforum.com/technical-work/hot-topics/400zr-2 .

2019 (1)

2018 (6)

2017 (2)

2016 (1)

2015 (1)

2014 (3)

2012 (1)

Augé, J. L.

Autenrieth, A.

Bayvel, P.

Bechtolsheim, A.

H. Chaouch, M. Filer, and A. Bechtolsheim, “Lessons learned from CFP2-ACO system integrations, interoperability testing and deployments,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.4.

Bosco, G.

Breverman, E.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

Cantono, M.

Carena, A.

Casellas, R.

Castoldi, P.

Chaouch, H.

M. Filer, H. Chaouch, and X. Wu, “Toward transport ecosystem interoperability enabled by vendor-diverse coherent optical sources over an open line system,” J. Opt. Commun. Netw. 10, A216–A224 (2018).
[Crossref]

H. Chaouch, M. Filer, and A. Bechtolsheim, “Lessons learned from CFP2-ACO system integrations, interoperability testing and deployments,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.4.

M. Filer and H. Chaouch, “Transmission performance of layer-2/3 modular switch with mQAM coherent ASIC and CFP2-ACOs over flex-grid OLS with 104 channels spaced 37.5  GHz,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.2.

Cox, J.

Cugini, F.

A. P. Vela, M. Ruiz, F. Fresi, N. Sambo, F. Cugini, G. Meloni, L. Potì, L. Velasco, and P. Castoldi, “BER degradation detection and failure identification in elastic optical networks,” J. Lightwave Technol. 35, 4595–4604 (2017).
[Crossref]

S. Shahkarami, F. Musumeci, F. Cugini, and M. Tornatore, “Machine-learning-based soft-failure detection and identification in optical networks,” in Optical Fiber Communication Conference (Optical Society of America, 2018), paper M3A.5.

Curri, V.

Dangui, V.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

Dikbiyik, F.

Dzanko, M.

Elbers, J.-P.

El-Sakkary, N.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

Elson, D. J.

Ferdousi, S.

Ferrari, A.

Filer, M.

M. Filer, M. Cantono, A. Ferrari, G. Grammel, G. Galimberti, and V. Curri, “Multi-vendor experimental validation of an open source QoT estimator for optical networks,” J. Lightwave Technol. 36, 3073–3082 (2018).
[Crossref]

R. Nagarajan, M. Filer, Y. Fu, M. Kato, T. Rope, and J. Stewart, “Silicon photonics-based 100  Gbit/s, PAM4, DWDM data center interconnects,” J. Opt. Commun. Netw. 10, B25–B36 (2018).
[Crossref]

M. Filer, H. Chaouch, and X. Wu, “Toward transport ecosystem interoperability enabled by vendor-diverse coherent optical sources over an open line system,” J. Opt. Commun. Netw. 10, A216–A224 (2018).
[Crossref]

M. Filer, J. Gaudette, M. Ghobadi, R. Mahajan, T. Issenhuth, B. Klinkers, and J. Cox, “Elastic optical networking in the Microsoft cloud [Invited],” J. Opt. Commun. Netw. 8, A45–A54 (2016).
[Crossref]

H. Chaouch, M. Filer, and A. Bechtolsheim, “Lessons learned from CFP2-ACO system integrations, interoperability testing and deployments,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.4.

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

M. Filer and H. Chaouch, “Transmission performance of layer-2/3 modular switch with mQAM coherent ASIC and CFP2-ACOs over flex-grid OLS with 104 channels spaced 37.5  GHz,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.2.

Forghieri, F.

Fresi, F.

Fu, Y.

R. Nagarajan, M. Filer, Y. Fu, M. Kato, T. Rope, and J. Stewart, “Silicon photonics-based 100  Gbit/s, PAM4, DWDM data center interconnects,” J. Opt. Commun. Netw. 10, B25–B36 (2018).
[Crossref]

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

Furdek, M.

Galdino, L.

Galimberti, G.

Gaudette, J.

Ghobadi, M.

Gibbemeyer, A.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Grammel, G.

Greenberg, A.

A. Greenberg, “SDN for the cloud,” keynote talk at SigComm,Aug.2015.

Grießer, H.

Habib, M. F.

Hofmeister, T.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

Issenhuth, T.

Jiang, Y.

Kamalov, V.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Kato, M.

Killey, R.

Klinkers, B.

Koley, B.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

Li, Y.

Loussouarn, Y.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Mahajan, R.

Martínez, R.

Meloni, G.

Mikkelsen, B.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Miller, G.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Mitchell, C.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Mukherjee, B.

Muñoz, R.

Musumeci, F.

S. Shahkarami, F. Musumeci, F. Cugini, and M. Tornatore, “Machine-learning-based soft-failure detection and identification in optical networks,” in Optical Fiber Communication Conference (Optical Society of America, 2018), paper M3A.5.

Nagarajan, R.

R. Nagarajan, M. Filer, Y. Fu, M. Kato, T. Rope, and J. Stewart, “Silicon photonics-based 100  Gbit/s, PAM4, DWDM data center interconnects,” J. Opt. Commun. Netw. 10, B25–B36 (2018).
[Crossref]

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

Newland, M.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Ngai, S.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

O’Shea, J.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Pan, J.

T. Richter, J. Pan, and S. Tibuleac, “Comparison of WDM bandwidth loading using individual transponders, shaped, and flat ASE noise,” in Optical Fiber Communications Conference and Exposition (OFC), San Diego, California, 2018, pp. 1–3.

Pan, Y.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Pilori, D.

Pincemin, E.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

Poggiolini, P.

Potì, L.

Rafique, D.

Richter, T.

T. Richter, J. Pan, and S. Tibuleac, “Comparison of WDM bandwidth loading using individual transponders, shaped, and flat ASE noise,” in Optical Fiber Communications Conference and Exposition (OFC), San Diego, California, 2018, pp. 1–3.

Rope, T.

Rouskas, G. N.

Ruiz, M.

Saavedra, G.

Sambo, N.

Searcy, S.

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

Semrau, D.

Shahkarami, S.

S. Shahkarami, F. Musumeci, F. Cugini, and M. Tornatore, “Machine-learning-based soft-failure detection and identification in optical networks,” in Optical Fiber Communication Conference (Optical Society of America, 2018), paper M3A.5.

Shaikh, A.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

Shi, K.

Simeonidou, D.

Stewart, J.

Szyrkowiec, T.

Thomsen, B. C.

Tibuleac, S.

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

T. Richter, J. Pan, and S. Tibuleac, “Comparison of WDM bandwidth loading using individual transponders, shaped, and flat ASE noise,” in Optical Fiber Communications Conference and Exposition (OFC), San Diego, California, 2018, pp. 1–3.

Tomblin, C.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Tornatore, M.

Vela, A. P.

Velasco, L.

Vilalta, R.

Virgillito, E.

Vusirikala, V.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

Wang, R.

Wu, X.

Xiong, Y.

Zervas, G.

Zhao, X.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

Zhou, B.

J. Lightwave Technol. (6)

J. Opt. Commun. Netw. (7)

Y. Xiong, Y. Li, B. Zhou, R. Wang, and G. N. Rouskas, “SDN enabled restoration with triggered precomputation in elastic optical inter-datacenter networks,” J. Opt. Commun. Netw. 10, 24–34 (2018).
[Crossref]

M. Dzanko, M. Furdek, G. Zervas, and D. Simeonidou, “Evaluating availability of optical networks based on self-healing network function programmable ROADMs,” J. Opt. Commun. Netw. 6, 974–987 (2014).
[Crossref]

M. Cantono, A. Ferrari, D. Pilori, E. Virgillito, J. L. Augé, and V. Curri, “Physical layer performance of multi-band optical line systems using Raman amplification,” J. Opt. Commun. Netw. 11, A103–A110 (2019).
[Crossref]

S. Ferdousi, F. Dikbiyik, M. F. Habib, M. Tornatore, and B. Mukherjee, “Disaster-aware datacenter placement and dynamic content management in cloud networks,” J. Opt. Commun. Netw. 7, 681–694 (2015).
[Crossref]

R. Nagarajan, M. Filer, Y. Fu, M. Kato, T. Rope, and J. Stewart, “Silicon photonics-based 100  Gbit/s, PAM4, DWDM data center interconnects,” J. Opt. Commun. Netw. 10, B25–B36 (2018).
[Crossref]

M. Filer, J. Gaudette, M. Ghobadi, R. Mahajan, T. Issenhuth, B. Klinkers, and J. Cox, “Elastic optical networking in the Microsoft cloud [Invited],” J. Opt. Commun. Netw. 8, A45–A54 (2016).
[Crossref]

M. Filer, H. Chaouch, and X. Wu, “Toward transport ecosystem interoperability enabled by vendor-diverse coherent optical sources over an open line system,” J. Opt. Commun. Netw. 10, A216–A224 (2018).
[Crossref]

Opt. Express (2)

Other (28)

T. Richter, J. Pan, and S. Tibuleac, “Comparison of WDM bandwidth loading using individual transponders, shaped, and flat ASE noise,” in Optical Fiber Communications Conference and Exposition (OFC), San Diego, California, 2018, pp. 1–3.

V. Kamalov, V. Dangui, T. Hofmeister, B. Koley, C. Mitchell, M. Newland, J. O’Shea, C. Tomblin, V. Vusirikala, and X. Zhao, “Lessons learned from open line system deployments,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2017, pp. 1–3.

V. Vusirikala, X. Zhao, T. Hofmeister, V. Kamalov, V. Dangui, and B. Koley, “Scalable and flexible transport networks for inter-datacenter connectivity,” in Optical Fiber Communications Conference and Exhibition (OFC), Los Angeles, California, 2015, pp. 1–3.

H. Chaouch, M. Filer, and A. Bechtolsheim, “Lessons learned from CFP2-ACO system integrations, interoperability testing and deployments,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.4.

Y. Loussouarn, E. Pincemin, Y. Pan, G. Miller, A. Gibbemeyer, and B. Mikkelsen, “Silicon photonic multi-rate DCO-CFP2 interface for DCI, metro, and long-haul optical communications,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2018), paper M1E.5.

E. Breverman, N. El-Sakkary, T. Hofmeister, S. Ngai, A. Shaikh, and V. Vusirikala, “Optical zero touch networking—a large operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

PSM4 MSA, http://psm4.org/ .

CWDM4 MSA, http://www.cwdm4-msa.org/ .

S. Shahkarami, F. Musumeci, F. Cugini, and M. Tornatore, “Machine-learning-based soft-failure detection and identification in optical networks,” in Optical Fiber Communication Conference (Optical Society of America, 2018), paper M3A.5.

M. Filer and H. Chaouch, “Transmission performance of layer-2/3 modular switch with mQAM coherent ASIC and CFP2-ACOs over flex-grid OLS with 104 channels spaced 37.5  GHz,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper Th1D.2.

“How Microsoft builds its fast and reliable global network,” https://azure.microsoft.com/en-us/blog/how-microsoft-builds-its-fast-and-reliable-global-network/ .

Packet Design, “BGP in the data center: part two,” https://www.packetdesign.com/blog/clos-architecture-in-the-data-center/ .

A. Greenberg, “SDN for the cloud,” keynote talk at SigComm,Aug.2015.

“Overview of availability zones in Azure,” https://docs.microsoft.com/en-us/azure/availability-zones/az-overview .

“AWS global infrastructure,” https://aws.amazon.com/about-aws/global-infrastructure/ .

M. Filer, S. Searcy, Y. Fu, R. Nagarajan, and S. Tibuleac, “Demonstration and performance analysis of 4  Tb/s DWDM metro-DCI system with 100G PAM4 QSFP28 modules,” in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2017), paper W4D.4.

AGC Research, “Inphi’s ColorZ-Lite Technology: offering an innovative solution for Nx100  G campus connectivity,” https://www.acgcc.com/inphis-colorz-lite-technology-offering-an-innovative-solution-for-nx100-g-campus-connectivity/ .

“What is DCI?” https://www.ciena.com/insights/what-is/What-is-DCI.html .

IEEE ComSoc, “Data center interconnect,” http://techblog.comsoc.org/category/data-center-interconnect/ .

AWS re:Invent, “Enterprise fundamentals: design your account and VPC architecture for enterprise operating models,” 2016, https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-enterprise-fundamentals-design-your-account-and-vpc-architecture-for-enterprise-operating-models-ent203 .

“Inphi offers 100G PAM4 QSFP28 for 80-km data center interconnect,” https://www.lightwaveonline.com/articles/2016/03/inphi-offers-100g-wavelength-pam4-qsfp28-for-80-km-data-center-interconnect.html .

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Optical network control & management plane evolution—a large datacenter operator perspective,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–4.

SNIA, “SFF specifications,” https://www.snia.org/technology-communities/sff/specifications .

GitHub, “What is SONiC?” https://azure.github.io/SONiC/ .

OpenConfig, http://www.openconfig.net/ .

E. Breverman, N. El-Sakkary, T. Hofmeister, A. Shaikh, and V. Vusirikala, “Data models for optical devices in data center operator networks,” in Optical Fiber Communications Conference and Exhibition (OFC), San Diego, California, 2019, pp. 1–3.

Swagger, https://swagger.io/ .

“OIF 400ZR,” https://www.oiforum.com/technical-work/hot-topics/400zr-2 .

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (18)

Fig. 1.
Fig. 1. Microsoft global DC + WAN footprint.
Fig. 2.
Fig. 2. (a) Legacy “mega DC” architecture and (b) current regional architecture.
Fig. 3.
Fig. 3. Impact of regional architecture on Microsoft deployed 100G DWDM ports.
Fig. 4.
Fig. 4. PAM4 DCI OLS line system “stamp.”
Fig. 5.
Fig. 5. Open line system concept.
Fig. 6.
Fig. 6. Long-haul OLS line system “stamps.”
Fig. 7.
Fig. 7. Long-haul OLS optical spectrum, showing fully filled C-band with data-carrying signals in the upper frequencies and ASE noise-loading throughout the remainder.
Fig. 8.
Fig. 8. DCI fiber quality statistics.
Fig. 9.
Fig. 9. (a) Fiber distance and (b) loss distributions of the first $ {\gt}1500$ deployed DWDM PAM4 line systems.
Fig. 10.
Fig. 10. (a) Fiber distance, (b) loss, (c) type, and (d) route length distributions across the first 26,000 km of deployed Microsoft long-haul OLS installations. Fiber types: “LC” = large core (large effective area, e.g., $ {\ge}80\,\,{\unicode{x00B5} }{{\rm{m}}^2}$ ), “SC” = small core (small effective area, e.g.,  $ {\le}55\,\,{\unicode{x00B5}} {{\rm{m}}^2}$ ).
Fig. 11.
Fig. 11. $Q$ -factor distribution of the first 50,000 deployed DWDM PAM4 ports.
Fig. 12.
Fig. 12. BER stability over a 50-day period.
Fig. 13.
Fig. 13. Temperature distribution over optical module infrastructure.
Fig. 14.
Fig. 14. Failure rates of deployed 100G technologies.
Fig. 15.
Fig. 15. Long-haul OLS BER statistics across 26,000 km of deployed infrastructure: (a) 16QAM, (b) 8QAM, and (c) QPSK.
Fig. 16.
Fig. 16. Typical deployment workflow screenshot.
Fig. 17.
Fig. 17. Screenshot taken from production tooling showing the workflow for the router linecard refresh process; “wanetmon” is an internal device health checker, “swan” is a specific WAN router variant.
Fig. 18.
Fig. 18. Production tooling screenshot: YAML underpinning of the linecard deployment workflow shown in Fig. 17.

Tables (1)

Tables Icon

Table 1. Global Fiber Specifications