Abstract

High spectral efficiency (SE) transmission is a critical topic for next-generation ultra-high-speed optical networks, particularly for intensity-modulation with direct-detection (IM-DD) systems where the degree of freedom for data delivery is severely constrained. In this article, two high SE transmission approaches, namely fast than Nyquist (FTN) and high order modulation formats (HoMF) signaling, are experimentally demonstrated for peak-power constrained IM-DD systems. For FTN signaling, not only the sequence decoder based on M-Viterbi algorithm (M-VA), but also colored noise suppressed processing including partial response precoding (PRP), partial response equalization (PRE), nonlinear differential precoding (NLDP), Tomlinson-Harashima precoding (THP), and simplified THP (SfTHP) are investigated cooperatively to enhance the robustness of the system against inter-symbol interference (ISI) caused by acceleration. For HoMF signaling, probabilistic shaping (PS) with Maxwell Boltzmann (MB) and inverse MB distributions are discussed for rate adaptation. In a slight bandwidth-limited IM-DD system, although the PRE or SfTHP coupled with M-VA decoder enabled 56 GBaud FTN PAM4 signal with 0.7 compression factor is successfully transmitted over 2-km single-mode fiber (SMF) with BER under the 7% hard-decision forward error correction (HD-FEC) threshold of 3.8 × 10⁻³, a 39.2 GBaud MB-PS PAM8 with the same spectral efficiency, 2/0.7, shows about 1.8 dB receiver sensitivity improvement. However, in an IM-DD system with 32 GHz cut-off bandwidth and severe in-band degradation, the maximum achievable transmission rate of FTN PAM4 and PS-PAM8 are 162 Gb/s and 145 Gb/s, corresponding to 2.53 and 2.266 bits/symbol SEs, respectively, at BER of 3.8 × 10⁻³. According to the experimental results, the proposed colored noise suppressed FTN signaling offers significant potential as a solution for bandwidth-starved high-speed IM-DD systems.