## Abstract

A new intersymbol interference (ISI)-free nonlinearity-tolerant frequency domain root M-shaped pulse (RMP) is derived for dispersion unmanaged coherent optical transmission systems. Beginning with the relationship between pulse shaping and intra-channel nonlinearity effects, we derive closed-form expressions for the proposed pulse. Experimental demonstrations reveal that by employing the proposed pulse at a roll-off factor of 1, the maximum transmission reach of a single-channel 56 Gb/s polarization-division-multiplexed quadrature phase-shift keying (PDM-QPSK) system can be extended by 33% and 17%, when compared to systems using a root raised cosine (RRC) pulse and a root optimized pulse (ROP), respectively. For a single-channel 128 Gb/s polarization-division-multiplexed 16-quadrature amplitude modulation (PDM-16QAM) system, the reach can be extended by 44% and 18%, respectively. Reach increases of 30% and 13% are also observed for a dense wavelength-division multiplexing (DWDM) 504 Gb/s PDM-QPSK transmission system. The tolerance to narrow filtering effect for the three pulses is experimentally studied as well.

© 2013 Optical Society of America

## 1. Introduction

The Kerr nonlinearity is an impairment in optical fibers that fundamentally limits the achievable capacity of long-haul transmission systems [1]. With the advent of high speed digital-to-analog converters (DAC) [2,3] and coherent detection, advanced digital signal processing (DSP) algorithms can be implemented at the transmitter and the receiver to compensate for fiber nonlinearity effects. For example, digital back-propagation (DBP) [4] has been extensively studied because it can effectively compensate intra/inter-channel fiber nonlinearities by solving the inverse nonlinear Schrödinger equation (NLSE) using the split-step Fourier method (SSFM). The main issue that inhibits DBP from being implemented in deployed systems is the high computation complexity associated with this approach, even though several reduced computation complexity algorithms have been proposed [5–7].

Transmitter based pulse shaping techniques have attracted recent attention as a nonlinearity mitigation scheme because these techniques do not require intense computation. It has been shown that a return-to-zero (RZ) pulse has a better tolerance to both self-phase modulation (SPM) and cross-phase modulation (XPM) than a non-return-to-zero (NRZ) pulse [8] because the RZ pulse has a wider spectrum than the NRZ pulse thus reducing the phase-matching between adjacent frequency components during propagation in a dispersive media [9]. The RZ pulse is rarely used because it occupies twice the amount of spectrum relative to the NRZ pulse thus halving the spectral efficiency and doubling the required bandwidth and sampling rate at the receiver. The root raised cosine (RRC) pulse has been widely used in digital communication systems due to its superior out-of-band attenuation, ISI-free characteristic and compact spectrum [10]. Nevertheless, it was not designed to have a strong nonlinearity tolerance when applied in optical fiber transmission systems. In [11,12], a pulse shape obtained by solving a numerical optimization problem to minimize the pulse width under the bandlimit constraint has shown improved nonlinearity tolerance compared to the RRC pulse with the same bandwidth. In this paper, we refer to this pulse and its root version as the optimized pulse (OP) and root optimized pulse (ROP). In our previous study [13], we proposed a frequency domain root M-shape pulse (RMP) for a single-channel polarization-division-multiplexed quadrature phase-shift keying (PDM-QPSK) system to further enhance the system’s nonlinearity tolerance while maintaining the same spectral efficiency. In this paper, we give a detailed analysis of the RMP. Furthermore, we extend our previous experimental study to a single-channel polarization-division-multiplexed 16-quadrature amplitude modulation (PDM-16QAM) system and a dense wavelength-division multiplexing (DWDM) PDM-QPSK system.

The paper is organized as follows: in Section 2, starting from the relationship between pulse shaping and intra-channel nonlinearity effects, we derive closed-form expressions for the MP and RMP. The time domain pulse truncation effect is then studied. This section concludes with a comparison of the chromatic dispersion (CD) induced pulse broadening of the aforementioned pulse shapes in order to support our principle. In Section 3, we experimentally compare the nonlinearity tolerance of the RMP to that of the RRC pulse and ROP at a roll-off factor of 1, for a single-channel 56 Gb/s PDM-QPSK system, a single-channel 128 Gb/s PDM-16QAM system and a DWDM 504 Gb/s PDM-QPSK system, respectively. Finally, the narrow filtering tolerance of the above pulses is investigated. In Section 4 the paper concludes.

## 2. The frequency domain root M-shaped pulse

#### 2.1 Nonlinearity-tolerant pulse shape design

Before the arrival of coherent technologies, optical transmission systems were primarily based on dispersion managed links, in which the pulses maintain their pulse shapes during transmission. As a result, the dominant intra-channel nonlinearity effect in these systems was intra-channel self-phase modulation (ISPM). Coherent transmission systems enable dispersion unmanaged links, where the chromatic dispersion can be compensated entirely in the electronic domain. Due to the accumulated chromatic dispersion, pulses disperse and overlap with adjacent pulses during transmission. The pulse overlap leads to two additional pulse-to-pulse nonlinear interactions, namely intra-channel cross phase modulation (IXPM) and intra-channel four-wave-mixing (IFWM). According to a previous study [14], IXPM strength is closely related to the pulse width. After propagation, the chromatic dispersion (CD) makes the pulse width much larger than the bit period and IXPM is actually weakened. This stems from that fact that IXPM induced frequency shift tends to be constant when the current pulse is overlapping with a large number of adjacent pulses, due to the law of large numbers. Therefore, a rapidly dispersing pulse can mitigate IXPM. Moreover, in a rapidly dispersing pulse, the spectral components will quickly walk-off of each other, thereby reducing the IFWM efficiency. For a given pulse bandwidth, a rapidly dispersing pulse in the time domain can be realized in the frequency domain by putting more energy contents in the pulse’s higher frequency components. This is the basic idea of our pulse design.

In addition, the target pulse has to meet the Nyquist zero inter-symbol interference (ISI) criterion [15], which states that the interference of the current pulse on adjacent pulses must be zero at the sampling points. The necessary and sufficient condition for the Nyquist zero ISI criterion in the frequency domain is the pulse’s Fourier transform $X(f)$ must satisfy:

where $T$ is the symbol period and $n$ is an integer. It can be interpreted from Eq. (1) that in order to satisfy the Nyquist zero ISI criterion in the frequency domain, the frequency shifted by $n/T$ replicas of $X(f)$ should sum to a constant value.To conclude, the target pulse in the frequency domain should meet the following two conditions: (1) put more energy contents in the pulse’s high frequency components within a finite bandwidth; (2) fulfill the Nyquist criterion given by Eq. (1).

#### 2.2 Closed-form expression for the proposed pulse

We propose a frequency domain M-shaped pulse (MP) that can satisfy the above two conditions. The frequency characteristic of the MP is expressed as:

Matched filtering is usually implemented in communication systems in order to maximize the signal-to-noise ratio (SNR) when the signal is corrupted by additive white Gaussian noise [10]. It is well known that the amplified spontaneous emission (ASE) noise from the Erbium-doped fiber amplifier (EDFA) can be considered as additive Gaussian noise. Recent studies reveal that the nonlinear interference can be modeled as additive Gaussian noise as well in dispersion unmanaged links with low-to-moderate nonlinearity [16], which is our study range in this paper. Therefore, it is appropriate to use the matched filtering to maximize the SNR in our analysis. The overall frequency response of the MP is split evenly between the transmitter filter and the receiver filter, to form the RMP. The frequency characteristic of the RMP is expressed as:

The time domain impulse response $m(t)$ of the MP and $rm(t)$of the RMP can be derived by taking the inverse Fourier transform of $M(f)$ and $RM(f)$and are given by:

#### 2.3 The depth factor$\beta $

The depth factor $\beta $of the proposed pulse can be used to adjust the energy contents between the higher frequency components and the lower frequency components. Although the $\beta $factor can take any non-negative value, in order to have more energy in the high frequency spectrum than in low frequency spectrum, $\beta $ should be equal or smaller than 1. Hence, in what follows, we only consider the cases for $0\le \beta \le 1$. Figure 1 visualizes the ideal frequency response and impulse response of the MP and RMP at $\beta $ factors of 1, 0.5 and 0.25. It can be seen that from Figs. 1(a) and 1(c), as the $\beta $ decreases, there are more energy contents in the high frequency spectrum. This translates to a pulse with a narrower full width at half maximum (FWHM) in the time domain, as shown in Figs. 1(b) and 1(d). Therefore, the aforementioned first pulse design criterion can be interpreted in another way: the desired pulse needs to have a narrower FWHM in the time domain. The $\beta $ factor also plays an important role in making a tradeoff between linear and nonlinear performance for a particular scenario, e.g., reduced DAC resolution in the higher frequencies, limited transmitter and receiver bandwidth or strong filtering in the link.

#### 2.4 The roll-off factor$\alpha $

The ideal frequency responses of the MP and RMP at roll-off factors $\alpha $ of 0, 0.5 and 1 are illustrated in Fig. 2. We also include the RC and RRC pulses in these figures for comparison. It can be seen that at a roll-off factor of 0, both the RC/RRC pulse and MP/RMP converge to the Nyquist pulse. As the roll-off factor increases, the MP/RMP has more energy contents in its higher frequency spectrum than the RC/RRC pulse. This confirms that the proposed pulse meets the first design criterion. Examination of Fig. 2 also indicates that the overall frequency response of the proposed pulse satisfies the aforementioned Nyquist ISI-free criterion at all roll-off factors.

The time domain impulse responses of the RC pulse, the MP, the RRC pulse and the RMP at roll-off factors $\alpha $ of 0, 0.5 and 1 are shown in Fig. 3. We can observe three main attributes of the proposed pulse from this figure. Firstly, at all roll-off factors, the MP is ISI-free at all the sampling instants. This agrees with that the MP meets the Nyquist criterion. Secondly, except for the case when the roll-off factor equals 0, the MP/RMP has a shorter full width at half maximum (FWHM) than the RC/RRC pulses. The difference in the FWHM width between the MP/RMP and the RC/RRC pulse becomes larger at a larger roll-off factor. This can be explained from Fig. 2 where at larger roll-off factors, the difference in the energy contents at the high frequencies between these two pulses becomes larger. This translates in the time domain to a larger difference in the FWHM width. Third, the impulse response of the MP/RMP has a slower decay rate than that of the RC/RRC pulse because the MP/RMP has a sharp transition band in its spectrum. For the same reason, the pulses at smaller roll-off factors generally have a slower decay rate than the pulses at larger roll-off factors.

#### 2.5 FIR filter windowing effects

The ideal frequency response of a bandlimited signal usually corresponds to a non-casual filter with an infinite impulse response extending from $-\infty $ to$+\infty $. However, in a real implementation, this filter needs to be approximated by a finite impulse response (FIR) filter by truncating the ideal impulse response through windowing. The slow decay rate of the RMP impulse response will lead to an increase in the FIR filter length and the short truncation of the impulse response introduces a strong ISI on the pulse [18]. The reason is that a truncation is a multiplication of the ideal filter’s impulse response and a finite duration window. In the frequency domain, this is equivalent to a periodic convolution of the ideal filter’s frequency response with the frequency response of the window. This modified frequency response may violate the Nyquist ISI-free criterion, leading to ISI, which can be easily observed from the pulse’s eyediagram. Figure 4 shows the eyediagrams of the MP after matched filtering with a rectangular truncation window of 16, 32 and 64 symbols. These eyediagrams were obtained using a QPSK sequence of 2^{12} symbols and an oversampling rate of 16 samples per symbol. It can be seen that, with a rectangular window of 16 and 32 symbols, the ISI induced vertical eye opening reduction is about 18% and 12%, respectively. Nevertheless, with a rectangular window of 64 symbols, the ISI induced vertical eye opening reduction is negligible. Therefore, the FIR filter length covering 64 symbols is sufficient. The increase in the FIR filter length will lead to high computation complexity. One solution would be performing the filtering in the frequency domain. For example, with a FIR filter length of 128 taps (64 symbols and 2 samples/symbol) the time domain direct convolution requires 128 multiplications and 127 additions per output. Whilst with the frequency domain filtering, the number of operations per output is reduced to 13 multiplications and 23 additions, assuming a fast Fourier transform (FFT) size of 1024 is used. However, for a fair comparison with other pulses, we use the time domain FIR filters with a window of 64 symbols for all the pulses in this paper.

Figure 5 shows the frequency response of the RRC pulse, ROP and RMP at a roll-off factor of 1, truncated with a rectangular windowing of 64 symbols and 8 samples/symbol. As can be seen, the main lobe null point of all of the pulses equals the baudrate *B*. The RRC has the largest out-of-band attenuation because of its gradual transition band. The RMP has slightly smaller out-of-band attenuation than the ROP in the stopband of [*B*, 1.25*B*] due to the characteristic of the rectangular window. Although we could use other types of windows to improve the out-of-band attenuation, they will also broaden the main lobe and violate the bandlimit constraint. Moreover, in a real system, the achievable out-of-band attenuation is limited by the finite resolution of the DACs. Therefore, the rectangular window is employed for all the pulse shaping filters in this paper.

#### 2.6 CD induced pulse broadening

Finally, we compare the broadening rate of the above three pulses in the presence of CD. Figure 6 shows the power profile of the RRC pulse, ROP and RMP without CD and with a cumulative CD of 17000 ps/nm, which is equivalent to the amount of CD experienced after 1000 km transmission in a standard single mode fiber (SSMF) with a dispersion of 17 ps/nm/km. As can be seen, without CD, the RMP has a slightly shorter FWHM than the RRC and ROP. But in the presence of the same amount of dispersion, the RMP has a broader propagating waveform than the RRC pulse and ROP and overlaps with more adjacent pulses. As the FWHM is not a true measurement of the width of a complicated pulse shape, we used the root-mean-square (RMS) width $\sigma $ to more accurately quantify the width of the pulse in the presence of CD [19]. Figure 7 numerically shows the ratio of $\sigma /T$ versus the transmission distance for different pulses, assuming only 17 ps/nm/km CD is considered while noise and nonlinearity effects are neglected. As we can see, all the pulse widths increase linearly with transmission distance. The slope of the $\sigma /T$ ratio for the RMP is larger than other pulses, indicating that the RMP experience more rapid pulse broadening. This agrees well with our previous analysis. This rapid broadening enables better nonlinearity tolerance for the RMP relative to the other pulses.

## 3. Experimental demonstration

#### 3.1 Single-channel PDM-QPSK experiment

A schematic of the experimental setup that we used to evaluate the intra-channel nonlinearity tolerance of the RRC pulse, ROP and RMP at a roll-off factor of 1 in a 56 Gb/s 14 Gbaud PDM-QPSK transmission system is depicted in Fig. 8. In the transmitter offline DSP, two random 2-level signals of length 2^{17} are filtered by pulse shaping FIR filters with a tap length of 128, corresponding to 64 symbols with an oversampling ratio of 2. The filtered outputs are resampled and then pre-emphasized to compensate for the DACs’ analog frequency response. After this, they are quantized to 6 bit resolution and preloaded in the memories of two field-programmable gate array (FPGA) boards driving two DACs. The transmitter laser is an external cavity laser (ECL) operating at 1554.94 nm with a linewidth of less than 100 kHz. The emitted CW light is modulated with an IQ modulator, driven by the I and Q signals from the two aforementioned DACs. The following PDM emulator consists of a pair of PBS/PBC and an ODL with a de-correlation delay of 1512 symbols at 14 GBaud. The PDM-QPSK signal is boosted and then attenuated by a VOA to get the desired launch power. Then it propagates inside a dispersion unmanaged optical recirculating loop which consists of 4 spans of 80 km of low loss standard single mode fiber (SMF-28e + LL) with an attenuation of 0.18 dB/km and an inline EDFA with around 5 dB noise figure in each span.

At the receiver, a noise loading EDFA followed by a VOA is employed to adjust received signal optical signal-to-noise ratio (OSNR). An OSA is used to measure the signal OSNR, which is converted to 0.1 nm noise bandwidth. A BPF with a 3 dB bandwidth of 0.25 nm and 20 dB bandwidth of 0.425 nm is used to reject out-of-band ASE noise accumulated in the transmission. The gain of the pre-amplifier is set such that the signal power hitting the coherent receiver always stays at 5 dBm. The out-of-band ASE noise of the pre-amplifier is filtered using a 0.8 nm BPF. In the coherent receiver, a ${90}^{\text{o}}$ optical hybrid mixes the filtered signal with a 15.5 dBm CW light from the LO laser, which is an ECL having the same specification as the transmitter one. After balanced detection, two real-time oscilloscopes running at 80 GSa/s with 8 bit resolution are used to capture 4 sets of waveforms from the optical front-end. The receiver-side DSP starts with the optical front-end compensation, which consists of the DC removal, IQ imbalance compensation and hybrid IQ orthogonality compensation using the Gram-Schmidt orthogonalization procedure [10]. The signal is resampled down to 2 samples per symbol before the frequency domain CD compensation. Timing recovery is performed using the square and filter method [20] to obtain an accurate sampling phase. The signal is then match filtered with the same pulse shaping FIR filter in the transmitter. Blind equalization is done using a 15-tap fractionally spaced (T/2) butterfly filters with the constant-modulus algorithm (CMA) [21] for the filter coefficient adaptation. The frequency offset compensation is done based on the FFT of the signal at the 4th power. The carrier phase is recovered using the superscalar parallelization based phase locked loop (PLL) combined with a maximum likelihood phase estimation [22]. The signal is not differentially decoded. The BER is directly counted over 2^{18} bits and a hard decision forward error correction (HD FEC) (7% overhead) BER threshold of $3.8\times {10}^{-3}$ is used.

The back-to-back BER versus the OSNR performance for the RRC pulse, ROP and RMP with a $\beta =0.5$ is shown in Fig. 9(a). The required OSNR to reach the HD FEC threshold for the RRC, ROP and RMP is 9.69 dB, 10.09 dB and 10.15 dB, respectively, corresponding to an implementation OSNR penalty of 0.69 dB, 1.09 dB and 1.15 dB, respectively, compared to theory. Since the ROP and RMP have more frequency components at high frequencies, they suffer more from the larger implementation noise introduced by the degraded effective number of bits (ENOB) of the DACs at high frequencies, thereby having a slightly larger OSNR penalty in the back to back scenario.

As we mentioned before, the $\beta $ factor is a tradeoff between linear and nonlinear performance. To quantify the effect of the $\beta $ factor, we measured the required OSNR to reach the HD FEC threshold at each launch power for the RMP with different $\beta $ factors, as shown in Fig. 9(b). The transmission distance is fixed at 4160 km. At low launch powers, the required OSNR is determined primarily by linear impairments. It can be seen from the figure that at low launch powers the RMPs with smaller$\beta $ factors usually require larger OSNRs to achieve the same BER performance. This can be explained by the fact that these pulses have more energy contents at high frequencies and therefore suffer more penalties from the reduced ENOB at high frequencies. At high launch powers intra-channel nonlinearity becomes the dominating factor. The RMPs with smaller$\beta $ factors show better nonlinearity tolerance because there are more energy contents in the higher frequencies that leads to more rapid pulse broadening and hence a reduction in intra-channel nonlinearity. Taking into consideration both linear and the nonlinear performance, we can see the RMP with a $\beta =0.5$ yields the best overall performance for the 14 GBaud PDM-QPSK system.

Figure 9(c) compares the required OSNR versus the launch power performance between the RRC pulse, ROP and RMP with a$\beta =0.5$at 4160 km. At low launch powers, e.g., −6 dBm, all three pulses have a similar required OSNR. As the launch power goes beyond −6 dBm, the RRC pulse and ROP require more OSNR to achieve the same BER performance than the RMP as nonlinearity becomes the dominant impairment. We use the tolerable launch power at 1 dB OSNR penalty as a metric to quantify the nonlinearity tolerance. The tolerable launch power for the RRC pulse, ROP and RMP are −3.9 dBm, −2.5 dBm and −0.9 dBm, respectively. The higher tolerable launch power confirms that the RMP has enhanced nonlinearity resilience.

Nonlinearity tolerance can also be evaluated by the maximum transmission reach. Figure 9(d) shows the maximum reach for single channel PDM-QPSK systems using different pulses. By using the RMP with a $\beta =0.5$, the maximum transmission distance can reach 8960 km at the HD FEC threshold, extending the maximum reach by 33% and 17% (2240 km and 1280 km in the absolute transmission length) compared to the systems using the RRC pulse and ROP, respectively. Optimal launch power at the maximum reach is another way to assess the nonlinearity performance. The optimal launch powers are −4.5 dbm, −3.5 dBm and −2 dBm for the RRC pulse, ROP and RMP PDM-QPSK systems respectively. Again, this validates the RMP outperforms the other two pulses in terms of nonlinearity tolerance.

#### 3.2 Single-channel PDM-16QAM experiment

The experimental setup for the 128 Gb/s 16 GBaud PDM-16QAM system is similar to that of the PDM-QPSK system except for following changes: in the transmitter DSP side, the source signal is 4-levels instead of 2-levels; in the recirculating loop, we used 3 spans of 80 km SMF-28e + LL to increase the resolution of the transmission distance for the maximum reach experiment; in the receiver offline DSP, blind equalization is done using 15-tap fractionally spaced (T/2) butterfly filters with the multi-modulus algorithm (MMA) [23] for the filter coefficient adaptation. The BER is directly counted over 2^{19} bits and a SD FEC BER threshold of $2\times {10}^{-2}$ with a 20% FEC overhead is assumed.

The back-to-back performance for the RRC pulse, ROP and RMP with a $\beta =0.75$ is shown in Fig. 10(a). The OSNR penalty for the RRC pulse, ROP and RMP is about 1.8 dB, 2.45 dB and 2.65 dB, respectively, compared to theory. The OSNR penalty difference between the RRC pulse and RMP is 0.85 dB, which is about 0.39 dB larger than that in the PDM-QPSK case. This is directly related to the fact that the 16QAM format has a higher requirement on the DAC’s ENOB and therefore the PDM-16QAM with the RMP shaping suffers more penalties due to the ENOB degradation in the high frequencies.

Figure 10(b) shows the required OSNR versus the launch power performance comparison for the RMP with different $\beta $ factors at 1200 km. Due to the larger OSNR penalty in the linear regime, the RMP with a $\beta =0.5$ is no longer the best pulse. Instead, the RMP with a$\beta =0.75$ is the optimal pulse in the PDM-16QAM case.

The required OSNR versus the launch power performance for the RRC pulse, ROP and RMP with a $\beta =0.75$ is depicted in Fig. 10(c). It can be seen that at 1 dB OSNR penalty, the tolerable launch power for the RRC pulse, ROP and RMP are −2.6 dBm, −1.1 dBm and 0.9 dBm, respectively. The improved tolerable launch power achieved by using the RMP is 3.5 dB and 2 dB, compared to the RRC pulse and ROP respectively. In the previous PDM-QPSK case, these values are 3 dB and 1.6 dB. This indicates that the RMP is even more effective in enhancing the nonlinearity tolerance for the PDM-16QAM system, compared to the PDM-QPSK case. One explanation for this is that the 16QAM’s multi-level signaling introduces pattern dependent phase shift via the Kerr nonlinearity. When a pulse is spread wide enough to interact with multiple adjacent pulses, the mean nonlinear phase shift tends to be constant and independent from the pulse pattern, due to the law of large numbers. As the RMP disperses more rapidly than the RRC pulse and ROP, it can help inhibits the pattern dependent phase shift and improve the 16QAM nonlinearity tolerance.

Figure 10(d) illustrates the maximum reach for the single-channel PDM-16QAM systems using different pulses. Owing to the strong nonlinearity tolerance, the system using the RMP with a $\beta =0.75$ can considerably extend the maximum reach by 44% and 18% (960 km and 480 km in the absolute transmission length), compared to that using the RRC pulse and ROP, respectively. The maximum transmission distance can reach 3120 km at the SD FEC threshold.

#### 3.3 DWDM PDM-QPSK experiment

Figure 11(a) depicts the experimental setup of the DWDM 14 GBaud PDM-QPSK system. The DWDM system consists of 9 channels spaced by 50 GHz. The laser source for the central channel is an ECL and the remaining 8 channels are DFBs. The 9 channels are combined using an AWG and bulk modulated in a PDM-QPSK transmitter as we described in section 3.1. They are interleaved into odd and even channels. An ODL is added in the path of the even channels to de-correlate them from the adjacent odd channels when the odd and even channels are re-combined using another interleaver. In the recirculating loop, a wave-shaper is inserted between the second EDFA and the third span as a gain flattening filter. The rest of the setup resembles the single-channel setup and all the measurements are performed on the central channel. The spectrum of the 9 channels DWDM signal is given in Fig. 11(b). As the RRC pulse puts most of its power in its central frequency components, it has a large peak power spectral density in the central frequencies when measured with a small resolution bandwidth. We also observe a slightly worse out-of-band attenuation for the RMP. However, after transmission, the dominant noise sources are the ASE noise and nonlinear noise. The impact from the slightly worse out-of-band attenuation is negligible.

The back to back performance of the RRC pulse, ROP and RMP with a $\beta =0.5$ is shown in Fig. 12(a). The OSNR penalty for the RRC pulse, ROP and RMP is about 1.22 dB, 1.43 dB and 1.86 dB, respectively, compared to theory. The OSNR penalty difference between the RRC pulse and RMP is 0.64 dB, which is slightly larger than the 0.46 dB in the single-channel PDM-QPSK case. This is attributed to the slightly larger inter-channel crosstalk from the slight worse out-of-band attenuation of the RMP.

Figure 12(b) shows the required OSNR versus the launch power per channel performance comparison for the RMP with different $\beta $ factors at 4160 km. It can be seen that, the RMP with a $\beta =0.5$ has the best performance considering both in the linear and nonlinear regions.

We investigated the performance comparison for the RRC pulse, ROP and RMP with a$\beta =0.5$ in Fig. 12(c). It can be seen that at 1 dB OSNR penalty, the tolerable launch power for the RRC pulse, ROP and RMP are −4.1 dBm, −2.8 dBm and −1.2 dBm, respectively. Compared to the single-channel case, the tolerable launch powers for all the pulses are reduced due to the presence of cross-phase modulation (XPM) effect.

The maximum reach performance for the DWDM PDM-QPSK systems using different pulses is illustrated in Fig. 12(d). By using the RMP with a $\beta =0.5$, the maximum transmission distance can reach up to 8320 km at the HD FEC threshold, extending the maximum reach by 30% and 13% (1920 km and 960 km in the absolute transmission length), compared to the systems using the RRC pulse and ROP. This demonstrates that the RMP can effectively extend the transmission reach for the DWDM systems.

Table 1 summarizes the aforementioned improved nonlinearity tolerance achieved by the proposed pulse for different modulation formats.

#### 3.4 Filtering tolerance experiment

In a transmission system, reconfigurable optical add-drop multiplexers (ROADM) will inevitably impose filtering effects on the spectrum of the signal. Therefore, in this section, we compare the narrow filtering tolerance of different pulses for a 14 GBaud PDM-QPSK signal. In order to compare the OSNR penalty from the filtering effect only, we conduct the experiment in the back-to-back scenario. The position of noise loading needs to be carefully chosen. If the noise loading is placed before the filter, the noise distribution after the digital adaptive equalizer would not be altered as long as the equalizer precisely inverse the filter’s frequency response. Therefore, the presence of the filter has no impact on the SNR after the equalizer. While if the noise loading is placed after the filter, the noise distribution would be altered by the equalizer when it tries to inverse the filter’s frequency response, leading to noise enhancement and a degraded SNR at the equalizer output. In real systems, the noise is introduced before and after the ROAMs along the link. Putting all noise loading after the filter leads to the strongest noise enhancement and represent the worst scenario. According to [24], performing noise loading after the filter matches well with the experiment and simulation results. Hence, we choose noise loading after the filter, as shown in the schematic of the experimental setup in Fig. 13. The different filtering effects are emulated by varying the bandwidth of the T-T filter.

Figure 14 shows the OSNR penalty versus T-T filter’s 3 dB bandwidth over the signal’s 3 dB bandwidth for different pulses. The signal’s 3 dB bandwidth is 0.225nm. The OSNR is measured by taking the signal power before the filter over the power of the loaded noise. When the filter’s 3 dB bandwidth equals the signal’s 3-dB bandwidth, the filter’s impact on the system is negligible. We use the required OSNR at HD FEC in this case as the reference OSNR to calculate the required OSNR penalty when the filter’s bandwidth reduces. It can be seen that, the ROP and RMP generally have a larger OSNR penalty than the RRC pulse. The reason is that they have more energy in the higher portion of the spectrum. Therefore, more signal power is cut off in the presence of narrow filtering, leading to a reduced OSNR at the filter output. One solution would be using a RMP with a larger $\beta $factor as it put less energy contents in the high frequency components. Nevertheless, this will also diminish the nonlinear tolerance. Therefore, an optimal $\beta $factor needs to be derived depending on the filtering effect and nonlinear effect in the link.

## 4. Conclusion

In this paper, we have proposed a new ISI-free frequency domain root M-shaped pulse and derived its closed-form expressions. Experimental verifications confirm the proposed pulse achieves an excellent intra-channel nonlinearity tolerance as it contains more energy at the high frequencies components, leading to a rapidly dispersing property. In the presence of the degraded ENOB of the DACs at the higher frequency or strong filtering effect in the transmission link, the depth factor of the proposed pulse can also be adjusted so that a good balance of linear and nonlinear performance can be reached.

## References and links

**1. **R. Essiambre, G. Kramer, P. Winzer, G. Foschini, and B. Goebel, “Capacity limits of optical fiber networks,” J. Lightwave Technol. **28**(4), 662–701 (2010). [CrossRef]

**2. **Y. M. Greshishchev, D. Pollex, S.-C. Wang, M. Besson, P. Flemeke, S. Szilagyi, J. Aguirre, C. Falt, N. Ben-Hamida, R. Gibbins, and P. Schvan, “A 56GS/S 6b DAC in 65nm CMOS with 256×6b memory,” in Proc. ISSCC2011, pp. 194–196.

**3. **C. Laperle, “Advances in high-speed ADC, DAC, and DSP for optical transceiversben,” in Proc. OFC2013, paper OTh1F.5.

**4. **E. Ip and J. Kahn, “Compensation of dispersion and nonlinear impairments using digital backpropagation,” J. Lightwave Technol. **26**(20), 3416–3425 (2008). [CrossRef]

**5. **L. B. Du and A. J. Lowery, “Improved single channel backpropagation for intra-channel fiber nonlinearity compensation in long-haul optical communication systems,” Opt. Express **18**(16), 17075–17088 (2010). [CrossRef] [PubMed]

**6. **T. Hoshida, L. Dou, W. Yan, L. Li, Z. Tao, S. Oda, H. Nakashima, C. Ohshima, T. Oyama, and J. Rasmussen, “Advanced and feasible signal processing algorithm for nonlinear mitigation,” in Proc. OFC2013, paper OTh3C.3. [CrossRef]

**7. **Y. Gao, J. C. Cartledge, J. D. Downie, J. E. Hurley, D. Pikula, and S. S.-H. Yam, “Nonlinearity compensation of 224 Gb/s dual polarization 16-QAM transmission over 2700 km,” IEEE Photon. Technol. Lett. **25**(1), 14–17 (2013). [CrossRef]

**8. **C. Behrens, S. Makovejs, R. I. Killey, S. J. Savory, M. Chen, and P. Bayvel, “Pulse-shaping versus digital backpropagation in 224Gbit/s PDM-16QAM transmission,” Opt. Express **19**(14), 12879–12884 (2011). [CrossRef] [PubMed]

**9. **S. Makovejs, E. Torrengo, D. Millar, R. Killey, S. Savory, and P. Bayvel, “Comparison of pulse shapes in a 224Gbit/s (28Gbaud) PDM-QAM16 long-haul transmission experiment,” in Proc. OFC2011, paper OMR5. [CrossRef]

**10. **J. G. Proakis, *Digital Communications,* 4th ed. (McGraw Hill, 2001).

**11. **B. Châtelain, C. Laperle, D. Krause, K. Roberts, M. Chagnon, X. Xu, A. Borowiec, F. Gagnon, J. C. Cartledge, and D. V. Plant, “SPM-tolerant pulse shaping for 40- and 100-Gb/s dual-polarization QPSK systems,” IEEE Photon. Technol. Lett. **22**, 1641–1643 (2010).

**12. **B. Châtelain, C. Laperle, K. Roberts, M. Chagnon, X. Xu, A. Borowiec, F. Gagnon, and D. V. Plant, “A family of Nyquist pulses for coherent optical communications,” Opt. Express **20**(8), 8397–8416 (2012). [CrossRef] [PubMed]

**13. **X. Xu, B. Châtelain, Q. Zhuge, M. Morsy-Osman, M. Chagnon, M. Qiu, and D. Plant, “Frequency domain M-shaped pulse for SPM nonlinearity mitigation in coherent optical communications,” in Proc. OFC2013, paper JTh2A.38. [CrossRef]

**14. **P. V. Mamyshev and N. A. Mamysheva, “Pulse-overlapped dispersion-managed data transmission and intrachannel four-wave mixing,” Opt. Lett. **24**(21), 1454–1456 (1999). [CrossRef] [PubMed]

**15. **H. Nyquist, “Certain topics in telegraph transmission theory,” Trans. Am. Inst. Electr. Eng. **47**(2), 617–644 (1928). [CrossRef]

**16. **A. Carena, V. Curri, G. Bosco, P. Poggiolini, and F. Forghieri, “Modeling of the impact of nonlinear propagation effects in uncompensated optical coherent transmission links,” J. Lightwave Technol. **30**(10), 1524–1539 (2012). [CrossRef]

**17. **M. Abramowitz and I. A. Stegun, *Handbook of Mathematical Functions: With Formulars, Graphs, and Mathematical Tables* (Dover, 1964).

**18. **A. V. Oppenheim and R. W. Schafer, *Discrete-Time Signal Processing,* 3rd ed. (Prentice Hall, 2009).

**19. **G. Agrawal, *Nonlinear Fiber Optics,* 4th ed. (Academic, 2006).

**20. **M. Oerder and H. Meyr, “Digital filter and square timing recovery,” IEEE Trans. Commun. **36**(5), 605–612 (1988). [CrossRef]

**21. **D. Godard, “Self-recovering equalization and carrier tracking in two-dimensional data communication systems,” IEEE Trans. Commun. **28**(11), 1867–1875 (1980). [CrossRef]

**22. **Q. Zhuge, M. Morsy-Osman, X. Xu, M. E. Mousa-Pasandi, M. Chagnon, Z. A. El-Sahn, and D. V. Plant, “Pilot-aided carrier phase recovery for M-QAM using superscalar parallelization based PLL,” Opt. Express **20**(17), 19599–19609 (2012). [CrossRef] [PubMed]

**23. **J. Yang, J.-J. Werner, and G. A. Dumont, “The multimodulus blind equalization and its generalized algorithms,” IEEE J. Sel. Areas Comm. **20**(5), 997–1015 (2002). [CrossRef]

**24. **P. J. Winzer, A. H. Gnauck, C. R. Doerr, M. Magarini, and L. L. Buhl, “Spectrally Efficient Long-Haul Optical Networking Using 112-Gb/s Polarization-Multiplexed 16-QAM,” J. Lightwave Technol. **28**(4), 547–556 (2010). [CrossRef]