## Abstract

We experimentally investigate the singularity problem in DP-QPSK 112-Gb/s receivers using the CMA. Three algorithms are compared: Constrained, Two-Stage, and Multi-User. Although these algorithms have been individually evaluated, they have not been compared by extensive experiments. The transmission setup emulates amplifier noise; first-order PMD; and chromatic dispersion. It is shown that all algorithms effectively mitigate singularities. However, under certain conditions, the Multi-User and the Constrained algorithms – both used for system startup – outperformed the Two-Stage, which does not distinguish between system operation and startup. In light of its effectiveness and low computational complexity, we recommend the Constrained algorithm.

©2011 Optical Society of America

## 1. Introduction

Dual-polarization quadrature phase shift keying (DP-QPSK) optical systems with coherent detection are becoming a de facto standard for 100-Gb/s long haul systems. The reason for this remarkable success is the seamless compensation of linear fiber distortions using signal processing techniques. Among them, the constant modulus algorithm (CMA) has been the favorite choice to blindly update the equalizer coefficients, which process inputs *x*
_{1} and *x*
_{2} to produce outputs *y*
_{1} and *y*
_{2}. The equalizer is usually represented by matrix **F** = [**f**
* _{ij}*],

*i*,

*j*= 1, 2, where ${\mathbf{\text{f}}}_{ij}\hspace{0.17em}=\hspace{0.17em}{\left[{f}_{ij}^{1},\hspace{0.17em}\dots ,\hspace{0.17em}{f}_{ij}^{M}\right]}^{T}$ are the M-tap filter weights. Alternatively, the equalizer structure can be expressed by the four vectors

**f**

*arranged in the so-called “butterfly structure”. The “Standard CMA” is implemented in a way that filters*

_{ij}**f**

_{11}and

**f**

_{12}are driven by output

*y*

_{1}, while

**f**

_{21}and

**f**

_{22}by output

*y*

_{2}. Since the statistical properties on which CMA is based are equal for input signals

*x*

_{1}and

*x*

_{2}, and since both sets of filters are adjusted independently, the recovered signals are subject to delay and permutation ambiguity [1]. Therefore, the equalizer can recover correctly both signals, or it can recover the same signal for both outputs, or else, a time shifted version of the same signal. The last two cases are known in the literature as the “singularity problem” [1–4]. It is well-known that the choice of the filter initial tap setup affects the filter convergence properties [1, 4] and, consequently, the occurrence of singularities. Indeed, it has been a common practice to use a “single spike” initialization [5], in which the central coefficients of

**f**

_{11}and

**f**

_{22}are set to “1” and the remaining coefficients to “0”. The single spike initialization itself already helps mitigating the singularity problem.

There have been several works addressing the CMA singularity problem in optical systems. References [1, 3, 6] suggest to impose constraints on the equalizer coefficients, based on a fundamental result of [7]. The result states that the transfer function of an optical system composed of rotators and retarders is unitary. Although such relationship applies to optical fiber channels with certain distortions, e.g. polarization mode dispersion (PMD), it does not hold if chromatic dispersion (CD) and polarization dependent loss (PDL) are the limiting effects. In addition, if included as parts of the communication channel, pulse shaping and fractionally-spaced sampling may also affect the symmetry of the channel coefficients. In particular, [6] proposes to adjust filters **f**
_{11} and **f**
_{12} using the CMA, while inferring **f**
_{21} and **f**
_{22} from **f**
_{11} and **f**
_{12}. In a further development, [1] suggests using the same approach to initialize the filter coefficients, but removing the constraint after convergence is achieved. We denote this approach as being the “Constrained CMA”. The “Two-Stage CMA” proposed in [3], in turn, uses a two-stage butterfly filter with a constrained first stage, and an unconstrained second stage. While the Constrained CMA is less complex, the Two-Stage CMA allows advanced performance monitoring, and does not need to distinguish between startup and tracking phases. In a different approach, [8] proposes to extend the Standard CMA cost function by adding the cross-correlation between outputs, thus avoiding singularities. Known as “Multi-User CMA” (MU-CMA), the algorithm was also investigated in optical communications [9].

In this paper we assess the performance of the three algorithms – Constrained CMA, Two-Stage CMA and MU-CMA – in terms of singularity occurrence, and compare their performance to that of the Standard CMA with single-spike initialization. Although they have been individually evaluated, to our knowledge, this is the first time they are compared using extensive experiments. We explore the individual impacts of amplifier noise, differential group delay (DGD) and residual chromatic dispersion (CD).

## 2. Singularity avoidance techniques

#### 2.1. Constrained CMA

The Constrained CMA [1] relies on a fundamental result obtained by Jones in [7]. The result states that the transfer function **T**(*jω*) of an optical system composed of retarders and rotators is unitary, i.e.:

Based on the symmetry of Eq. (1), the algorithm proposes to, first, let filters **f**
_{11} and **f**
_{12} be adapted by the CMA applied to output *y*
_{1}. After *y*
_{1} converges to a constant modulus, the algorithm sets **f**
_{22} = –*TR*[**f**
_{11}]^{*} and **f**
_{21} = –*TR*[**f**
_{12}]^{*}, where *TR*[**f**
* _{ij}*] means time-reversed

**f**

*, i.e., $TR[{\mathbf{\text{f}}}_{ij}]\hspace{0.17em}=\hspace{0.17em}{\left[{f}_{ij}^{M},\hspace{0.17em}\dots ,\hspace{0.17em}{f}_{ij}^{1}\right]}^{T}$. The new equalizer matrix is then used as the initial condition for the subsequent continuous operation using the Standard CMA, where, again,*

_{ij}**f**

_{11}and

**f**

_{12}are driven by

*y*

_{1}, and

**f**

_{21}and

**f**

_{22}by

*y*

_{2}. Note however that, although Eq. (1) applies to optical fiber channels with certain distortions (e.g. PMD), it may not hold for the complete channel model, including pulse shaping and fractionally-spaced sampling. In addition, PDL and CD also do not fit in the hypotheses behind Eq. (1). In practice, however, even if the symmetry of Eq. (1) for the complete channel is not fulfilled, previous works [1] have observed that the filter initialization by Constrained CMA drives the equalizer output to a point that, although sub-optimal, lies in the vicinity of the singularity-free optimal solution. Thus, after initialization, the independent adaptation of the four filter coefficients leads the equalizer, finally, to the singularity-free optimal solution. These observations are corroborated by the experimental results in this paper.

#### 2.2. Two-stage CMA

The Two-Stage CMA [3] is a concatenation of two CMA-based “butterfly” filters with equalizer matrices **F1** and **F2**, where constraints **f1**
_{22} = –*TR*[**f1**
_{11}]^{*} and **f1**
_{21} = –*TR*[**f1**
_{12}]^{*} are imposed to the first stage. The second stage updates filters **f2**
_{11} and **f2**
_{12}; and **f2**
_{21} and **f2**
_{22} independently, as in the Standard CMA. Although the first stage produces T/2-spaced samples, the filter coefficients are adjusted every two samples. The second stage, in turn, processes T/2-spaced samples to produce T-spaced samples, as in the Standard CMA. This special structure also allows for additional performance monitoring and parameter estimation [3]. Even though the Two-Stage CMA and the Constrained CMA rely on the same property, they have fundamental differences: the Two-Stage CMA does not distinguish between initialization and operation, and its second stage always run freely. Thus, there are two possible outcomes. First, the first stage drives the equalizer to a sub-optimal solution, and the second stage produces an optimal, singularity-free solution. Second, the first stage drives the equalizer to a sub-optimal solution, and the second stage leads the solution back to a singularity. The final outcome depends on the interaction of noise and other fiber effects during the equalizer convergence phase.

#### 2.3. MU-CMA

The Standard CMA attempts to bring the equalized constellation symbols to a circumference of unitary radius. The CMA cost function *J*(*F*) for a two-user case was originally defined as:

The MU-CMA improves the Standard CMA to avoid the singularity problem. In essence, it tracks the cross-correlation between equalized signals and augments the CMA cost function by an additional correlation term:

*r*(

_{ij}*δ*) is the cross-correlation function between outputs

*y*

_{1}and

*y*

_{2}, defined as ${r}_{ij}(\delta )\hspace{0.17em}=\hspace{0.17em}E\hspace{0.17em}\left[({y}_{i}(k){y}_{j}^{*}(k-\delta ))\right]$,

*δ*∈ ℕ. In practice, values

*δ*

_{1}and

*δ*

_{2}are bounded by the equalizer length. It can be shown that

*J*is minimized using following the stochastic gradient algorithm:

_{MU–CMA}*μ*is the step size, and Δ

*[*

_{l}*k*] is given by:

In Eq. (5)
**X**[*k*] = [**x**
_{1}[*k*] **x**
_{2}[*k*]]* ^{T}*, where

**x**

_{1}[

*k*] and

**x**

_{2}[

*k*] are line vectors with

*M*fractionally spaced input samples. In practice, the exact cross-correlation

*r*(

_{ij}*δ*) is not available, and must be estimated from the data. In this paper we implemented the MU-CMA algorithm as in Fig. 1(a). The filter coefficients are updated periodically, but not in every symbol, to allow a sufficiently accurate estimation of

*r*(

_{ij}*δ*) from a limited size set of symbols, called correlation estimation vector (CEV). The calculation applies for 0 ≤

*δ*≤ ⌊

*M*/2⌋, which, using a T/2 spaced equalizer, corresponds to the possible delays between

*y*

_{1}and

*y*

_{2}. The cross-correlation estimation schemes for the minimum (

*δ*= 0) and maximum (

*δ*= ⌊

*M*/2⌋) delays are depicted in Figs. 1(b) and 1(c), where the arrows indicate multiplications involved in the estimation process. Note that the case where

*δ*= ⌊

*M*/2⌋ requires the interval between consecutive filter iterations to be at least CEV length + ⌊

*M*/2⌋ symbols long.

Although the MU-CMA has been developed and evaluated for continuous operation [9], we propose here to use it during the convergence phase only, switching later to the Standard CMA. Using the MU-CMA for system startup only, avoids the complexity of computing cross-correlations during normal system operation. Accordingly, our tests with experimental data showed that increasing the number of MU-CMA iterations over 5, 000, before switching to the Standard CMA, does not affect the number of singularities, validating this approach.

## 3. Experimental setup

A pulse pattern generator produces four 2^{11} mutually delayed pseudo-random bit sequences (PRBS) at 7 Gb/s, multiplexed later into one sequence at 28 Gb/s. This sequence and its inverted version are split and delayed by 52 bits to form the in-phase (I) and quadrature (Q) components that drive two nested MZM based modulators, whose optical input is a tunable DBF laser. The resulting QPSK signal is then submitted to an RZ pulse carver. A block that consists of a polarization beam splitter, a 300 symbols delay line polarization maintaining fiber in one of the branches, and a polarization beam combiner, produces the polarization multiplexed signal. A polarization scrambler, a first order PMD emulator and a tunable CD compensator generate the desired transmission conditions. On the receiver side, optical noise is loaded to fine-tune the OSNR. The polarization diversity coherent receiver contains four pairs of balanced photodetectors. The four eletrical outputs are sampled at 50 GSamples/s by a 4 channel oscilloscope (8 bits nominal resolution) and then stored for offline post-processing.

## 4. Experimental results and analysis

The following curves show the experimental results in terms of the number of singularities in 1, 000 randomly polarized data sequences of 10^{6} samples. During offline processing we submitted the data sequences to normalization and orthogonalization procedures, followed by resampling by a factor of 28/25, to obtain the rate of 2 Samples/symbol. After equalization and source separation by the tested filter, we performed carrier recovery, decision and decoding. We then computed the cross-correlation between two decoded bit sequences for delays up to half filter length, setting the threshold for singularity occurrence to 0.5, computed over 50,000 bits. We repeated this procedure 1, 000 times with different data strings for each tested transmission condition. The filter length was set to 15 taps for all tested algorithms (the Two-Stage CMA used 15 taps in each stage). The Constrained and MU-CMA equalizers used a convergence phase of 5, 000 iterations before switching to the Standard CMA (see Fig. 2). In order to assess the convergence properties of the investigated algorithms, the signal to noise ratio (SNR) of the equalizer constellation was estimated from the data [10]. Data analysis for all investigated algorithms was performed over 50,000 symbols, between outputs 145,000 and 195,000. This is dictated by the MU-CMA, whose CEV length was set to 20 symbols, yielding an initialization phase of 5,000 iterations × [20 (CEV length) + 7 (⌊*M*/2⌋)]=135,000 symbols. Additional 10,000 symbols were included for the Standard CMA convergence, giving a total of 145,000 symbols used for convergence and initialization.

Table 1 shows the influence of the CEV length on the occurrence of singularities when using the MU-CMA. Here, the only source of distortion is amplifier noise, resulting in an OSNR = 17 dB. Clearly, a CEV length of 20 symbols is enough to guarantee a singularity-free operation when amplifier noise is the only source of distortion. Besides suffering from singularities, the MU-CMA with short CEV lengths (< 14) may also lead to poor equalizer convergence. This is evidenced by Table 2, which exhibits the number of convergence failures for the same setup (we considered as convergence failure an estimated SNR below 4 dB). It is worth mentioning that a convergence failure is tested negative for singularities. In light of the results in Tables 1 and 2, we used a CEV length of 20 throughout the rest of the paper.

For reference purposes, Figs. 3(a) and 3(b) relate BER, measured OSNR and estimated SNR for the Standard CMA with amplifier noise only. The polynomial fit in Fig. 3(b) was obtained by a first order fitting using the least squares criterion. An estimated SNR = 7.5 dB corresponds to a measured OSNR = 16.0 dB that, in turn, yields a BER = 7 · 10^{–4}.

The scatter plots in Figs. 4(a) and 4(b) show the estimated SNR for the *y*
_{1} and *y*
_{2} outputs in presence of amplifier noise at OSNR = 17 dB. Figs. 4(c) and 4(d) show the box plots for the same data, indicating the smallest observation, lower quartile, median, upper quartile, and largest observation. The Two-Stage algorithm exhibits the highest median for the estimated SNR in *y*
_{1}. This is an artifact from using two 15-taps filters, while the others use a single 15-taps filter. However, one striking remark is that the estimated SNR in *y*
_{2} is in average lower than in *y*
_{1}. This indicates that the algorithm privileges the orientation that is used for adapting the first stage. We can also conclude that the symmetry relationship employed in the first stage is not optimal, and some mismatch has to be compensated in the second stage.

Figures 5(a) and 5(b); and Figs. 5(c) and 5(d); show the estimated SNR and the corresponding box plots for *y*
_{1} and *y*
_{2} for the system under residual chromatic dispersion, at CD = 50 ps/nm, OSNR = 18 dB. The results confirm the conclusions obtained when amplifier noise was the only source of distortion. The convergence properties for the system impaired by first-order PMD is shown in Figs. 6(a) and 6(b); and Figs. 6(c) and 6(d) (DGD = 20 ps, OSNR = 18 dB). One can observe a single case where the Standard CMA and the Constrained CMA equalizers did not attain convergence for *y*
_{1}. Such *outliers* were not included in the corresponding box plot to maintain the figure scaling. Once again, the MU-CMA outperformed the remaining algorithms for output *y*
_{1}, however, its behavior for *y*
_{2} was slightly inferior.

Since convergence failures are tested negative for singularities, we must investigate their occurrence, before proceeding with the singularity analysis itself. Table 3 summarizes the number of convergence failures in all scenarios investigated in the paper. Just a few spurious cases were observed for three algorithms (MU-CMA, Standard CMA and Constrained CMA) subject to PMD (8 occurrences in 11.000 sequences), confirming the validity of the singularity analysis that follows.

Table 4 shows the number of observed singularities when amplifier noise is the only source of distortion (OSNR = 17 dB). It is interesting to observe that the single-spike initialization alone already yields a low singularity probability (25 singularities over 1,000 sequences). Accordingly, if properly dimensioned (MU-CMA with CEV length ≥ 20), the three singularity-avoidance algorithms bring the number of observed singularities to values close to zero.

Figure 7(a) shows the number of observed singularities for various residual CD values. Increasing residual CD did not have a noteworthy impact on the occurrence of singularities. Figure 7(b) presents the number of observed singularities for several DGD values. The singularity properties of the Standard CMA with single-spike, and the Two-Stage CMA, revealed a clear dependence on the DGD. It is interesting to note that the number of singularities increased as the DGD became comparable with half of the single period, and decreased again as it approached a full symbol period. This indicates that, although the first stage using the matrix symmetry effectively leads the equalizer to a singularity-free condition, a second stage may bring the equalizer back to an undesired operation point. Lastly, it is worth pointing out that all algorithms exhibited singularities, confirming their stochastic nature.

## 5. Conclusion

We have performed a detailed experimental analysis of three singularity-avoidance techniques under linear transmission effects, for a 112-Gb/s DP-QPSK optical system. All investigated algorithms effectively mitigated singularities. However, under certain conditions, the Multi-User and Constrained algorithms, used for system startup, outperformed the Two-Stage, which does not distinguish between system operation and startup. In light of its low computational complexity, we recommend the Constrained algorithm. We also found that there is some statistical uncertainty present in these algorithms, so that total singularity avoidance cannot be ensured.

## Acknowledgment

This work was supported by the Innovation Center, Ericsson Telecomunicações S.A., Brazil.

## References and links

**1. **L. Liu, Z. Tao, W. Yan, S. Oda, T. Hoshida, and J. C. Rasmussen, “Initial tap setup of constant modulus algorithm for polarization de-multiplexing in optical coherent receivers,” OSA/OFC/NFOEC(2009).

**2. **S. Faruk, Y. Mori, C. Zhang, and K. Kikuchi, “Proper polarization demultiplexing in coherent optical receiver using constant modulus algorithm with training mode,” OptoElectron. and Commun. Conf. Tech. Digest(2010).

**3. **C. Xie and S. Chandrasekhar, “Two-stage constant modulus algorithm equalizer for singularity free operation and optical performance monitoring in optical coherent receiver,” OSA/OFC/NFOEC pp. 1–3 (2010).

**4. **S. J. Savory, “Digital coherent optical receivers: algorithms and subsystems,” IEEE J. Quantum Electron. **16**, 2120–2126 (2010).

**5. **J. Johnson, R., P. Schniter, T. Endres, J. Behm, D. Brown, and R. Casas, “Blind equalization using the constant modulus criterion: a review,” Proc. IEEE **86**, 1927–1950 (1998). [CrossRef]

**6. **K. Kikuchi, “Polarization-demultiplexing algorithm in the digital coherent receiver,” IEEE/LEOS Summer Topical Meetings pp. 101–102 (2008). [CrossRef]

**7. **R. C. Jones, “A new calculus for the treatment of optical systems,” J. Opt. Soc. Am. **31**, 488–493 (1941). [CrossRef]

**8. **C. B. Papadias and A. Paulraj, “A space-time constant modulus algorithm for SDMA systems,” Vehicular Tech. Conf. **1**, 86–90 (1996).

**9. **A. Vgenis, C. S. Petrou, C. B. Papadias, I. Roudas, and L. Raptis, “Nonsingular constant modulus equalizer for PDM-QPSK coherent optical receivers,” IEEE Photon. Technol. Lett. **22**, 45–47 (2010). [CrossRef]

**10. **T. Benedict and T. Soong, “The joint estimation of signal and noise from the sum envelope,” IEEE Trans. Inform. Theory **13**, 447–454 (1967). [CrossRef]