## Abstract

We present a simple theoretical model of transverse mode instability in high-power rare-earth doped fiber amplifiers. The model shows that efficient power transfer between the fundamental and higher-order modes of the fiber can be induced by a nonlinear interaction mediated through the thermo-optic effect, leading to transverse mode instability. The temporal and spectral characteristics of the instability dynamics are investigated, and it is shown that the instability can be seeded by both quantum noise and signal intensity noise, while pure phase noise of the signal does not induce instability. It is also shown that the presence of a small harmonic amplitude modulation of the signal can lead to generation of higher harmonics in the output intensity when operating near the instability threshold.

© 2013 Optical Society of America

## 1. Introduction

Recently, a phenomenon known as Mode Instability, or Transverse Mode Instability (TMI), has emerged as one of the greatest limitations on the power scalability of large mode area (LMA) ytterbium-doped fiber amplifiers [1, 2]. The phenomenon manifests itself as a temporal fluctuation, typically on a ms timescale, of the output beam profile as the output power reaches a certain threshold. Detailed experimental investigations of TMI have shown that the power and relative phase of the light in the fundamental and higher-order modes of the amplifier fluctuates on a timescale which depends strongly on the core diameter of the fiber, and that only the first higher-order mode (HOM) is involved, except for fibers with very large core diameters [3, 4]. While the onset of TMI does not in itself prevent the amplification of the signal beyond the instability threshold, the resulting dramatic decrease in beam quality renders the amplifier useless for applications that require a stable output beam.

An initial attempt to understand the physical mechanism responsible for TMI proposed that a self-induced long-period grating (LPG) could cause a transfer of power from the fundamental mode (FM) to a HOM of the fiber [5]. This LPG is induced by mode beating between the light in the FM and a small amount of light unintentionally coupled into the HOM, since the resulting spatial intensity oscillation creates a matching index grating due to the ytterbium doping in the fiber core. The ytterbium ions can cause the required refractive index perturbation either directly, since their contribution to the refractive index of the doped core depends on the population inversion, which again depends on the local intensity, or indirectly though the thermo-optic effect. In the latter case, the intensity oscillation of the signal leads to spatially varying stimulated emission, which due to the quantum defect creates a spatially oscillating temperature profile in the fiber. Since the refractive index of fused silica depends on temperature, the mode beating between the FM and the HOM again leads to a LPG, which has the correct period to couple the two modes [6]. However, it was shown in [7] that a phase lag between the intensity oscillation and the LPG must exist in order to have an efficient coupling between the modes. Using a beam propagation model, it was shown that such a phase lag can appear if the light in the HOM is slightly redshifted relative to the light in the FM and the slow response time of the thermal nonlinearity is taken into account. By seeding the model with a small amount of light in the HOM redshifted by a few kHz, it was shown that the light in the HOM would experience a large nonlinear gain. Later numerical simulations by Ward *et al.*[8] showed that thermally induced TMI could be initiated by a transient signal and proposed that longitudinal heat flow in the fiber is responsible for the onset of TMI. However, additional simulations by the authors, in which the longitudinal heat flow was neglected, failed to support this hypothesis.

In a recent paper, we developed a semi-analytical model of the thermal nonlinearity in fiber amplifiers and showed that quantum noise could act as a seed for TMI [9]. However, that model did not allow us to study the temporal and spectral characteristics of TMI. In this paper, we extend the method to a full coupled-mode model of thermally induced TMI. While our model makes a number of simplifying assumptions, we nonetheless believe that it explains the most important features of TMI. Our paper is organized as follows: In section 2 we present our theoretical model and discuss the approximations made. In section 3 we consider operation at the threshold and derive an approximate analytic solution of the coupled-mode equations, which is valid when the HOM is only weakly excited. Using this approximate solution, we consider seeding of TMI by quantum noise and signal intensity noise. Through a numerical solution, we investigate the temporal and spectral features of TMI near the threshold as well as the dependence of the threshold power on the spectral width of the signal. In section 4 we consider operation beyond the stability threshold, and study the temporal and spectral dynamics in this case, as well as investigate how the average HOM content of the output signal varies as power is increased. Finally, in section 5, we consider an amplitude modulated input signal and show how the thermally induced nonlinear interaction between the modes leads to the generation of additional sidebands in the spectral characteristics of the TMI phenomenon.

## 2. Theory

#### 2.1. Coupled-mode equations

The quasi-monochromatic electric field of the signal propagating in the fiber is written in terms of a slowly-varying envelope

**u**is the polarization unit vector,

*E*is the temporally slowly-varying field envelope and

*ω*

_{0}is the carrier angular frequency of the signal. Using an analogous expression for the induced polarization, the frequency domain wave equation in the scalar approximation can be written as

*E*(

**r**,

*ω*) is the Fourier transform of

*E*(

**r**,

*t*),

*ε*is the complex relative permittivity of the fiber and

*P*(

_{NL}**r**,

*ω*) is the slowly-varying nonlinear induced polarization due to the heating of the fiber. The relative permittivity is written in terms of its real and imaginary parts as

*ε*is the real relative permittivity of the fiber,

_{f}*g*is the bulk gain coefficient due to the rare-earth doping of the fiber core,

*k*

_{0}=

*ω*

_{0}/

*c*is the vacuum wave number and the subscript ⊥ denotes the transverse coordinates

*x*,

*y*. We have disregarded material dispersion of the fiber, since we shall restrict ourselves to considering signals with a linewidth of less than a few tens of kHz in this paper.

The nonlinear induced polarization is related to the change in temperature of the fiber Δ*T* and the electric field by

*η*is a thermo-optic coefficient, which relates the change in relative permittivity of the fiber to the change in temperature through

*ω*−

*ω*

_{0}and

*k*=

*ω*/

*c*. The change in temperature Δ

*T*obeys the heat equation

*ρ*is the density,

*C*is the specific heat capacity, and

*κ*is the thermal conductivity of the fiber material, all of which are assumed to be constant throughout the fiber cross section and independent of temperature. We have assumed that the longitudinal heat diffusion is negligible compared to the transverse heat diffusion, and hence we have omitted the

*z*derivative part of the Laplacian in the heat equation. The heat source is due to the quantum defect associated with the gain medium, and the heat power density

*Q*is related to the signal intensity

*I*by

*λ*and

_{s}*λ*are the signal and pump wavelengths, respectively. Fourier transforming Eq. (7) with respect to time yields

_{p}*q*=

*iρCω*/

*κ*. Eq. (9) can be solved by an appropriate Green’s function

*G*[10], and Δ

*T*in the frequency domain is given by

*g*is independent of time, and that it is uniform within the rare-earth doped region of the fiber core, which we denote

*S*. Both these assumptions are approximations, since

_{d}*g*is given by the population inversion of the gain medium, which depends on the signal intensity. If the signal field is composed of multiple transverse modes and frequency components, the intensity will oscillate in both space and time, and this will result in spatio-temporal oscillations in

*g*if the signal intensity is high compared to the saturation intensity of the gain medium as shown in [5, 6]. It is clear that the minima of

*g*will coincide with the maxima of

*I*, and by considering Eq. (8) we see that neglecting the spatio-temporal oscillations of

*g*leads to an overestimate of the oscillations in Δ

*T*. Including the effects of gain saturation in our model leads to a much more complicated formalism, which is beyond the scope of this paper. Nevertheless, we shall see that our simplified model explains the major qualitative features of TMI, and also provides quantitative predictions that agree reasonably well with experiments.

To derive coupled-mode equations, we expand the electric field in a set of orthogonal transverse modes

*A*are the slowly-varying mode amplitudes,

_{m}*β*

_{0,m}are the propagation constants of the modes at

*ω*=

*ω*

_{0}and the normalized mode functions satisfy the eigenvalue equation

*β*(

_{m}*ω*) is the propagation constant for mode

*m*. Note that the thermal perturbation of the refractive index is neglected in the calculation of the mode functions. This approximation breaks down in rod-type fiber amplifiers with very large cores, in which the mode-field diameter decreases with increasing average power [11]. By inserting the mode expansion in Eq. (14) into Eq. (6) and Eq. (13) and invoking the paraxial approximation, we can obtain a set of coupled-mode equations for the mode amplitudes

*β*=

_{nm}*β*

_{0,n}−

*β*

_{0,m}and the inverse group velocity

*β*

_{1,n}= 1/

*v*. Terms involving the group velocity dispersion and higher-order dispersion are neglected, since we consider signals of narrow linewidth in this paper. Furthermore, we have introduced

_{g,n}Although our model can include an arbitrary number of transverse modes, the detailed experimental analysis of TMI has shown that only the FM and the first higher-order mode of LMA fibers are involved, except for fibers with very large mode areas [4]. In this paper we therefore include only the fundamental *LP*_{01} mode and one of the two degenerate *LP*_{11} modes of a simple step-index fiber (SIF). We also assume that the fiber is water cooled and the appropriate boundary condition for the heat equation at the fiber surface is therefore

*h*is the convection coefficient for the cooling fluid. The Green’s function

_{q}*G*is in this case given by the expansion

*r*,

*ϕ*are the usual cylinder coordinates and the radial Green’s functions

*g*are given by

_{m}*I*and

_{n}*K*are the modified Bessel functions of the first and second kind, respectively,

_{n}*q*=

*iρC*Ω/

*κ*and the coefficients

*C*are given by

_{n}*R*being the outer radius of the fiber and $a=2{h}_{q}/\sqrt{q}\kappa $.

Introducing scaled mode amplitudes ${p}_{i}=\sqrt{{n}_{c}{\epsilon}_{0}c/2}{A}_{i}$ and keeping only phase-matched terms, we obtain the coupled-mode equations

*n*=

_{eff,n}*β*

_{0,n}/

*k*

_{0}and the overlap integrals

*K*are given by

_{n}*C*are given by the correlations

_{ij}*G*. The imaginary part of

_{nnnn}*G*is responsible for a nonlinear gain on the Stokes side of the spectrum. Due to the long response time of the thermal nonlinearity, caused by the slow heat diffusion in the fiber, the spectrum of

_{nnnn}*G*is extremely narrow, typically on the order of 100 Hz or less, depending on fiber design.

_{nnnn}The second nonlinear term gives rise to a cross-phase modulation (XPM) effect between the light in the two modes, and the last term is responsible, through the imaginary part of *G _{nmnm}*, for transfer of power between the modes, as we shall see later. Interestingly, it turns out that

*G*has a significantly wider spectrum compared to the other Green’s function overlap integrals, typically on the order of a kHz, again depending on fiber design.

_{nmnm}The physical mechanism responsible for the power transfer between the modes is the presence of a thermally induced LPG when light is propagating in both modes. The mode beating pattern creates a spatially and temporally varying temperature grating due to quantum defect heating, which results in the aforementioned LPG through the thermo-optic effect. Because of the long thermal diffusion time, the thermally induced LPG will be out of phase with the mode beating pattern, unless the latter is stationary. It has been argued in [7] that a phase lag between the mode beating pattern and the thermally induced LPG is required for power transfer between the modes to occur, and our coupled-mode equations should therefore have steady-state solutions. This is indeed the case, as we shall show next.

Finally, we note that by keeping only phase-matched terms in the coupled-mode equations, we have neglected the effect of a FWM interaction between the modes, which could also potentially transfer power between them. However, this effect is highly suppressed since the group-velocity dispersion is far too low to provide phase-matching for the narrow-band signals we are considering.

#### 2.2. Steady-state solution

It is easy to see that the coupled-mode equations admit a steady-state solution given by

*P*

_{0,}

*is the initial power in mode*

_{n}*n*. Φ

*is the nonlinear phase*

_{n}*P*being the power in mode

_{n}*n*

*γ*are given by

_{nm}*G*(

**r**

_{⊥},

**r**′

_{⊥}, −

*ω*) =

*G*(

**r**

_{⊥},

**r**′

_{⊥},

*ω*)

^{*}, from which it immediately follows that the nonlinear parameters

*γ*are all real. It is therefore clear that there is no transfer of power between the modes in our steady-state solution, and the only effect of the thermal nonlinearity is to cause SPM and XPM. One might therefore naively expect that our simple model is unable to explain TMI, and that more elaborate models with fewer approximations or additional physical mechanisms are required. However, as we shall show in the following, the steady-state solution is not stable, and the presence of amplitude noise will lead to a nonlinear transfer of power between the modes.

_{nm}The steady-state solution given in Eq. (29) is of course highly idealized in the sense that the bandwidth of the signal is infinitesimally small. Actual CW laser sources have a finite bandwidth, and it is therefore of interest to examine whether a steady-state solution with a finite bandwidth exists. We consider a solution which in the time domain has the form

*θ*(

*t*) is a stochastic phase which gives rise to a finite bandwidth of the CW field. It is easy to show that Eq. (35) is indeed a solution to the coupled-mode equations, where the deterministic phase Φ

*is given by Eqs. (30) and (31). Phase noise alone is thus not sufficient to rule out a steady-state solution, but when amplitude noise is included, it is no longer possible to find a stable steady-state solution, and we therefore expect that amplitude noise, either due to quantum fluctuations or due to intensity noise of the seed laser, is responsible for TMI.*

_{n}## 3. Operation at threshold

In this section, we derive approximate analytical solutions of the coupled-mode equations, which are valid in cases where the average output power does not exceed the threshold for TMI. These solutions show that both quantum noise and intensity noise of the input signal can act as a seed for TMI, and thus lead to transfer of power between the FM and a HOM of the fiber. By solving the coupled-mode equations numerically, we verify the validity of the approximate analytical solutions and also study the temporal dynamics of TMI.

#### 3.1. Quantum noise seeding

We first consider the case where a perfectly monochromatic signal is launched in the FM of the fiber amplifier, with no signal launched in the HOM. We can then show by solving Eqs. (24) and (25) to first order in *p*_{2} that the presence of quantum noise in the HOM leads to a nonlinear transfer of power from the FM to the HOM, and that this transfer exhibits a threshold-like dependence on output power. In the following we have assumed that *n*_{eff,1} ≈ *n*_{eff,2} ≈ *n _{c}* and

*v*

_{g,1}≈

*v*

_{g,2}, in which case the coupled-mode equations to first order in

*p*

_{2}become

*K*=

*K*

_{1}≈

*K*

_{2}and the group velocity term has been transformed away by shifting to a retarded time frame. We take the solution of Eq. (36) to be the CW solution given by Eq. (29). Inserting this solution into Eq. (37) we obtain

*χ*(Ω) = 4

*π*

^{2}

*K*Im[

*G*

_{2121}(Ω)] and

*P*

_{1}(

*z*) is given by Eq. (32). We can solve this differential equation to obtain the energy spectral density in the HOM at the output

*L*is the length of the fiber and the average gain coefficient is given by It is clear from the solution given by Eq. (39) that any frequency components present in the HOM for which

*χ*> 0 will experience a nonlinear gain in addition to the gain provided by the rare-earth doping. While we have assumed that no signal is launched into the HOM, quantum fluctuations of the field will always be present, and certain frequency components of this quantum noise can thus experience nonlinear gain. Writing the solution in terms of the power spectral density (PSD)

*S*of mode

_{n}*n*, we have

*S*

_{2}(0, Ω) =

*h*̄(

*ω*

_{0}+ Ω). The total output power in the HOM is thus given by

*R*= 20

_{c}*μ*m, a core refractive index

*n*= 1.45 and with a V parameter of 3. This fiber thus supports the fundamental

_{c}*LP*

_{01}mode as well as the degenerate

*LP*

_{11}modes. Although actual double-clad fiber amplifiers have both an inner and outer cladding, we assume for simplicity that the radius of the inner cladding is sufficiently large that the index step associated with the inner/outer cladding boundary has a negligible impact on the modes guided in the core. The outer radius of the fiber

*R*= 500

*μ*m and the convection coefficient of the cooling fluid

*h*= 1000 W/(m

_{q}^{2}K). Since we are considering an Yb-doped fiber amplifier, we take the pump wavelength to be 975 nm and the launched signal wavelength to be 1032 nm. We shall refer to this fiber amplifier as Fiber A, and the parameters are summarized in Table 1.

To calculate the nonlinear coupling coefficient *χ* we insert *ψ*_{1}(*r*, *ϕ*) = *R*_{1}(*r*) and *ψ*_{2}(*r*, *ϕ*) = *R*_{2}(*r*) cos*ϕ* into Eq. (19), where *R*_{1} and *R*_{2} are the radial mode functions for the *LP*_{01} and *LP*_{11} modes, respectively. Using the expansion for the Green’s function given in Eq. (21) we obtain an expression for *G*_{2121}

*χ*is calculated using

*η*= 3.5 × 10

^{−5}K

^{−1},

*κ*= 1.4 W/(Km) and

*ρC*= 1.67 × 10

^{6}J/(Km

^{3}). The result, shown in Fig. 1, shows that

*χ*is positive for negative values of Ω and has a sharp peak at Ω/2

*π*≈ 1 kHz. The overlap integrals given by Eq. (26) are Γ

_{1}≈ 0.90 for

*LP*

_{01}and Γ

_{2}≈ 0.66 for

*LP*

_{11}.

Defining the HOM content as *ξ* = *P*_{2}/(*P*_{1} + *P*_{2}) we find from Eq. (42)

_{1}− Γ

_{2}and we have used the fact that the dominant contribution to the integral comes from the narrow region around the peak in

*χ*to make the approximation

*ω*

_{0}+ Ω ≈

*ω*

_{0}. This fact also allows us to use Laplace’s method to evaluate the integral, which yields

*χ*″ denotes the second derivative of

*χ*with respect to Ω, and Ω

*denotes the frequency of the maximum of*

_{p}*χ*. Assuming an input power of 1 W in the FM, the HOM content is plotted as a function of output power in the FM in Fig. 2a and clearly shows a threshold-like behavior near a FM output power of approximately 450 W. Defining a threshold output power

*P*as the output power for which

_{th}*ξ*= 0.05, we find

*P*= 448 W by a numerical solution of Eq. (45). To investigate the dependence of the TMI threshold on the input signal power, we have also plotted the HOM content as a function of FM output power for an input signal power of 10 W and 50 W in Fig. 2a. The corresponding threshold powers are 480 W and 537 W, respectively.

_{th}The PSD of the output signal in the HOM is given by Eq. (41) and is shown in Fig. 2b for a FM output power *P*_{1}(*L*) = *P _{th}*. It is evident that the light is redshifted by approximately 1.5 kHz relative to the light in the FM, which corresponds to the peak in the nonlinear coupling coefficient

*χ*.

Since the light generated in the HOM is redshifted and has a finite spectral width, both of which are determined by the shape of the spectrum of *χ*, the interference between the remaining light in the FM and the light in the HOM is expected to display a complicated temporal behavior. It is evident from Eq. (43) that the nonlinear coupling coefficient *χ* depends on the shape of the mode functions, and thus on the fiber design, and hence the temporal dynamics of TMI is different for different fiber amplifiers. The dependence of the spectral properties of *χ* on the core diameter and V parameter of a SIF was investigated in [9] where it was found that the frequency and width of the peak of *χ* decrease with increasing core diameter, and that the maximum value of *χ* decreases with decreasing *V* parameter. The temporal dynamics of TMI is therefore expected to be slower for fibers with larger core diameter, and the threshold is expected to be higher for fibers with lower *V* parameter.

To investigate the impact of cooling efficiency on TMI, we have calculated *χ* for *h _{q}* = 10 W/(m

^{2}K), corresponding to passive air cooling, and

*h*= 10

_{q}^{4}W/(m

^{2}K), corresponding to very efficient water cooling. The results are indistinguishable from Fig. 1, which means that the TMI threshold and spectral properties are predicted to be exactly the same. This is a surprising result in light of experimental results presented in [8], which shows a higher threshold for an efficiently cooled fiber compared to an air-cooled fiber. We note that our calculations assume symmetric cooling of the fiber, which is not always the case experimentally. An asymmetric cooling of the fiber, such as e.g. when the fiber is attached to a heat sink, could lead to an overall temperature gradient, which would distort the mode profiles and thus alter the nonlinear coupling coefficient between the modes. Our calculations also assume that fiber parameters such as density, heat capacity, thermal conductivity and thermo-optic coefficient are temperature independent, and since the overall temperature of the fiber depends greatly on the cooling efficiency, this assumption may not be valid. This issue warrants further investigation, but is outside the scope of this paper.

#### 3.2. Intensity noise seeding

While we have shown that pure phase noise in the input signal does not induce TMI, we will now show that the presence of intensity noise in the input signal can act as a seed for TMI, provided that a small amount of the signal is launched in the HOM. To do this, we consider the coupled-mode equations to first order in *p*_{2} given by Eq. (36) and Eq. (37). We assume that the signal launched into each mode is given in the time domain by

*ε*is a zero-mean random variable representing the intensity noise of the input signal, and we further assume that |

_{N}*ε*| ≪ 1.

_{N}We again take the zeroth-order solution to the FM signal to be the CW solution given in Eq. (29) and can then derive the PSD of the output signal in the HOM, given by Eq. (41). The input PSD in the HOM is given by

*p*

_{2}(0,

*t*)

*p*

_{2}(0,

*t*+

*t*′)

^{*}〉 is the autocorrelation function of the input signal in the HOM. Using Eq. (46) the PSD of the input signal in the HOM can be written as

*R*is the relative intensity noise (RIN) of the input signal, and is given by

_{N}*P*

_{1}=

*P*

_{1}(

*L*)−

*P*

_{0,1}. The total output power in the HOM

*P*

_{2}(

*L*) is then found by integrating

*S*

_{2}(

*L*, Ω) over all frequencies. In terms of the HOM content

*ξ*we find

*χ*, we can assume that the RIN is independent of frequency and use Laplace’s method to evaluate the integral. This yields

*P*

_{1}≈

*P*

_{1}(

*L*) in the denominator in the second term. We have plotted the HOM content as a function of output power in the FM

*P*

_{1}(

*L*) for Fiber A in Fig. 3 assuming an initial HOM content

*ξ*(0) = 0.01, an initial FM input power

*P*

_{0,1}= 1 W and three different values of the relative intensity noise

*R*: 10

_{N}^{−13}Hz

^{−1}, 10

^{−12}Hz

^{−1}and 10

^{−11}Hz

^{−1}. In the same figure, we have plotted the HOM content for quantum noise seeding for comparison. The threshold powers for TMI in each case of intensity noise seeding are found to be 351 W, 320 W and 288 W, respectively, which is significantly lower than the 448 W found for the quantum noise seeded case, but still on the same order of magnitude.

From Eq. (52) we see that the threshold power has an approximately logarithmic dependence on the RIN, and measures taken to reduce the intensity noise of the input signal are therefore expected to result in only modest improvements in the TMI threshold. The same is true for the dependence of the TMI threshold on the initial HOM content *ξ*(0), which explains why efforts to optimize the in-coupling of the signal are found to have little impact on the TMI threshold in experiments.

#### 3.3. LP_{01} - LP_{02} coupling

So far we have only considered coupling between *LP*_{01} and *LP*_{11}. However, some large-core fibers may support additional guided modes, such as *LP*_{02}-like modes, in particular for operating powers for which the thermo-optic effect is strong enough to significantly alter the guiding properties of the fiber [6]. While the presence of an additional *LP*_{31} mode has been reported in TMI of a large-pitch fiber with a mode field diameter of 75 *μ*m [4], *LP*_{02}-like modes have as yet not been observed to take part. To investigate this, we have calculated the nonlinear coupling coefficients for both *LP*_{01} − *LP*_{11} coupling and *LP*_{01} − *LP*_{02} coupling for a SIF with a V parameter of 5 and all other parameters the same as Fiber A. The result is presented in Fig. 4 and clearly shows that the nonlinear gain of the *LP*_{01} − *LP*_{02} coupling is much less than for the *LP*_{01} − *LP*_{11} coupling in this case. Calculating the quantum noise seeded threshold power for a threshold HOM content *ξ _{th}* = 0.05, we find

*P*= 350 W and

_{th}*P*= 608 W for the two cases. It is thus not surprising that an

_{th}*LP*

_{02}-like mode has not been observed to take part in TMI, as the coupling to the

*LP*

_{11}-like modes must be expected to be much stronger.

#### 3.4. Numerical results

The results presented so far are based on approximate solutions of the coupled-mode equations, and it is therefore of interest to compare these solutions to a full numerical solution of the coupled-mode equations. In particular, our semi-analytical results are based on the assumption of a perfectly monochromatic signal. As can be seen from Fig. 1 the nonlinear coupling coefficient *χ* has a very narrow spectrum on the order of a few kHz, which is comparable to the linewidth of typical single-frequency fiber laser sources. We shall therefore use a numerical solution to study the effect of a finite signal linewidth on the TMI threshold.

Our implementation of the numerical solution of the coupled-mode equations, Eq. (24) and Eq. (25), is straightforward. We use a standard ODE integrator [13] to step the solution forward in *z* and the correlations are computed using fast Fourier transforms. While it would be possible in principle to include rate equations to determine the *z*-dependence of the gain coefficient *g*(*z*), we have chosen to consider a simplified case in which *g*(*z*) is constant. The analytical results derived in the previous sections showed that the power threshold for TMI was only dependent on the total gain. While this is only strictly true when rapid spatio-temporal oscillations of *g* can be neglected, we nevertheless believe that the numerical results derived with a constant gain are sufficiently accurate to provide valuable insight into the dynamics of the TMI phenomenon.

We consider a SIF with the same parameters as Fiber A for all the results in this section, and consider coupling between *LP*_{01} and one of the two degenerate *LP*_{11} modes. For the input signal, we assume a CW signal with only phase noise to which we add random quantum noise by injecting one photon per mode [12]. The phase noise of the input signal is modeled in the time domain as

*θ*(

*t*) is the result of a Gaussian random walk. This phase noise model provides a signal with a constant amplitude but with a Lorentzian lineshape.

To investigate the influence of the signal bandwidth on the TMI threshold, we have run simulations for input signal bandwidths of 1 Hz, 1 kHz and 10 kHz (FWHM). In all cases, the input signal power *P*_{0,1} = 1 W for the FM and *P*_{0,2} = 0 W for the HOM. The fiber length *L* = 1 m and the gain coefficient *g* = ln(500)/(Γ_{1}*L*). In the absence of any nonlinear mode coupling, the fiber amplifier should thus provide 27 dB gain. Fig. 5 shows the output PSD for the FM and HOM for all three simulations. The HOM spectrum for the 1 Hz case, plotted in Fig. 5b shows the presence of light redshifted relative to the FM by approximately 1.5 kHz, corresponding to the peak of the nonlinear coupling coefficient shown in Fig. 1. The shape of the spectrum also agrees well with what is seen in Fig. 2b.

For the 1 kHz case, the HOM light is seen to experience a redshift of approximately 1.5 kHz as in the 1 Hz case, and is also spectrally broadened, although the latter effect is less visible due to the width of the input signal. The redshift is also present in the 10 kHz case, but is hardly noticeable on the plot due to the larger spectral width of the input signals.

The HOM content *ξ* and the average power in each mode as a function of *z* for the 10 kHz input signal linewidth case are plotted in Fig. 6 and shows that the TMI threshold is reached when the power in the FM reaches approximately 450 W, in good agreement with our semi-analytical result in the previous section. The results for the 1 Hz and 1 kHz cases are indistinguishable from this case, and are thus not shown.

In order to investigate the temporal dynamics of TMI, we plot the squared norm of the time-domain mode amplitudes at the output |*p _{n}*(

*L,t*)|

^{2}, which gives the instantaneous power in mode

*n*. These are plotted in Fig. 7 for the 1 Hz input signal linewidth case and shows a rather chaotic fluctuation of power between the FM and the HOM on a timescale of a few ms, which is what would be expected from the width and redshift of the spectrum in Fig. 5b. It is also consistent with recent experimental findings [14], which showed that TMI in a Yb-doped PCF amplifier with a core diameter of 38

*μ*m manifested itself as a chaotic oscillation of the beam intensity on a ms timescale. The results for the 1 kHz and 10 kHz cases display a similar behavior and are therefore not reproduced here.

Considering the results of our simulations, it appears that the bandwidth of the signal does not influence the TMI threshold or the temporal dynamics of the mode fluctuations to any great extent, which is also consistent with experiments [8].

## 4. Operation beyond threshold

We now investigate the behavior of the output signal when a fiber amplifier is operated above the TMI threshold. Since the perturbative approach presented in section 3 is invalid in cases where the power in the HOM becomes comparable to the power in the FM, we investigate this regime by numerically solving the coupled-mode equations for 1 m of Fiber A, but with a higher gain coefficient *g* = ln(1000)/(Γ_{1}*L*). We consider a case in which the input signal power is 1 W and the signal linewidth is 1 kHz. We further assume that the signal is launched into the FM, with no signal launched in the HOM, and add quantum noise to this signal.

Considering the average mode power and HOM content as a function of *z*, shown in Fig. 8, we see that the HOM content increases dramatically as the TMI threshold is reached. At this power level, power is thus transferred from the FM to the HOM. This transfer proceeds until the HOM content reaches approximately 0.8, at which point the process is reversed and power is transferred back to the FM from the HOM. The power flow between the modes quickly reverses again, however, and the HOM content appears to converge to 0.5 as the total signal power increases.

The output PSD of the light in the FM and HOM is shown in Fig. 9. It is seen that the light in both modes is now significantly redshifted relative to the input signal, and that the FM spectrum has undergone significant broadening. This broadening, which was absent in the simulations for the amplifier operating at the threshold power, can be explained by considering the power flow reversals between the modes described above. The light in the HOM is initially seeded by quantum noise and thus has a spectral width which is determined by the bandwidth of the nonlinear gain, since the bandwidth of the FM light is comparable to the nonlinear gain bandwidth. As the output power increases beyond the TMI threshold and the power flow between the modes reverses, the relatively broadband light in the HOM is coupled back into the nearly depleted FM. With each reversal of the power flow, the light is redshifted and also additionally broadened due to the nonlinear gain, and the end result is that the light in both modes is redshifted and spectrally broadened relative to the input. We have tested that the redshift is indeed due to the coupling between the modes by removing the quantum noise from the simulation, in which case no power transfer between the modes occurred and no spectral broadening was observed.

The temporal dynamics of the TMI is again studied by plotting the instantaneous mode power of each mode as a function of time, which is shown if Fig. 10. The mode fluctuations are chaotic and the characteristic timescale is somewhat shorter compared to the result for operation at the TMI threshold shown in Fig. 7. This shorter timescale of the mode fluctuations is most likely a result of the additional spectral broadening of the light, which we discussed above. We also note that a complete transfer between the modes can occur on a sub-ms timescale. Such a complete transfer of power between the modes has been experimentally observed in [3].

## 5. Amplitude modulated input signal

We shall now consider what happens if the input signal contains a small amplitude modulation with a modulation frequency close to the frequency of the peak of the nonlinear coupling coefficient. The modulation frequencies we consider would thus be in the range of a few 100 Hz to a few kHz, depending on the fiber core diameter [9]. Amplitude modulations in this frequency range can be caused by various external electrical or mechanical disturbances [15], and may also act as a seed for TMI.

#### 5.1. Perturbative calculation

In terms of our coupled-mode model, we consider input mode amplitudes in the time domain on the form

*a*is the modulation depth relative to the average amplitude of the mode, Ω

*is the angular modulation frequency and we assume*

_{m}*a*≪ 1. We again employ the perturbative solution to the coupled-mode equations used in section 3, with the output PSD in the HOM

*S*

_{2}(

*L*, Ω) given by Eq. (41). We find the input PSD in the HOM

*S*

_{2}(0, Ω) by inserting Eq. (54) into Eq. (47) which gives

*.*

_{m}Inserting Eq. (55) into Eq. (41) and integrating over Ω yields the average output power in the HOM

*χ*Ω

*= −*

_{m}*χ*− Ω

*arising from the first delta-function in Eq. (55), since this term is very small. Dividing by the total output power, we find the output HOM content*

_{m}*a*, we have calculated the threshold power for Fiber A for three different values of

*a*: 10

^{−4}, 10

^{−5}and 10

^{−6}. The corresponding TMI thresholds are: 309 W, 372 W, and 434 W. In all three cases, the modulation frequency Ω

*/2*

_{m}*π*= 1 kHz,

*ξ*(0) = 0.01 and

*P*

_{0,1}= 1 W. The threshold criterium was

*ξ*= 0.05, the same as was used for quantum noise and RIN seeding. The output HOM content as a function of FM output power for the three values of modulation depth is plotted in Fig. 11. From Eq. (57) it is clear that the TMI threshold has an approximately logarithmic dependence on the modulation depth, which is also seen in Fig. 11. As was the case for RIN seeding, the same is true for the dependence on the input HOM content

*ξ*(0).

#### 5.2. Numerical results

To check the validity of the approximate solution given in Eq. (57) and to investigate the TMI behavior beyond threshold when the input signal contains a small harmonic amplitude modulation, we have solved the coupled-mode equations numerically using the same method as in section 3.4. The input mode amplitudes in the time domain are given by

*a*= 10

^{−4}and the modulation frequency Ω

*/(2*

_{m}*π*) = 1 kHz. The input signal powers were

*P*

_{0,1}= 0.99 W and

*P*

_{0,2}= 0.01 W.

We first performed a simulation in which the fiber amplifier is operating at the TMI threshold. The gain coefficient is therefore set to *g* = ln(350)/(Γ_{1}*L*), which provides a gain sufficient to reach the threshold power. The mode power and HOM content as a function of *z* are plotted in Fig. 12 and shows that the TMI threshold is reached when the output FM power is approximately 325 W. The approximate semi-analytical calculation of 309 W thus slightly underestimates the threshold power in this case.

The output PSD of the FM and HOM are shown in Fig. 13. It is clear that the HOM Stokes sideband due to the amplitude modulation has acted as a seed for the TMI and experienced a significant nonlinear gain, since the strong FM carrier component is able to couple efficiently to this sideband and power is transferred from the FM carrier to the HOM Stokes sideband. On the other hand, the HOM anti-Stokes sideband is depleted, since it couples to the strong FM carrier, but the direction of the transfer of power is from the HOM to the FM. This additional transfer of power from the HOM to the FM is not taken into account in the approximate semi-analytical calculation, and may partly explain why this calculation underestimates the FM power at the TMI threshold.

We also see the appearance of an additional Stokes sideband at Ω = −2Ω* _{m}* in the HOM. The seed for this additional sideband is most likely due to intra-modal four-wave mixing (FWM) between the sidebands and carrier of the HOM. The FWM process can generate additional side-bands at Ω = ±2Ω

*, and the second Stokes sideband at Ω = −2Ω*

_{m}*can then be amplified by the nonlinear gain provided by the presence of the strong FM carrier. We have tested this hypothesis by simulating the amplitude modulated signal in the HOM only, without any quantum noise and no signal in the FM. The output signal in this case shows the presence of additional symmetrically distributed sidebands at the second harmonic of the modulation frequency.*

_{m}Recent experiments have utilized a beam aperture measurement to investigate the temporal dynamics of TMI, in which the intensity in a small part of the output beam near field image is measured by a fast photo-diode [4, 8, 14]. The idea is that the temporal mode fluctuations result in an intensity variation, which is then recorded and analyzed. In the experiments by Otto *et al.*[4] and Ward *et al.*[8], a transition region was found in which the aperture intensity measurements showed a harmonic behavior when the fiber amplifier was operated close to the TMI threshold. On the other hand, no such behavior was found by Karow *et al.*[14].

To compare our simulations with these experiments, we calculate the intensity of the output beam at a fixed point in the beam profile and plot the corresponding normalized frequency spectrum |*I*(Ω)|/|*I*(0)|. The intensity is calculated at a point located at *r* = 20 *μ*m, *ϕ* = 0, so that the interference between *LP*_{01} and *LP*_{11} is strong. The result for the simulation discussed above is plotted in Fig. 14 and shows a single strong peak at Ω/(2*π*) = 1 kHz and a much weaker peak at the second harmonic Ω/(2*π*) = 2 kHz. This result is easily understood from the spectra shown in Fig. 13 as an interference between the FM carrier and the amplified Stokes sideband of the HOM.

We now consider the behavior of the fiber amplifier when operating above the TMI threshold. We therefore set the gain coefficient *g* = ln(400)/(Γ_{1}*L*), with all other parameters being the same as above. From the mode power and HOM content plotted in Fig. 15 we see that a significant fraction of the power has been transferred from the FM to the HOM at this gain, with the output HOM content *ξ*(*L*) ≈ 0.69.

From the output PSD, plotted in Fig. 16, we see that the Stokes sideband at Ω = −1 kHz of the HOM has been amplified as expected. However, we also clearly see the presence of additional Stokes and anti-Stokes sidebands in the HOM spectrum, which are generated by the aforementioned FWM process. In the FM, the original carrier at Ω = 0 has been depleted and a new Stokes sideband at Ω = −2Ω* _{m}* has appeared, and the original anti-Stokes sideband, which was depleted at the lower gain, has been restored by the FWM process. The additional Stokes sidebands act as seeds for the nonlinear power transfer between the modes. This cascading process continues as power is increased, generating and amplifying additional Stokes sidebands, until the power reaches the threshold for RIN or quantum noise induced TMI at which point the noise seeding will produce a broad spectrum which masks out the discrete sidebands.

We note that the cascade process described above requires that the modulation frequency is close to the frequency of the peak of the nonlinear gain. A modulation frequency much higher than the nonlinear gain peak frequency will lead to sidebands that do not experience significant nonlinear gain. Intensity modulations of lower modulation frequencies may lead to a larger gain of higher-order sidebands, generated by FWM, compared to the lowest-order sideband if the frequency offset of the higher-order sidebands match the nonlinear gain peak frequency, depending on the relative strength of the sidebands.

We again plot the intensity spectrum of the output beam, shown in Fig. 17, from which we can identify the presence of the second, third and fourth harmonic of the modulation frequency in the output intensity. These harmonics are caused by interference between the different Stokes sidebands of the FM and HOM output.

Finally, we consider operation well above the TMI threshold, setting the gain coefficient *g* = ln(700)/(Γ_{1}*L*). The average mode power and HOM content as a function of *z* are plotted in Fig. 18 and shows the average power being transferred between the modes after the TMI threshold is reached. The amplifier is operating well above the threshold for quantum noise induced TMI, which results in the broad output spectra seen in Fig. 19. The discrete sidebands seen at lower power levels are no longer present in the spectra.

The broad spectra of the output mode amplitudes result in a chaotic fluctuation in the output intensity, as is evident from the spectrum shown in Fig. 20. The intensity spectrum has a broad shape with a maximum around 1 kHz and falls off with increasing frequency. The shape of the curve will depend on fiber parameters, in particular the core diameter [9], but the overall shape is quite similar to what was found in [4, 8].

The behavior of the TMI dynamics with increasing power matches quite well what was observed experimentally by Otto *et al.*[4] and Ward *et al.*[8], and it is therefore possible that the transition behavior observed in these experiments are caused by the mechanism proposed here. The experiment by Karow *et al.*, in which only chaotic intensity fluctuations were observed, is also naturally explained by our model, in the sense that a sufficiently strong harmonic modulation of the input signal is required to produce the harmonic behavior seen near threshold.

## 6. Conclusion

In this work we have presented a coupled-mode approach to the modeling of TMI in high average power fiber amplifiers. Our model has the advantage of numerical simplicity and efficiency, and also lends itself well to approximate semi-analytical solutions.

In the formulation of the model, a number of simplifying assumptions were made, the most limiting being the neglect of the effect of gain saturation and thermal lensing. As discussed in section 2, the first of these two assumptions leads to an overestimate of the strength of the thermally induced grating and thereby an overestimate of the coupling between the modes. In the second approximation, the effect of the thermally induced refractive index perturbation on the mode functions is ignored. This effect can be significant in rod-type fibers with very large cores, and will lead to a stronger confinement and hence overlap of the modes [6, 11]. As shown in [9], a larger overlap between the modes and the doped region of the core will in general lead to a lower threshold power. By neglecting this effect, our model is therefore expected to significantly overestimate the TMI threshold power for very-large-core fiber amplifiers. It may be possible to at least partly relax these key assumptions, but this is beyond the scope of the present paper. In addition, a number of other approximation were made, that we believe are less critical. Nevertheless, we summarize them briefly here. We ignored longitudinal heat flow in the heat equation on the grounds that the radial temperature gradient is much larger than the longitudinal temperature gradient. It was hypothesized in [8] that longitudinal heat flow could cause the onset of TMI, but additional simulations by the authors did not support this hypothesis. Indeed, the work presented in this paper shows that longitudinal heat flow is by no means required to explain the presence of TMI. Another simplifying assumption is our adoption of homogeneous cooling boundary conditions. Although our results were quite insensitive to the convection coefficient of the coolant, it is possible that an asymmetric cooling boundary condition, corresponding to e.g. a fiber placed on a heat sink, would produce different results. This could be implemented in our model by choosing a Green’s function obeying the appropriate boundary condition. In the numerical solutions of our coupled-mode model, we assumed a constant gain coefficient, which is a rather crude approximation of the actual gain coefficient in a backward pumped fiber amplifier. This assumption could be lifted by introducing rate equations for ytterbium to calculate the longitudinal variation of the gain coefficient. We have also included only the FM and one HOM in our calculations, but the model can be easily extended to include more modes. Finally, for numerical convenience we have chosen to focus on a SIF amplifier, but the model can be used for any type of fiber for which the modes can be calculated.

In spite of these limitations, the threshold powers obtained from the model seem to match reasonably well with experiments, and the qualitative features of TMI observed in recent experiments seem to agree well with our results. Here we summarize main results and conclusions obtained. We have shown how both quantum noise, RIN and amplitude modulation of the signal can act as a seed for TMI, and have shown by an approximate semi-analytical solution of our model that the upper limit on the TMI threshold is due to quantum noise. We also found that the threshold is quite insensitive to perturbations of the amount of signal accidentally launched into a HOM, which is consistent with the experimental results reported in [8, 14]. Considering coupling between the *LP*_{01} and *LP*_{02} modes, we found that the coupling between these two modes was significantly weaker compared to the coupling between *LP*_{01} and *LP*_{11} of the same fiber. This explains why *LP*_{02}-like modes have, to the best of our knowledge, so far not been observed in experiments on TMI.

Experimentally it has been found that various types of cooling of the fiber amplifier can lead to variations in the TMI threshold power [8]. While our model allows for a varying cooling efficiency through the thermal boundary condition, we found no effect of varying the efficiency of the cooling. As discussed above, however, our model assumes symmetric cooling of the fiber, which may not be appropriate for experiments in which the fiber is mounted on a heat sink.

By numerically solving the coupled-mode equations of our model, we investigated the behavior of the fiber amplifier above the TMI threshold and found that in the case of quantum noise seeding, the average power in each of the two modes seems to converge to half the total power as the output power is increased well beyond the threshold. The instantaneous power in each mode was found to fluctuate chaotically on a ms timescale. We also found that the TMI threshold was insensitive to the bandwidth of the phase-noise broadened signal, which is consistent with experiments in which the bandwidth of the signal was broadened by phase-modulation [8].

Finally, we have investigated the effect of a small harmonic amplitude modulation of the input signal on TMI, and shown how it may lead to the generation of discrete frequency components in the intensity of the output beam through a combination of thermally induced intra-modal FWM and inter-modal coupling. This effect may explain the behavior in the transition region observed in some recent experiments [4, 8].

In terms of mitigation strategies, the ever-present quantum noise means that in order to push the output power beyond the quantum noise TMI threshold, which can be as low as a few hundred W, fiber designs which reduce the overlap between the FM and any HOMs, as well as with the doped region, should be considered [9]. PCF fiber designs combining index guiding with photonic bandgaps to provide nearly single-mode operation in a limited bandwidth can also be effective in increasing the TMI threshold, but due to the sensitivity of the bandgap location to the heating of the core [16], such fibers must be designed for use specifically at high operating power.

Intensity noise reduction can also increase the TMI threshold, but as we have shown, the threshold power is rather insensitive to the RIN level, and so the effect of noise reduction is expected to be modest. On the other hand, the presence of discrete spectral components of the RIN at a frequency close to the peak nonlinear gain of the fiber may lead to a substantial reduction of the TMI threshold, and active stabilization measures to reduce such spectral components may be worthwhile.

## Acknowledgments

This work was funded by the Danish Council for Independent Research - Technology and Production (FTP).

## References and links

**1. **F. Stutzki, F. Jansen, T. Eidam, A. Steinmetz, C. Jauregui, J. Limpert, and A. Tünnermann, “High average power large-pitch fiber amplifier with robust single-mode operation,” Opt. Lett. **36**, 689–691 (2011). [CrossRef] [PubMed]

**2. **T. Eidam, C. Wirth, C. Jauregui, F. Stutzki, F. Jansen, H.-J. Otto, O. Schmidt, T. Schreiber, J. Limpert, and A. Tünnermann, “Experimental observations of the threshold-like onset of mode instabilities in high power fiber amplifiers,” Opt. Express **19**, 13218–13224 (2011). [CrossRef] [PubMed]

**3. **F. Stutzki, H.-J. Otto, F. Jansen, C. Gaida, C. Jauregui, J. Limpert, and A. Tünnermann, “High-speed modal decomposition of mode instabilities in high-power fiber lasers,” Opt. Lett. **36**, 4572–4574 (2011). [CrossRef] [PubMed]

**4. **H.-J. Otto, F. Stutzki, F. Jansen, T. Eidam, C. Jauregui, J. Limpert, and A. Tünnermann, “Temporal dynamics of mode instabilities in high-power fiber lasers and amplifiers,” Opt. Express **20**, 15710–15722 (2012). [CrossRef] [PubMed]

**5. **C. Jauregui, T. Eidam, J. Limpert, and A. Tünnermann, “The impact of modal interference on the beam quality of high-power fiber amplifiers,” Opt. Express **19**, 3258–3271 (2011). [CrossRef] [PubMed]

**6. **K. R. Hansen, T. T. Alkeskjold, J. Broeng, and J. Lægsgaard, “Thermo-optical effects in high-power Ytterbium-doped fiber amplifiers,” Opt. Express **19**, 23965–23980 (2011). [CrossRef] [PubMed]

**7. **A. V. Smith and J. J. Smith, “Mode instability in high power fiber amplifiers,” Opt. Express **19**, 10180–10192 (2011). [CrossRef] [PubMed]

**8. **B. Ward, C. Robin, and I. Dajani, “Origin of thermal modal instabilities in large mode area fiber amplifiers,” Opt. Express **20**, 11407–11422 (2012). [CrossRef] [PubMed]

**9. **K. R. Hansen, T. T. Alkeskjold, J. Broeng, and J. Lægsgaard, “Thermally induced mode coupling in rare-earth doped fiber amplifiers,” Opt. Lett. **37**, 2382–2384 (2012). [CrossRef] [PubMed]

**10. **K. D. Cole and P. E. Crittenden, “Steady-Periodic Heating of a Cylinder,” ASME J. Heat Transfer **131**, 091301 (2009). [CrossRef]

**11. **F. Jansen, F. Stutzki, H.-J. Otto, T. Eidam, A. Liem, C. Jauregui, J. Limpert, and A. Tünnermann, “Thermally induced waveguide changes in active fibers,” Opt. Express **20**, 3997–4008 (2012). [CrossRef] [PubMed]

**12. **R. G. Smith, “Optical Power Handling Capacity of Low Loss Optical Fibers as Determined by Stimulated Raman and Brillouin Scattering,” Appl. Opt. **11**, 2489–2494 (1972). [CrossRef] [PubMed]

**13. **P. N. Brown, G. D. Byrne, and A. C. Hindmarsh, “VODE: A Variable Coefficient ODE Solver,” SIAM J. Sci. Stat. Comput. **10**, 1038–1051 (1989). [CrossRef]

**14. **M. Karow, H. Tünnermann, J. Neumann, D. Kracht, and P. Weßels, “Beam quality degradation of a single-frequency Yb-doped photonic crystal fiber amplifier with low mode instability threshold power,” Opt. Lett. **37**, 4242–4244 (2012). [CrossRef] [PubMed]

**15. **J. Chen, J. W. Sickler, E. P. Ippen, and F. X. Kärtner, “High repetition rate, low jitter, low intensity noise, fundamentally mode-locked 167 fs soliton Er-fiber laser,” Opt. Lett. **32**, 1566–1568 (2007). [CrossRef] [PubMed]

**16. **M. Laurila, M. M. Jørgensen, K. R. Hansen, T. T. Alkeskjold, J. Broeng, and J. Lægsgaard, “Distributed mode filtering rod fiber amplifier delivering 292W with improved mode stability,” Opt. Express **20**, 5742–5753 (2012). [CrossRef] [PubMed]