1. Introduction

Compared with organic light-emitting diode (OLED) displays, liquid crystal displays (LCDs) defect in thickness and contrast [1,2]. However, the recent rapid development of mini-LEDs enables a small-pitch mini-LED array as the backlight, achieving a very short optical distance and local dimming with dense partitions, thus empowering LCDs approaching OLEDs in thickness and contrast [2–4]. Moreover, inherent advantages of LCDs, such as high peak brightness, long service life, and mature fabrication technology, are maintained. Most LCDs use color filters to realize full-color images, and three subpixels form one pixel, as shown in Fig. 1(a). In this manner, the color filters absorb at least two-thirds of the light, severely reducing light efficiency. To eliminate color filters, field sequential color (FSC) LCDs sequentially display multiple fields (also known as subframes) to realize temporal color mixing [5], as Fig. 1(b) shows. Compared with traditional LCDs based on spatial color mixing, FSC-LCDs can theoretically achieve three times the light efficiency and spatial resolution without using color filters and subpixels [6]. Therefore, the FSC-LCD using a mini-LED backlight is an ideal image source for augmented reality (AR), virtual reality (VR), head-up displays (HUDs), and other applications needing high brightness and spatial resolution [7–11]

 

Fig. 1. (a) Traditional LCDs based on spatial color mixing. (b) FSC-LCDs based on temporal color mixing. (c) The color breakup phenomenon.

Download Full Size | PDF

However, the color breakup hinders the application of FSC-LCDs [12,13]. Because field images are presented sequentially, they cannot be perfectly combined on the retina when a saccadic eye movement or pursuit of a moving object induces a speed between the image and the viewer’s eyes, as Fig. 1(c) shows. The color breakup not only leads to visual fatigue but also seriously reduces image quality. Therefore, low-color-breakup driving is vital for FSC-LCDs.

Previous studies stated that a refresh rate up to 540 Hz or higher could effectively suppress color breakup [14,15], but a high refresh rate means complex driving, high cost, and challenges to the LC response time. To suppress color breakup at a reasonable refresh rate, researchers have intensively investigated changing how fields are presented. One strategy is to use an additional field, i.e., a 4-field driving scheme. The additional field concentrates image content likely to induce color breakup and thus reduces the luminance of the remaining three fields, such as 240Hz-Stencil and Edge-Stencil proposed by Lin et al. [16,17]. Recently, we improved the 4-field scheme by matching the image content of each partition with the most suitable driving method through deep learning [18]. That study addressed the problem that previous 4-field algorithms are sensitive to image content, realizing adaptative driving. The above 4-field methods can effectively suppress color breakup without causing distortion. However, four fields require extra time resolution. For example, a 120-Hz frame rate requires a 480-Hz native refresh rate under the 4-field scheme, which is still challenging for commercial LCDs.

In this manner, the 3-field scheme is preferred because of a lower refresh rate requirement. However, fewer fields mean less freedom in balancing color breakup and image fidelity. For example, Zhang et al. proposed the local-primary-desaturation (LPD) method [19,20], which desaturates the primary colors of each backlight partition. This method can reproduce images almost without distortion by accurately controlling the desaturated backlight beyond the color gamut of image content. However, the color gamut is hardly compressed for image content with rich colors, causing poor color breakup suppression at this point. Lin et al. proposed the Green-based 180Hz-Stencil method [21], which mainly shows an image’s green content and part of the red and blue content in a multi-color field by considering green contributes the most in perceptual color difference. The remaining red and blue are presented in the other two fields. A multi-color field concentrating green information with two dimmed mono-color fields suppresses color breakup effectively. However, the red and blue are easily redundant in the multi-color field since the green backlight of the first field is invariantly calculated through the root-mean-square of green content. Therefore, greenish pictures tend to encounter distortions in form of desaturated green.

The Stencil family conducts an instructive concept for color breakup reduction—a multi-color field can largely concentrate the content inducing color breakup. Nevertheless, previous studies adopted an invariant strategy to derive the multi-color field from input images. Doing so is practical for 4-field driving since an additional field can always guarantee little distortion. Whereas, when it comes to 3-field driving, there is little flexibility in controlling distortion, causing a long-standing dilemma between color breakup and image fidelity.

As discussed above, this study aims to develop a 3-field FSC algorithm that can balance color breakup and distortion for various image content. Inspired by the Stencil family, we first deeply exploit the backlight signal of the multi-color field through rigorous multi-objective optimization (MOO). The optimization can produce the Pareto optimality with a customizable balance between distortion and color breakup, enabling the method to be, to our knowledge, the first FSC algorithm fully adaptive to image content.

In addition, the new method should support video-rate driving, whereas iteration-based MOO must not satisfy this requirement. For the acceleration of a complicated display driving algorithm, the adoption of deep learning has been reported, e.g., neural network-based backlight generation and grayscale compensation for local dimming [22–25]. Therefore, this study next proposes a lightweight backlight generation network (LBGNN) using the MOO-generated backlight data as the training set. Compared with previous networks for display driving, our LBGNN has much fewer parameters, resulting in a runtime as fast as 2.3 ms per frame (based on a GeForce RTX 3060 graphic card). Meantime, the high performance brought about by MOO is maintained.

In Sec. 2, we will introduce the MOO-based generation method of the backlight dataset and discuss the network structure and training process of the LBGNN, as well as its implementation in FSC-LCDs. Sec. 3 will demonstrate effectively suppressed color breakup with imperceptible distortions for various images, significantly surpassing existing 3-field approaches. Sec. 4 will discuss the selection of some parameters in the algorithm and application scenarios, followed by conclusions in Sec. 5.

2. Method

2.1 MOO-based training set generation

We assume a 13.6-inch LCD using a mini-LED backlight with its configuration shown in Table 1. The LCD obeys typical specifications of currently mainstream mini-LED-based tablets. Such a mini-LED array is commercially available, as reported in our previous study [18], which used a real mini-LED array with the same parameters.

Table 1. Configuration of the mini-LED LCD for this study

View Table | View all tables in this article

For a 3-field algorithm, RGB backlight signals and LC transmittance of the three fields need to be determined from an input image, forming a highly ill-posed problem. Nevertheless, the Stencil family provides a very efficient practice that a multi-color field with two mono-color fields can effectively achieve design goals, e.g., color breakup and distortion. The performance of existing Stencil algorithms is severely limited by a consistent approach to calculating the multi-color field. To break through the limit, we adopt MOO to optimize the field’s backlight rigorously. The objectives here are low color breakup and low distortion. When multiple objectives are considered, pursuing a single optimal solution is usually impossible, but MOO is needed to obtain the Pareto optimality, which denotes a solution set in which none of the objectives can be improved without degrading other objectives. In our problem, no solution in the set can outperform any other one in both distortion and color breakup; in addition, no solution with better distortion and color breakup can be further found.

Specifically, we adopt the multi-objective genetic algorithm (MOGA), a representative MOO algorithm, which transforms a MOO problem into single-objective problems based on the linear weighting method [26]. Furthermore, the dimming partition of the FHD display is 18 × 32 (60-by-60 pixels in a dimming block), and a total of 18 × 32 × 3 parameters of the multi-color field’s backlight need to be optimized. To prevent a difficult convergence caused by too many parameters, we parallelly optimize every dimming block, so the number of parameters to optimize in an individual problem is reduced to three (i.e., RGB backlight values of each dimming block). Decomposing the ultra-high-dimensional problem into parallel low-dimensional problems can ensure satisfactory convergence and efficiency of the optimization.

Figure 2 shows the backlight optimization algorithm’s flow, where the fitness incorporating distortion and color breakup is crucial. For the calculation of the fitness, first, a “display image” and a “color breakup image” should be obtained. Figure 3 illustrates this process in detail. The input image is the content over a dimming block, BL₁ is a randomly initialized color backlight, and the first field fully displays the green content sensitive to human eyes. Therefore, the LC transmittance T₁ is obtained by dividing the luminance I_G of the input image’s green channel by the luminance BL_G of the backlight’s green channel. The image content (I_R1, I_G1, I_B1) displayed in the first field is obtained by multiplying the backlight luminance BL₁ and T₁. The second field is to fully show the remaining red luminance I_R2, so the backlight BL₂ is the maximum value of I_R2, and the LC transmittance T₂ is obtained by dividing I_R2 by BL₂. The third field displays the remaining blue luminance I_B2, calculated with the same method as the second field. The luminance of the displayed image ($I_R^{\prime}$, $I_G^{\prime}$, $I_B^{\prime}$) is the sum of the luminance of the three fields. Based on the display image, the color breakup image is acquired by horizontally shifting the second and third fields for a specific number of pixels relative to the first and second fields, respectively, and then synthesizing the shifted fields. Here, fifteen is adopted as the shift value, calculated based on a specific moving speed of users’ eyes (150 degrees/s). Sec. 4 will demonstrate that although the moving speed is fixed in the algorithm development, the performance is robust against the speed.

 

Fig. 2. Flow chart of backlight optimization based on MOGA

Download Full Size | PDF

 

Fig. 3. The process of obtaining the display image and the color breakup image from the backlight of a dimming block.

Download Full Size | PDF

The fitness function is obtained as follows. First, the distortion ΔE is the average color difference between the original image and the display image, as given by Eq. (1), where the color difference is calculated by the CIE76 color difference formula in Eq. (2). The color breakup value CBU is the average color difference between the input image and the color breakup image, as given by Eq. (3). Finally, by incorporating ΔE and CBU, Eq. (4) provides the fitness function, where the parameter α is a customized weight.

The flow in Fig. 2 then randomly initializes 30 groups of backlight values for every dimming block, and the fitness value corresponding to each group is evaluated by the above calculation method. After that, if the stop criterion is satisfied, the backlight with the maximum fitness is outputted; otherwise, the backlights are updated by the selection, crossover, and mutation operators. The three operators obey the conventional genetic algorithm [27].

The selection operator adopts the tournament selection strategy. Four backlights are randomly selected from the initialized backlights, and the one with the largest fitness is selected until 30 new backlights are chosen. The crossover operator uses the weighted average crossover method. Two backlight values are randomly picked from the backlights acquired by the selection operator as parents, and their offspring are generated by averaging the parents. The mutation operator uses the differential evolution method [28]. Three backlights are randomly picked from the backlights outputted by the selection operator and sorted in terms of fitness. The offspring backlight BL_new is obtained by Eq. (5), in which BL_f1 is the backlight with the largest fitness, BL_f2 has the middle fitness, and BL_f3 has the lowest fitness. F = 0.5 is a typical value in differential evolution algorithms [29,30]. Finally, half of the 30 offspring backlights are generated by the crossover operator and the other half by the mutation operator, used as new backlights for the next iteration.

Using the above MOGA with five weights (α = 0.1, 0.3, 0.5, 0.7, and 0.9) to optimize for the input image in Fig. 3, we achieve a backlight set at the Pareto optimality, as the front (the dashed red line) in Fig. 4(a) shows. The five points on the front denote backlights with different priorities between distortion ΔE and color breakup CBU. Furthermore, the input image is also processed by three traditional 3-field algorithms, i.e., LPD, 180Hz-Stencil, and the simple RGB scheme, as well as two typical 4-field algorithms, 240Hz-Stencil [16] and Edge-Stencil [17]. Their ΔE and CBU are also marked in Fig. 4(a). As a result, the traditional 3-field algorithms are far from reaching the Pareto optimality, demonstrating the significant improvement introduced by MOO. Additionally, the Pareto front’s bottom is very close to the 4-field algorithms, i.e., our Pareto-optimized 3-field algorithm can even achieve a comparable color breakup with conventional 4-field algorithms, further highlighting the benefit of MOO.

 

Fig. 4. (a) Color breakup (CBU) and distortion (ΔE) produced by the proposed MOGA-based algorithm with five weights α, three traditional 3-field algorithms (RGB, 180Hz-Stencil, and LPD) and two 4-field algorithms (240Hz-Stencil and Edge-Stencil). The dashed red line denotes the Pareto front. (b) Optimal backlight values under different α and corresponding display images and color breakup images.

Download Full Size | PDF

We like to discuss the effect of the weighting α further. Figure 4(b) shows display images and color breakup images corresponding to different α. A smaller α produces a larger proportion of the green component, meaning the redundant red and blue in the first field are less significant, thus resulting in a low distortion rate. On the contrary, increasing the green component’s proportion means more image content displayed in the first field and the other two more dimmed, thus suppressing color breakup better. In practice, the weight α should be customized according to specific application scenarios. Here, we choose α=0.3 to balance distortion and color breakup because this weight produces the minimum color breakup when the distortion is within the just noticeable difference (JND) of 2.3 in the CIE76 color space [31,32].

Finally, by parallelly optimizing every dimming block’s backlight using the MOGA above, we can get the multi-color field’s backlight.

2.2 Lightweight backlight generation neural network

The above iterative optimization for hundreds of dimming partitions is almost impossible to be performed in real-time. Therefore, we propose the LBGNN to achieve real-time backlight generation. The network structure and its training are introduced as follows.

2.2.1 Network structure

The structure of our LBGNN is shown in Fig. 5(a). An original image (1080 × 1920) is bilinearly interpolated to 3 × 288 × 512 as the input for a reduced calculation amount. The output is a color backlight pattern of 3 × 18 × 32. The feature extraction module of the network consists of three residual blocks (RBs) [33], and the feature fusion module consists of two depth separable convolutions (DSCs) [34]. The residual block uses the skip connection inside to realize the constant mapping of the network in Fig. 5(b), which effectively alleviates the degradation problem caused by too deep a network. When the input channel number is different from the output, they are aligned by a 1 × 1 convolution.

 

Fig. 5. Network structures of (a) the entire LBGNN, (b) the residual block, (c) the depthwise-separable convolution, and (d) the output layer.

Download Full Size | PDF

The depth-separable convolution is composed of depthwise convolution (DW) and pointwise convolution (PW) in Fig. 5(c). As Fig. 6(a) shows, in standard 3 × 3 convolution, a convolution kernel is convolved with the input feature layer to obtain an output feature layer. The depth of the convolution kernel is the same as the input channels, and the number is the same as the output channels. A total of 18,432 parameters need to be trained. Figure 6(b) shows the convolution process of DSC1 in LBGNN. A 3 × 3 convolution kernel merely performs convolution on one input channel in the DW convolution, so the channel number in the middle feature layer is the same as the input channel. However, the feature information of different channels in the same spatial position is not effectively utilized. Therefore, PW convolution uses a 1 × 1 kernel to further fuse different channels’ features. Thus, a total of 2,624 parameters need to be trained. Compared with standard convolution, the above structure can effectively reduce parameters, but it may sacrifice the accuracy of feature extraction. Therefore, DSC is used in the feature fusion module.

 

Fig. 6. (a) Standard convolution process and the number of parameters to be trained. (b) Depthwise separable convolution process and the number of parameters to be trained.

Download Full Size | PDF

The stride and padding of all 3 × 3 convolution layers are one pixel to ensure that the width and height dimensions of the feature layer remain unchanged after convolution. The convolution layer inside RB and DSC is followed by the Batch Normalization (BN) layer [35] to suppress the vanishing exploding gradient and accelerate the training. A pooling layer is set between each function block to gradually reduce the width and height dimensions to match the output backlight size. The output layer makes the output channel number match the number of color backlight channels through a 1 × 1 convolution kernel in Fig. 5(d). Finally, the total parameter number is 78,896.

2.2.2 Training

Training and test images come from the DIV2K database [36], containing 900 2 K resolution images. First, all images in the database are bilinearly interpolated to 1080 × 1920. Our MOO proposed in Sec. 2 is applied to 900 images to obtain 900 backlight patterns, of which 700 images are used as our training set, 100 are used as the validation set, and the remaining 100 for the test set. The LBGNN’s parameters are initialized using the Kaiming initialization [37]. Equation (6) gives the loss function, which is the mean square error between the backlight outputted by the network and the backlight in the training set. The parameters of the Adam optimizer [38] are set to β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, and the learning rate is 0.0002. The training time of 300 iterations is approximately four hours (NVIDIA GeForce RTX 3060).

2.2.3 Test

With the Pytorch framework on an RTX 3060 GPU, the training result of LBGNN are shown in Table 2 in terms of accuracy and inference time, where the accuracy is counted as the mean squared error (MSE) between a backlight produced by the neural network and its counterpart in the MOO-generated test set. At the same time, as Table 2 shows, we train two other neural networks for comparison, whose feature extraction module and feature fusion module are wholly composed of RBs and DSCs, denoted as all-RB and all-DSC, respectively. The two comparative networks differ from LBGNN in parameter numbers to investigate the parameter number’s influence.

Table 2. Performance of different neural networks on the test set

View Table | View all tables in this article

Our LBGNN achieves an inference time of 2.3 ms per frame, which meets the real-time requirement (2.8 ms per frame for a 360 Hz field rate). In contrast, the all-RB network achieves nearly the same MSE but has a slower inference time that cannot satisfy real-time due to more parameters. On the other hand, the all-DSC network accomplishes a faster inference time with fewer parameters but produces a worse MSE (0.62% vs. 0.47%). Note that the MSE indicates how a backlight predicted by the neural network approximates its MOO-generated reference but does not directly suggests the final distortion and color breakup. Therefore, the absolute value of MSE should be further investigated concerning distortion and color breakup, as discussed in Sec. 3. That section will demonstrate LBGNN (MSE = 0.47%) can guarantee the distortion and color breakup performance be consistent with the MOO-generated reference. However, the all-DSC network (MSE = 0.62%) cannot.

Benefiting from the lightweight network structure, an image of any size inputted into our LBGNN will be first resized to 288 × 512. Thus, the inference time (i.e., the backlight generation time) is little affected by the resolution of input images. In addition, in an actual display, besides backlight generation, grayscale compensation should also support real-time, achieved by previous studies [22–25] and commercial products with various resolutions. Therefore, even with a resolution higher than FHD, real-time display driving is feasible by combining the proposed backlight generation method and the developed grayscale compensation approaches.

2.3 FSC-LCD driving based on LBGNN

The LBGNN above predicts a multi-color backlight BL₁(BL_R, BL_G, BL_B) for an input image, adopted as the first field’s backlight of our 3-field driving, followed by two mono-color fields, as Fig. 7 shows. Backlight signals of the two mono-color fields and LC transmittance of the three fields are determined per conventional FSC algorithms, introduced below. Because the proposed method resembles existing Stencil-family algorithms by adopting one multi-color field, we name it LBGNN-Stencil.

 

Fig. 7. LC transmittance and backlight signals in the LBGNN-Stencil 3-field driving.

Download Full Size | PDF

First, the real backlight distribution on the LC layer needs to be simulated, considering the spread light profile produced by a mini-LED. Here we adopt the widely used two-dimensional Gaussian function as the light spread function (LSF), as given by Eq. (7). By adjusting σ, the LSF of different backlight configurations can be simulated. In Fig. 8(a), our dimming block containing 4 × 4 mini-LEDs is critically uniform when σ=10, so σ=10 is selected. By adopting the backlight signal in Fig. 7, Fig. 8(b) shows its real backlight distribution $B L_1^{\prime}$ ($B L_R^{\prime}$, $B L_G^{\prime}$, $B L_B^{\prime}$) by convoluting the backlight signal with the LSF. Note that the value of σ varies with the configuration of a real backlight unit, which trivially affects our LBGNN-Stencil algorithm because it only depicts a sufficient backlight uniformity.

 

Fig. 8. (a) Uniformity of a single dimming block (4 × 4 mini-LEDs) with different σ the LSF. (b) The real backlight distribution obtained with σ=10.

Download Full Size | PDF

Next, the LC transmittance T₁ of the first field is calculated by Eq. (8), where I_G is the green channel’s luminance of the input image. The front-of-screen image I₁ of the first field presents the green content as far as possible, as well as some red and blue content. The last two fields show the remaining red and blue luminance, I₂ and I₃. In this way, Eq. (9) provides the calculation method of I₂ and I₃, where I_R and I_B are the red and blue channels’ luminance of the input image, respectively.

The backlight signals BL₂ and BL₃ are obtained by calculating the maximum value of I₂ and I₃, respectively, to fully display the image’s remaining red and blue luminance. $B L_2^{\prime}$ and $B L_3^{\prime}$ are obtained from the real backlight simulation, then the LC transmittance T₂ and T₃ can be calculated by Eq. (10).

Multiplying the real backlight with the LC transmittance for the three fields results in three front-of-screen images, merging which reproduces the input image, as Fig. 9(a) shows. To further investigate color breakup, we simulate color breakup images by shifting each field for 15 pixels, as Fig. 9(b) shows. We recall that the trivial effect of the shift value will be discussed in Sec. 4.

 

Fig. 9. (a) Displaying three fields in front of the screen quickly and sequentially reproduces the input image. (b) Color breakup acquired by shifting the fields.

Download Full Size | PDF

3. Result

In this section, we verify the proposed LBGNN-Stencil through simulation and experiment. Distortion and color breakup are objectively evaluated using simulated front-of-screen and color breakup images. Also, color breakup is experimentally verified using real images sequentially displayed. We test the traditional RGB, 180Hz-Stencil, LPD, and the proposed LBGNN-Stencil methods on the test set containing 100 images. Furthermore, to investigate the effect of network accuracy, the all-DSC network replaces LBGNN as the fifth method.

Obeying the conventions in the FSC area [19–21], the average CIEDE2000 color difference is adopted to evaluate distortion (ΔE) and color breakup (CBU). Figures 10(a) and (b) show ΔE and CBU of the 100 test images corresponding to the five driving methods by boxplots. The RGB method produces no distortion and terrible color breakup. The LPD method can faithfully reproduce the original image with little distortion by aligning the backlight’s color gamut with the image content. However, color breakup suppression by LPD highly depends on image content; thus, various test images lead to severe color breakup. The 180Hz-Stencil better suppresses color breakup by leaving dimmed red and blue content in the two mono-color fields, but, as discussed before, this method sacrifices distortion much, even beyond the JND of one in terms of CIEDE2000. Finally, the proposed LBGNN-Stencil achieves superior performance to the above methods. Its objective CBU metric is improved by 21.09% compared with 180Hz-Stencil. Meantime, the distortion is well controlled within the JND of one, demonstrating balanced distortion and color breakup as we expect. In addition, if replacing the LBGNN with the more lightweight all-DSC network, though the color breakup performance is approximately maintained, the distortion easily exceeds the JND. Hence, the degradation in network accuracy introduced by the all-DSC network is unacceptable despite its faster inference time.

 

Fig. 10. Performance of RGB, LPD, 180Hz-Stencil, all-DSC network, and the proposed LBGNN-Stencil methods: (a) distortion ΔE of 100 test images with a red line indicating the JND of one in terms of CIEDE2000; (b) color breakup CBU of the 100 test images.

Download Full Size | PDF

Four diverse images in the test set are selected to further demonstrate the results. Figures 11(a)∼(d) show their color breakup images and front-of-screen images produced by RGB, LPD, 180Hz-Stencil, and LBGNN-Stencil algorithms. Here the all-DSC network is not considered because it has been proven unqualified to replace LBGNN.

 

Fig. 11. Simulated synthesized front-of-screen images and color breakup images under RGB, LPD, 180Hz-Stencil, and the proposed LBGNN-Stencil methods: (a) Castle, (b) Girl, (c) Tower, and (d) Food.

Download Full Size | PDF

With no effort to address color breakup, RGB produces the most severe in all cases. In Figs. 11(a) and (b), LPD effectively works for images with consistent colors, whereas it still induces apparent color breakup for the images with diverse colors in Figs. 11(c) and (d), especially the vivid picture in (d). Meantime, in Figs. 11(c) and (d), 180Hz-Stencil suppresses color breakup better than LPD at the cost of a more severe distortion, reflected by subjective observation and ΔE exceeding the JND. By contrast, in all cases, LBGNN-Stencil eliminates color breakup to almost invisible, while distortions are controlled within the JND.

For experimental verification, a 180-Hz LCD (ASUS VG295QM) quickly presents three field images to mimic a 3-field FSC-LCD with a frame rate of 60 Hz. Color breakup images are captured by rapidly moving a camera to mimic the movement between an eye and the display, as Fig. 12 shows. The photographs match the simulated color breakup images in Fig. 11 well. By subjective observation, the color breakup performance of the four algorithms is consistent with the simulation-based result. In the experiment, image fidelity is not investigated because the capturing unavoidably introduces uncontrollable errors. We suggest evaluating image fidelity by comparing synthetic field images with the original image is reliable, as the above simulation-based objective evaluation did.

 

Fig. 12. Front-of-screen images of the test images are sequentially displayed by a 180 Hz LCD to mimic an FSC-LCD, and captured images with color breakup under RGB, LPD, 180Hz-Stencil, and the proposed LBGNN-Stencil methods: (a) Castle, (b) Girl, (c) Tower, and (d) Food.

Download Full Size | PDF

4. Discussion

4.1 Relationship between the moving speed and color breakup

In the MOO-based backlight generation before (Sec. 2.2), we simulated color breakup images by shifting each field for 15 pixels. By comparing color breakup images and original images, color breakup levels were evaluated to push the optimization forward. According to the geometry in Fig. 13, the number of shifted pixels is determined by the moving speed of eyes and the display’s field of view (FOV), refresh rate, and resolution, as given by Eq. (11). For the tablet-style screen with the specifications in Table 1, the above shift value of 15 pixels was acquired under a typical moving speed of 150 degrees/s.

 

Fig. 13. Geometry to calculate the shift value, where the screen size is 16.9 cm by 30.0 cm, the display resolution is 1080 by 1920, and the viewing distance is 30 cm for a FOV of 52 degrees.

Download Full Size | PDF

Nevertheless, the shift value varies with the moving speed; thus, whether the algorithm based on the specific shift value can effectively work for other speeds is problematic. Usually, the moving speed ranges from 100 to 300 degrees/s [13,39,40]. Hence, we adopt five speeds in this range to obtain shift values and corresponding color breakup images for LBGNN-Stencil and the previously investigated traditional methods (RGB, LPD, and 180Hz-Stencil). CBU values in terms of average CIEDE2000 color difference are recalculated for the 100 test images, as Table 3 shows. Figure 14 shows color breakup images for one of the test images. As a result, the speed increase leads to a more significant color breakup for the three traditional methods. At the same time, LBGNN-Stencil exhibits more robustness against the speed, i.e., an improvement consistently exceeding 20% compared to traditional methods. The robustness essentially comes from the proposed method’s superior ability in color breakup suppression since the image content likely to induce color breakup is primarily concentrated in the multi-color field. Therefore, although we adopted a fixed moving speed to develop LBGNN-Stencil, the performance is robust against the speed.

 

Fig. 14. The color breakup images under RGB, LPD, 180Hz-Stencil, and LBGNN-Stencil methods at different moving speeds.

Download Full Size | PDF

Table 3. Average CBU of 100 test images at different moving speeds (corresponding pixel shifts marked)

View Table | View all tables in this article

4.2 Application scenarios

This study is based on direct-lit LCDs, the default choice when discussing mini-LEDs and local dimming. In comparison, in an edge-lit LCD, LEDs are placed at the edge(s), making it difficult to achieve local dimming. Nevertheless, some studies have shown that the edge-lit LCD can also adopt local dimming to a certain extent. For example, Yoon et al. designed a semi-partitioned light guide plate (LGP) and placed LEDs in the middle of the LGP to illuminate a specific block [41]. Masuda et al. realized 2D local dimming by stacking multiple LGPs [42]. Chen et al. embedded edge-lit mini-LEDs into U-shaped grooves at the corners of sub-LGPs and controlled the brightness of each zone by combining edge-lit and direct-lit [43]. Based on the above edge-lit LCDs with local dimming, our proposed FSC method can also be utilized.

Furthermore, as mono-color LEDs (i.e., RGB) but not white-light LEDs are used for the backlight, we like to discuss the light source’s influence on FSC-LCDs. Currently, quantum dot (QD) technology is emerging for more saturated RGB mini-LEDs and a wider color gamut [44]. However, the more saturated primaries may induce more visible color breakup due to more significant inter-field color differences [32]. Multi-primary QDs can alleviate such color breakup [45], however, at the cost of a higher refresh rate. Therefore, the proposed low-color-breakup driving should be valuable for wide-color-gamut FSC-LCDs by simultaneously controlling color breakup and image fidelity for a 3-field scheme.

Finally, the requirements for micro-displays in the VR/AR era include high resolution, high brightness, high contrast, compactness, high reliability, etc. Besides the low color breakup and high image fidelity achieved by the proposed driving method, our FSC-LCD intrinsically offers high spatial resolution and light efficiency. Meantime, the mini-LED backlight with local dimming provides contrast and thickness comparable to OLEDs. These advantages make FSC-LCDs a strong competitor for high-end VR/AR displays [46]. In comparison, it is still challenging for OLEDs to maintain high brightness with a long working life. Reflective micro-displays need a light engine that occupies a considerable room.

5. Conclusion

Three-field FSC driving is preferred over 4-field driving due to a lower refresh rate; however, existing 3-field driving methods are challenging to balance image fidelity and color breakup for various image content. To address this problem, we first built a backlight data set using MOO that simultaneously considers distortion and color breakup. Based on this training set, we proposed the LBGNN to generate the first field’s backlight with real-time implementation. Our LBGNN-Stencil driving reduces color breakup by more than 20% (in terms of average color difference) compared with 180Hz-Stencil, which is currently the best algorithm in color breakup suppression. Meantime, LBGNN-Stencil controls the distortion within the JND of human eyes. Therefore, the proposed method achieves extremely low color breakup with imperceptible distortion in real-time, making it promising for high-brightness and high-resolution applications such as VR, AR, and HUDs.