1. Introduction

Laser Scanning Microscopy (LSM) techniques represent essential tools for multiple fields of science. Well known examples are medicine and biology where LSM variants such as Confocal Laser Scanning Microscopy (CLSM) or Multiphoton Laser Scanning Microscopy (MPLSM) provide the possibility to non-invasively acquire in-focus images from selected depths (a.k.a. optical sections) from both living and fixed specimens [1, 2]. The contrast mechanisms of a certain microscopy technique can be suitable to image specific aspects of an investigated sample, yet may exclude others [3], which applies as well to LSM techniques. For example Second Harmonic Generation Microscopy (SHG) is a perfect choice for imaging noncentrosymetric structures, no matter if these are fluorescent or not, but cannot be used to image isotropic structures, whereas Two-Photon Excitation Fluorescence Microscopy (TPEF) can image fluorescent structures, irrespectively of their isotropy, but cannot be used to image non-fluorescent ones [2, 4, 5]. For this reason, in order to acquire an in-depth understanding of an investigated sample, LSM users frequently need to image it using different LSM techniques and workmodes, under different acquisition configurations or at different moments in time. The resulting images typically need to be stored, retrieved and transmitted, together, so that the data sets can be easily associated, analyzed and interpreted at a later date or by different users.

A side effect that accompanies multi-modal and multiplexed LSM imaging consists in the need to manage sequences of images produced by using various techniques, workmodes and configurations. Relevant information about the image content and acquisition parameters are typically stored in additional files. These are usually associated to the LSM micrograph by file naming strategies (e.g. common file names or common file name elements). Such solutions are simple, but simplicity comes accompanied by the risk of accidental loss of the additional files or of confusion between the associated files and the related images. Such situations frequently occur, especially when the image sets are transferred multiple times between different users. The need for protection against accidental loss or mismatch can be eliminated by storing such information or parameters in the header of the image file. This strategy is prone however to the risk of the header itself to be lost upon unsupervised routine tasks, such as file format conversion. An alternative strategy consists in having some critical parameters stored into the image filename. Obviously, the filename offers limited storage capacity.

The solutions mentioned above are simple, rather efficient and quite robust, but it can be observed that each of these strategies has its own disadvantages. A more elegant solution, that to this moment had remained unexplored, could consist in the embedding of additional information in LSM images by watermarking [6]. We recall that watermarking represents the embedding of additional information (the watermark) into digital host objects (image, video, sound, text, etc.) [7]. While standard watermarking strategies introduce permanent distortion, reversible versions can recover the original host without any error [8]. The fact that additional information can be embedded with zero permanent distortions in digital images by reversible watermarking motivated us to investigate the use of such strategies for embedding into LSM images not only supportive textual information about the image content and acquisition parameters, but also supportive 2D data, see Fig. 1.

 

Fig. 1 The proposed strategy for connecting LSM micrographs based on reversible watermarking. The Slave Image is embedded together with optional supportive text data into the Master Image; both images and the optional supportive text data can be restored from the watermarked Master Image.

Download Full Size | PDF

In particular, our experiment was aimed at embedding into a LSM image of interest a second image, collected by a complementary technique on the same field-of-view (FOV). Throughout the paper we refer to the host image as a Master Image (MI), and to the supportive 2D image which is inserted into the MI, as a Slave Image (SI). The embedding of this complementary image, the SI, into the main one, the MI, would provide to a potential end-user of the main image the possibility to extract supportive information that would facilitate his data interpretation or analysis tasks. The proposed strategy is also relevant for associating pairs of LSM images collected on the same FOV under different acquisition configurations (e.g. different excitation wavelengths), different moments in time or at different depths in the sample volume. The potential advantage for the end-user is easy to predict. For example let us imagine a biophotonics scientist (BS) analyzing a CLSM MI based on fluorescence signals generated by tissue embedded exogenous agents (e.g. quantum dots). If the information included in the MI is sufficient for the BS to perform the analysis of interest, the BS can proceed on investigating the MI without having to extract the watermark. However, if additional information is needed for consolidating the BS’ understanding of the biomolecular context, the BS can resort to extracting a potential CLSM SI embedded into the MI by watermarking, which is collected on the same FOV and is based on endogenous fluorescence signals collected at different excitation/emission wavelengths. The SI could potentially be as well accompanied by supportive text data providing information about the acquisition conditions, the imaging parameters or other details of interest.

In this paper, we present the results of embedding into LSM images data collected on the same FOV by using complementary LSM variants. The availability of such supporting data could facilitate the interpretation and analysis tasks of an end-user. The proposed approach ensures that pairs of images are connected to each other without any risk of mistake. Furthermore, the fact that transferring one of the images in a pair allows the recovery of its corresponding pair eliminates the cost of requesting the latter from a database and transferring it. Obviously, if the examination of the embedded version is not enough (due to potentially necessary resampling or compression), one can still request and transfer the original files. Finally, since the embedding is performed by reversible watermarking, the original host image can be recovered at zero distortion.

2. Materials and methods

The image sets used as support in our experiment have been collected on Hematoxylin & Eosin (HE) slides containing skin tissue fragments affected by squamous cell carcinoma (SCC), which is one of the most common cutaneous neoplasms. These support samples had been previously collected from patients for routine histopathological examinations, and stored in the histopathology archives of the Clinical Emergency Hospital of Bucharest. The samples have been de-identified prior to the access by the authors, and their use in the present context did not require institutional review board approval. The two complementary LSM techniques used in this experiment were: CLSM and Transmission Laser Scanning Microscopy (TLSM). Although HE stained tissue fragments are typically used in diagnostic assays based on conventional light microscopy, the fluorescent properties of Eosin from HE-stained tissues have been exploited in multiple experiments based on fluorescence microscopy, e.g [9–12].

For our experiment, multiple regions of the support samples have been simultaneously imaged by CLSM and TLSM. By CLSM we have imaged the fluorescent signal emitted by the investigated samples upon excitation with a laser beam of 488 nm wavelength, at a specific depth, while with TLSM we have imaged the transmitted excitation light (488 nm) through the corresponding specimen volume. Examples of the image pairs collected with a 10x/0.3 N.A. objective, providing a FOV of 1mm x 1mm are presented in Fig. 2. The complementarity of CLSM and TLSM data is derived from the associated contrast mechanisms. While CLSM can be used to acquire optical sections at specific depths based on the reflected light or on the fluorescence emission, TLSM provides information on how the excitation light is transmitted through the whole specimen volume, and consequently on how the excitation light is absorbed in this volume. Freshly prepared, aqueous solution of pure Hematoxylin shows a maximum absorption at about 290 nm, but as the solution ages, two peaks appear, at 445 nm and at 560 nm. Upon further oxidation the height of these two bands is reduced, the isosbestic point being situated at 486 nm [13]. Thus, when using the 488 nm beam for imaging in TLSM the HE-stained SCC sample provides a reliable map of where Hematoxylin adhered to the tissue, without depth discrimination. Such information is easily interpretable by a trained pathologist, as the conventional optical microscopy images typically used in histopathology assays correlate well to the TLSM ones.

 

Fig. 2 Three sets of CLSM/TLSM images collected on a HE stained tissue fragment affected by SCC.

Download Full Size | PDF

With CLSM we collected the fluorescence signals generated in specific optical sections for the same FOVs as with TLSM, using a pinhole aperture of 1 Airy. In our case the collected fluorescent signals were mainly generated by eosin, but also by several endogenous fluorophores present in the human skin such as flavins, carotinoids or bilirubin; Eosin is typically not regarded as a fluorochrome, but exhibits nonetheless high fluorescence emission. The absorption and emission maxima of eosin are 527 and 550 nm, respectively, with both peaks laying in the green light range; however, highly satisfactory emission intensities have been observed in previous experiments under excitation with blue light (450–490 nm), e.g. [14]. The images in Fig. 2 were collected using a Leica TCS SP1 system at 1024 × 1024 pixels resolution, without averaging. The histopathological interpretation of the TLSM/CLSM image sets used as support falls outside the scope of our work, and will not be discussed here.

Our experiment consists in selecting one of the images in a CLSM/TLSM pair as a MI, and the second one as SI, and inserting the SI into the MI by reversible watermarking. As it can be seen in Fig. 2, the CLSM images exhibit more details and better contrast than the corresponding TLSM ones. In the same time, there are significant content differences between the CLSM and TLSM images of the same pair. Therefore one can expect different performances when considering the CLSM image as MI and the TLSM as SI, or vice versa, an aspect that will be discussed in the next section.

The SI and the MI are of the same size. The maximum embedding bit-rates of recent re- versible watermarking schemes [15–17] are in the range of 2-3 bits per pixel (bpp). At such high bit-rates, the host images look very noisy. On the other hand, the embedding is almost imperceptible for bit-rates of around 1 bpp. Obviously, in order to meet such bit-rates, the size of the SI should be considerably reduced. Typically, the lossless compression is known to at most halve the size of the images. Therefore, lossless compression cannot provide the size reduction required for image embedding. There are several other methods to reduce the size of graylevel images: subsampling, quantization, lossy compression, etc. A very convenient solution is the use of lossy compression under the JPEG or JPEG 2000 standards, since it ensures a good trade-off between size and visual quality. Mixed solutions as, for instance, subsampling followed by compression can be used as well for data volume reduction.

Reversible watermarking appeared as an active topic in the last years [15–22]. The most efficient algorithms reported so far in the literature [17–22] are based on the prediction error expansion (PEE) scheme introduced in [23]. The principle of PEE is to predict pixels and, if the prediction error is less than a predefined threshold and no overflow or underflow appears, the error is expanded and a bit of data is embedded into the prediction error. The expansion consists in the addition of the prediction error to the current pixel. At detection if the same predicted value is found, one gets two times the original prediction error plus the embedded bit of data. The data bit is immediately recovered as the least significant bit of the prediction error and the original pixel value is restored. The pixels in the case of which the prediction error exceeds the threshold are shifted in order to provide at detection a larger prediction error than the one of the embedded pixels. The shift is of the order of the threshold. Finally, the pixels that can be neither embedded, nor shifted because of overflow or underflow are solved either by using flag bits or by using an overflow/underflow map.

Since at most one bit of data can be embedded into each pixel, the embedded bit-rate is limited to 1 bpp. Larger embedding bit-rates are obtained either by multilevel or by multibit embedding. The multilevel embedding simply chains two or more PEE procedures. The multibit embedding adds to the current pixel m times the prediction error with m > 1. Since the prediction error increases m + 1 times at detection, one can embed an integer word in [0, m], i.e., up to log₂(m + 1) bpp. The multibit embedding provides better results than the simple multilevel one.

The marking proceeds in raster scan order, pixel by pixel, starting from the upper left corner of the image. The detection proceeds in reverse order of the marking, starting with the last embedded pixel. Both the embedding bit-rate and the distortion depend on the quality of the prediction. The first PEE schemes were based on the high performance predictors developed for lossless compression, like the median edge detector (MED) [23] or the gradient-adjusted predictor (GAP). Nowadays schemes are based on noncausal predictors tailored for reversible watermarking like the average of the rhombus composed of the four horizontal and vertical pixels [24], the context adaptive rhombus [18] and so on. Particularly good results are provided by the scheme based on local prediction [21, 22] for the rhombus context. Supportive code for watermark insertion and extraction is available as Code 1, [25].

The basic idea of the local prediction based scheme is to compute, for each pixel, a least squares predictor in a block centered on the pixel. Half of the pixels within the block are modified (they have been embedded or shifted) and half are original, unmodified pixels. Once predicted, the current pixel is modified (either embedded or shifted). Obviously, in order to recover the same block at detection, the current pixel should not be considered in the computation of the current predictor. This problem is solved by replacing the pixel with an estimated value (the fixed rhombus average). We mention that the local prediction scheme provides the best performances (quality with respect to embedding bit-rate) reported so far in literature. From our tests it appeared that local prediction reversible watermarking is likely to outperform other recently proposed schemes [15–20] on LSM images as well. The results for embedding TLSM and CLSM images by using the local prediction reversible watermarking are presented in the next section.

Before going any further it should be noted that, together with the SI, one could also embed using the same watermarking strategy additional supportive textual information (Fig. 1) related to both SI and MI micrographs (e.g., acquisition parameters, imaging conditions, sample preparation protocols, content description, etc.). Since the size of the additional text-type information is usually of the order of 1-4 kBytes, its insertion does not represent a burden for the reversible watermarking algorithms. For 1024x1024 images, a data volume of 4 Kbytes is equivalent to an embedding bit-rate of only 0.031 bpp.

3. Results and discussions

We present the results of embedding by local prediction reversible watermarking the lossy compressed TLSM images of Fig. 2 into the corresponding CLSM ones and vice versa. The average embedding bit-rate for the three sets of complementary LSM images are presented in Fig. 3(a) and the compression rate provided by the JPEG 2000 compressor of Matlab (The MathWorks Inc., USA) are presented in Fig. 3(b) The distortions introduced by the watermarking and compression schemes are evaluated using the peak signal-to-noise ratio (PSNR) between the original and the current version of the image. As expected, there are major differences between the results on TLSM images and the ones on CLSM images, which will be discussed in the following paragraphs.

 

Fig. 3 Average reversible watermarking results (a), and compression results (b), for the CLSM/TLSM image sets.

Download Full Size | PDF

Let us first consider the CLSM images as MIs and the TLSM images as SIs. As it can be observed from Fig. 3(a), the embedding performance of the local prediction based reversible watermarking scheme on the CLSM images varies considerably from set to set. Local prediction, like all PEE schemes, relies on the correlation between neighboring pixels. This correlation is considerably weaker in CLSM images than in the case of natural images, mostly due to the high contrast nature of this type of image. Nevertheless, the CLSM images provide sufficient bit-rates to allow the insertion of the corresponding compressed TLSM images. In order to match the low embedded bit-rates offered by set 1 and 3, the quality of the compressed SI should be decreased. For instance, at bit-rates of 0.1 bpp the SIs appear slightly blurred (the corresponding average PSNR is 26 dB for set 1 and 25 dB for set 3). On the other hand, the distortions introduced by reversible watermarking on the MIs are imperceptible for the human observer. Thus, at embedding bit-rates of 0.1 bpp, the average PSNR between the original and the watermarked versions of the MIs is higher than 35 dB on set 1 and higher than 31 dB on set 3. The constraints are less severe for set 2. For instance, by embedding at 0.35 bpp one has an average PSNR of 31 dB for both MI and SI.

An example of embedding in a CLSM image (Fig. 2: set 3, image 4) its TLSM counterpart is presented in Fig. 4, together with a schematic diagram of the proposed watermarking strategy, illustrated in Fig. 4(a). While the given results refer to the entire image, in order to ensure good visibility only a quarter of the image is shown in Fig. 4. A PSNR of 26.58 dB was obtained between the original SI (Fig. 4(b)) and its compressed version to 0.155 bpp (Fig. 4(c)). While the loss in resolution is noticeable when directly comparing the two images, the overall aspect of the original SI is maintained. At a PSNR of 30.52 dB, there are no visible differences between the original MI (Fig. 4(d)) and its watermarked version at 0.156 bpp (Fig. 4(e)).

 

Fig. 4 Embedding a TLSM SI in a CLSM MI. (a) Schematic diagram of the proposed watermark insertion strategy; TLSM SI (b) before and (c) after compression at 0.155 bpp; the corresponding CLSM MI (d) before and (e) after being watermarked with the compressed SI.

Download Full Size | PDF

Next, let us consider the TLSM images as MI and the CLSM images as SI. The large embedding bit-rates available in this case ensure considerably more flexibility than in the case of embedding into CLSM images. One can select the embedding bit-rate in order to balance the quality of the master and slave images. Three scenarios can be envisioned: (I) the quality of the MI is prioritized; (II) both MI and SI are equally important; (III) the quality of the SI is prioritized. For the first scenario, an average PSNR of 38.5 dB for MI can be obtained at bit-rates of 0.6 bpp (for set 1 and 3) and 0.7 bpp (set 2), maintaining an acceptable quality for the SI as well (a PSNR of 26 dB on set 1, 25.75 dB on set 2 and 26.22 dB on set 3). For the second scenario, the same PSNR can be obtained for both MI and SI (28.19 dB on set 1; 27.63 dB on set 2; 28.79 dB on set 3). Because of the linear nature of the CLSM compression results (Fig. 3), we recommend to slightly prioritize the MI quality (30.65 dB for MI, 27.91 dB for SI on set 1; 30 dB for MI, 27 dB for SI on set 2; 30.83 dB for MI, 28.48 dB for SI on set 3). In other words, when both MI and SI are equally important, a slight increase of the SI PSNR does not justify a more consistent decrease of the MI PSNR. Finally, for the last scenario, an average SI PSNR of 29.3 dB (at 1.3 bpp), 28.6 dB (at 1.42 bpp) and 30.1 dB (at 1.28 bpp) on set 1, 2 and 3, respectively fulfills the requirement. The corresponding average distortion of MI is 25.17 dB (set 1), 24.9 dB (set 2) and 25.1 dB (set 3). We remind that the original MI can be completely restored by extracting the SI.

An example for each scenario of embedding into a TLSM image its CLSM counterpart is presented in Fig. 5. For the first scenario, the MI shown in Fig. 5(d) (38.75 dB at 0.621 bpp) was watermarked so as to embed the SI presented in Fig. 5(a) (26.82 dB at 0.621 bpp). As expected, there are no visible differences between the MI of Fig. 5(d) and the original version (Fig. 4(b)), while a mild loss of edge clarity can be noticed on the SI (Fig. 5(a)) when compared to the original (Fig. 4(d)). Next, the MI in Fig. 5(e) (32.9 dB at 0.935 bpp) was watermarked so as to embed the SI in Fig. 5(b) (28.41 dB at 0.933 bpp), considering both images as being equally important. Both the MI and SI do not present any visible artifacts when compared to their original versions. Finally, the MI in Fig. 5(f) (24.92 dB at 1.328 bpp) was watermarked so as to embed the SI in Fig. 5(c) (30.81 dB at 1.325 bpp). When compared to the original, the MI has a higher contrast and appears slightly noisy. There are no visible differences between the SI and its original version.

 

Fig. 5 Embedding a CLSM SI in a TLSM MI. First row: compressed versions at 0.621 bpp (a), 0.933 bpp (b) and 1.325 bpp (c) of the CLSM SI in Fig. 4(d); Second row: watermarked versions with 0.621 bpp (d), 0.935 bpp (e) and 1.328 bpp (f) of the TLSM MI in Fig. 4(b).

Download Full Size | PDF

Next, the number of test images was increased to over two hundred by adding seven additional sets of complementary CLSM/TLSM micrograph pairs collected on HE stained SCC tissue fragments. We have first examined the maximum embedding bit-rate for both CLSM and TLSM images and then we have investigated the embedding at lower bit-rates. On average, the bit-rate upper bound offered by the CLSM images is 0.41 bpp at an average PSNR of 24.19 dB. The average distortion introduced by lossy compression for the embedded TLSM images is 31.3 dB. By investigating the embedding at lower bit-rates, namely 0.1, 0.2 and 0.3 bpp, it appears that a good compromise is 0.1 bpp with an average PSNR of 25.8 dB for TLSM (no annoying artifacts) and 38.4 dB for CLSM images. For the case of embedding into the TLSM images, multibit PEE provides an average maximum bit-rate of 1.44 bpp at 21.95 dB and an average PSNR of 29.13 dB for the compressed CLSM images. The tests at lower embedding bit-rates shows that at least 1 bpp (26.88 dB for CLSM and 31.61 dB for TLSM) can be achieved in the case of all TLSM images.

The advantage of accessing the content of a pair of connected LSM images by storing/transmitting only one image comes at the expense of extra time for inserting and extracting the embedded image (compression time is negligible). The associated runtimes mainly depend on the image size, computer configuration, operating system and compiler. In the case of our experiment the average time for extracting the 1024 × 1024 pixels embedded LSM images was 1.2s. The same average time was required also for embedding, due to the symmetry of the local prediction reversible watermarking algorithm. As mentioned above, the processing time depends on the size of the image. Thus, for 512 × 512 images, the average time for embedding/extracting was 0.3s. The above runtimes relate to watermark insertion and extraction using the Desktop PC configuration that we have used: 3.06 GHz Intel Core2 Duo E7600 CPU, 2 GB DDR3 RAM and a 64-bit Windows 8.1 (Microsoft, USA) operating system; coding of the fast local prediction reversible watermarking scheme of [22] has been performed using the C++ environment (GCC 4.8 of the GNU compiler collection).

The experiments that we have performed so far refer only to using reversible watermarking for embedding pairs of LSM micrographs. However, the proposed strategy can be used also for Many-to-One (M-t-O) strategies, consisting in embedding into a single MI multiple SIs. A potential application of such M-t-O schemes could consist in inserting into an LSM MI SIs collected with more than one complementary LSM techniques. Morevoer, such approaches could also be extended to the 3D domain, in the purpose of inserting into a LSM MI SIs consisting of optical sections collected at different depths in the volume of the sample. It is important to note that in the case of such M-t-O strategies the data volume scales proportionally to the number of SIs, thus such approaches should include also additional stages for data volume reduction, e.g. lowering the digital resolution of SIs by resampling, or bit-depth reduction. Optimizing the data volume of an SI set so that it matches the embedded bit-rate provided by the host image at a quality that satisfies the application at hand represents an interesting research topic, which we plan to further explore in future work.

4. Conclusions

In this paper we propose a strategy based on reversible watermarking for firmly connecting complementary LSM images of the same FOV collected in different LSM workmodes, with different LSM techniques, at different acquisition configurations or at different moments in time. A data set generated by using such an approach consists in a single file (the MI) which hosts supportive 2D data (the SI). Thus by storing/transmitting a single LSM image, which maintains the original file format, the information of the embedded complementary image is made available. A potential end-user of the generated data set can either inspect the MI without resorting to the supportive SI, or extract the latter in order to acquire additional context knowledge that would facilitate his interpretation and analysis tasks. We have demonstrated the above mentioned concept by embedding into TLSM images corresponding CLSM versions, and vice versa. The same strategy can be used for any other pairs of LSM images depicting complementary information. Moreover, the proposed watermarking strategy can also be used for inserting in the MI along with SI additional supportive textual information that details the experiment (e.g. acquisition configuration, sample description, etc.). In order to avoid artifacts, we have proposed a strategy that requires the SI to be lossy compressed prior to the embedding procedure in order to match the available embedding bit-rate. Several cases have been investigated and no annoying artifacts have been observed for the watermarked images or for the compressed extracted ones. The proposed approach for embedding supportive 2D data into an MI, allows the recovery of the original host images at zero distortion.