Open Access
Add to BibSonomyAbstract
A multi-beam bilateral teleoperation system of holographic optical tweezers accelerated by a graphics processing unit is proposed and evaluated. This double-arm teleoperation system is composed of two haptic devices and two laser-trapped micro-beads. Each micro-bead is trapped and moved following the trajectory of each haptic device, and the forces to which the micro-beads are subjected, which are generated by Stokes drag, are measured and fed back to an operator via the haptic devices. This real-time telexistence was quantitatively evaluated based on the time response of the trapped beads and the fed-back forces. And the demonstration of touching red blood cells shows the effectiveness of this system for biomedical application.
©2012 Optical Society of America
1. Introduction
Optical tweezers provides a suitable method for noncontact micromanipulation in closed workspaces [1, 2]. Therefore, there are needs to manipulate living cells such as red blood cells for analyzing the details such as the mechanical characteristic [3–5]. Since one laser beam can manipulate only one object, multi-beam optical tweezers is needed for multiple degree of freedom (DOF) manipulation. The time-shared scanning (TSS) method, the generalized phase contrast (GPC) method, and the computer-generated hologram (CGH) method are well-known multi-beam generation methods [6–8]. In the present study, we focus on holographic optical tweezers (HOT) which uses the CGH method, which can be extended to three-dimensional manipulation. The manipulability of HOT was greatly limited by the speed of the beam control. This was mainly due to the computation time for generating the hologram. However, recent progress in parallel computing using the graphics processing unit (GPU) contributed to a breakthrough regarding the problem [9–11]. Since the calculation of optics and photonics exhibits parallelism, parallel computing is suitable to compute optics and photonics phenomenon numerically. This real-time generation of hologram made it possible to manipulate mass small objects real-timely via HOT. Thus the position information inputted by an operator immediately influences the changes of position or status of the objects in micro-world. Here, HOT acts as an interface between the operator and the micro-world for telexistence such as an input device of computer mouse or keyboard [12–15]. Therefore, it is important to feed back the forces to which the manipulated objects are subjected, as was discussed in previous studies using one laser beam [16, 17].
In this research, we describe a newly developed bilateral teleoperation system that is capable of multiple manipulations and multiple force feedback. The proposed system is composed of a double-arm master-slave system, which uses HOT accelerated by a GPU. This system is controlled with a force reflection type bilateral master-slave control technique [18]. The masters are two haptic devices (Sensable PHANTOM Omni), and the slaves are two laser-trapped micro-beads made of polystyrene (diameter: 3 μm, refractive index: 1.59). A schematic diagram of the proposed system is shown in Fig. 1 . The positions of the micro-beads are controlled along the trajectories of the haptic devices, and the forces to which the slaves are subjected are measured and fed back to the haptic devices. We describe the method of real-time position control and force measurement used herein. We experimentally evaluated this system based on the time response of the positions and measured forces in two-dimension for force sensing and detailed analysis. And we demonstrated touching red blood cells to reveal the effectiveness of this system for biomedical application.
Fig. 1 Schematic diagram of the teleoperation system using holographic optical tweezers. The coordinate systems of the master, slave, camera, and hologram are denoted as Σm, Σs, Σc, and Σh, respectively.
2. Teleoperation systems
2.1 Optical system
The optical system of HOT is shown in Fig. 2 . An infrared laser source (Yb fiber laser, IPG Photonics) having a wavelength of 1064 nm is used and introduced to a neutral density filter (ND) to adjust the laser power. A polarizer (PL) is placed for adjusting the polarization plane to the aligned nematic liquid crystal display of a spatial light modulator. A beam expander (L3, L4) expands the diameter of a laser beam so that the display of a spatial light modulator is irradiated with a laser beam as large as possible. An iris diaphragm (Iris) reshapes a laser beam diameter. Then, a polarized laser beam is introduced to a spatial light modulator (SLM: Hamamatsu Photonics X10468-03, frame rate: 60 Hz) [19]. Then, the two-dimensional phase distribution of a laser beam is modulated with the SLM which displays a Kinoform hologram. The displayed hologram is transferred to the back focal plane of an objective lens (OL: Olympus UPLSAPO 100XO, 100x, N.A.: 1.40) via relay lenses (L1, L2) and dichroic mirror (DM). The relay lenses scale down the diameter of a laser beam for increasing the max spatial frequency of the modulated laser beam. This process contributes to the increase of max diffraction angle and the expansion of working space. Multiple laser spots are generated on the focal plane of the OL. Then, target objects are trapped at the laser spots.
Holographic optical tweezers has a problem of generating ghost traps or disturbances (i.e. disturbances of control system). This trap is caused by a zero-order beam which is a non-diffracted beam on the SLM. Especially, the power of a zero-order beam increases in dynamic control because of the decrease of the diffraction efficiency on the SLM. The reduction of diffraction efficiency depends on the response of the liquid crystal of a SLM. Therefore, a zero-order beam should be removed for stable and high-speed manipulation. Here, the optical system for filtering a zero-order beam is implemented [20]. A composite hologram ϕ’h which is composed of a Fourier hologram ϕh and a Fresnel phase lens ϕlens separates zero-order beam and first-order beam as shown in Fig. 3 , and calculated as follows:
Fig. 3 The phase distribution of centered hologram is an example of composite hologram by using a Fresnel phase lens shown in left side and a Fourier hologram shown in right side for separating a zero-order beam and a first-order beam. This operation contributes to eliminate the disturbance caused by a zero-order beam.
A Fresnel lens of concave type is calculated as follows [21]:
Here, r is the radius from the center of a hologram (i.e. the distance from an optical axis). λ is the wavelength of a laser beam. f is the desired focal length of a Fresnel phase lens.Furthermore, L3 and L4 are placed not to be a conjugate relation; more specifically the distance between these lenses is longer than the total focal length for converging a zero-order beam at the place of spatial filter (SPF). This SPF has a circular reflection part made of Cr/Au coat. SPF reflects and removes the focused zero-order beam which is not diffracted at the SLM. On the other hand, a diffracted beam generates multi-beam at a Fourier plane with this optical system. The efficiency of this filtering system was experimentally confirmed.
In this optical system, the designed parameter of maximum workspace diameter and resolution was 160 μm and 0.3 μm. In fact, workspace diameter is more than 160 μm, and resolution is 0.2 μm at the center position, and these results were experimentally confirmed.
2.2 Control and force feedback system
The block diagram of a bilateral control system is shown in Fig. 4 . Master position vector Xm is transformed to position Xcm in the camera coordinates (common coordinate) system of camera images with Tpmc. The positions of masters and slaves are managed in this common coordinate system. Then, Xcm is transformed to Xsm in the slave coordinate system with Tpcs. The CGH generator calculates holograms ϕ (sampling size: 512 × 512) using a GPU (NVIDIA Tesla C1060) in real time. The Gerchberg-Saxton method is used to calculate the holograms [22]. The processing rate of a GPU is 250 Hz and 30 times faster than the rate (8 Hz) of a central processing unit (Intel Core i7 975). Using the optical system, laser spots are generated at Xsi and slaves settle at Xs in the slave coordinate system.
Here, we propose a force measurement method using the analysis of microscopic camera images shown in Fig. 5 . If the trap stiffness parameter Kt is known, then the forces to which slaves are subjected can be estimated as follows [23]:
where Kt is assumed as a constant in space-time and is measured by the drag force method [24]. For measuring Kt, we use two laser-trapped beads, the total laser power of which are the same as that of the experimental condition listed in Table 1 .Fig. 5 Schematic diagram of the force measurement by using the image analysis of microscopic camera images.
Table 1. Experimental condition for the evaluation of bilateral teleoperation
Microscopic images are acquired by a high-speed CCD camera (Point Grey Research Grasshopper GRAS-03K2C, pixel resolution: 640 × 480, frame rate: 200 fps). Xs is transformed to Xcs by analyzing images obtained from the CCD camera. We use the normalized cross correlation method for detecting Xcs. Ideally, we would like to determine the positions of the multiple beams in the common coordinates Xcsi with the CCD camera. However, the laser positions are invisible because the lasers are scattered or occluded by slaves or other objects. Therefore, estimation of Xcsi is important for positioning and force sensing.
Here, we employ the simple way of using the position of the master Xcm for measuring the slave forces. Using Xcm, Xcs, and the unit exchange factor (from the common coordinates to the slave coordinates) Kpcs, we can estimate Fs without Xcsi as follows:
The detected forces are on the order of pN and are too small to sense using haptic devices. Therefore, we need to amplify the detected forces using force feedback gain sf. The amplified forces Fs’ (on the order of mN) are fed back to the masters. The refresh rate of the forces is approximately 100 Hz because of the limitation of the control system. Therefore, the proposed system synchronizes each control element at 100 Hz, except for the SLM. Ideally, the refresh rate must be 1 kHz for continuous force feedback [25].
The manipulation workspace is limited by the field of view of the microscope, and we limited the workspace of this experiment to 23 μm × 17 μm to increase the resolution of force sensing which depends on the ratio by CCD camera pixels to observed area. This space is inside the laser movable area, which is decided according to the maximum spatial frequency of the SLM (25 lp/mm) [26]. And the positioning resolution is 0.2 μm, which is evaluated experimentally. This resolution is decided based on the calculation size of the hologram.
3. Evaluation of teleoperation system
For the evaluation of bilateral teleoperation system, the manipulation of slaves was done in the water solvent. The experimental condition is shown in Table 1 and the experimental results are shown in Fig. 6 . Figure 6 shows the clockwise operation of the two masters and clockwise rotation of the two slaves. In the slave movement, the cross markers indicate the target positions of multi-beam Xcm, and the squares indicate the slave positions Xcs. These markers are processed and displayed on the CCD camera images in real time. The square tracked each slave smoothly, and detection of Xcs using image analysis was successful. The crosses (Xcm) approximately agreed with the squares (Xcs) under static condition (at 0 ms in Fig. 6). This is because this system uses the equality of Xcm and Xcs of a steadily trapped bead under static condition for positioning calibration, and then Tpcs is decided as following linear map:
Here, Tpsc is coordinate transformation from slave coordinate to common coordinate by using camera system shown in Fig. 4, and this parameter is constant which depends on the hardware. Htrans and Hrot shows the coordinate transformation about the translation (offset) and the rotation, and spcs is the motion ratio from common coordinate to slave coordinate. Tpcs is necessary to compensate the difference between slave coordinate system and common coordinate system. Figure 7 shows the time response of the position and the force associated with the left-side master-slave. With respect to the position response, the slave position follows the master position with a time delay (Fig. 7(a)). And the measured slave force indicates the tendency of Stokes drag compared with estimated Stokes drag calculated with the slave velocity (Fig. 7(b)). But the slave force is approximately 20 times stronger than Stokes drag. This problem mainly comes from the time-delay and the deviation of multi-beam control. The slave forces can be measured by using the difference between slave positions Xcs and laser positions Xcsi. Ideally, we should use these parameters, but Xcsi cannot be observed. Therefore, the position instruction Xcm was used as substitute for Xcsi. But Xcm is not equal to Xcsi because Xcsi has the time-delay and the deviation against Xcm.Fig. 6 Images of the teleoperation of clockwise rotation. Upper images show the operation of masters and lower show the movement of slaves which moves along the trajectory of masters.
Fig. 7 Results of bilateral teleoperation of the left-side master-slave. The upper one graph shows the position and the lower two graphs show the force. -X or -Y indicates the position or force of x-axis or y-axis. In the force graphs, right-side y-axis shows estimated Stokes drag calculated from the slave velocity. (a) Position response. (b) Left-side y-axis is slave force Fs and right-side is Stokes drag. (c) Left-side y-axis is modified slave force fs and right-side is Stokes drag.
As for time-delay, ramp response of Fig. 8 shows the time-delay Td is 70 ms, and it shows the large error between Xcsi and Xcm. It is interesting that the Td is almost same to the response time of the liquid crystal of SLM (response of the liquid crystal: rise 30 ms, fall 80 ms). Therefore, we proposed the modified slave forces fs which were compensated the time-delay. If the time delay Td is assumed to be 70 ms and constant, modified slave force fs is indicated as follows by using this relation (Xcsi(t) = Xcm(t-Td): Td = 0.07 s):
Fig. 8 Ramp response of a laser beam and a slave against position instruction of a ramp input in trajectory control. Xcsi was measured by observing a laser beam which was reflected by a mirror set up on the plane of OL.
The modified slave force which was calculated by applying this modification to previous experimental data is shown in Fig. 7(c). The difference between modified slave force and Stokes drag is smaller than that between slave force and Stokes drag (Fig. 7(b)), and shows the tendency of Stokes drag. This modified slave force has time-delay but the accuracy of slave forces was improved.
As for deviation, the static condition at 0 ms of Fig. 7(a) shows the static deviation of the slave position against the master position. This deviation affects the error of feedback forces such as the force in the static condition at 0 ms of Fig. 7(c). This deviation comes from the positioning resolution limit and the calibration error. The positioning resolution of this system is 0.2 μm and 13% of the radius of a bead (diameter: 3 μm). We think the tendency of slave forces is obtained regardless of the error, but this error should be compensated for accurate force measurements. On the other hand, the calibration was done by one steadily trapped bead, and Eqs. (6), 7) was used for coordinate transformation. And in this calibration, we used two positioning points of a trapped bead because of two-dimensional manipulation. But the deviation occurred. It is thought that there are two reasons. First, the positioning resolution limit affects the calibration error. Second, the linear approximation of this calibration is not sufficient and we need higher approximation. This problem about positioning deviation is our future works.
4. Demonstration of teleoperation
For evaluating the effectiveness of the bilateral teleoperation for practical application, the demonstration of touching red blood cells of horses was done. The output laser power from OL was set to 100 mW and the optical trap stiffness Kt was 64 pN/μm. And force feedback gain sf was set to 2.5 × 1010 for generating strong force feedback because the error between slave forces and actual forces was reduced by using modified slave forces. This teleoperation was done by squeezing red blood cells with two slaves as shown in Fig. 9 . Here, two red blood cells which adhere to each other were caught by the slaves. The modified slave forces are indicated as force vectors at the center of slaves. Figure 9 shows that the contact of slaves to the red blood cells was successfully detected.
Fig. 9 Demonstration of bilateral teleoperation of touching red blood cells. Upper images show slave movements and lower graph shows the magnitude of modified slave forces.
5. Conclusion
In this research, we have demonstrated multi-beam bilateral teleoperation of holographic optical tweezers accelerated by a GPU. This system can feed back two forces to which two slaves are subjected. The property of this method is multi-trapping and multi-force-sensing. Therefore, it is possible to squeeze or strain micro-scale objects dexterously with measuring forces at multiple points. We discussed the methodology and the problem associated with the accuracy and the response of multi-beam control. Especially, the response time is big problem in high speed manipulation using the GPU acceleration because it causes the error of force sensing. And we proposed the solution of this problem by accepting time-delay of force feedback. Finally we tried to touch red blood cells to sense its hardness. This two-dimensional demonstration with the constructed system reveals the possibility of telexistence between the operator and the micro-world, which means that it is possible to analyze the mechanical characteristic of living cells in real time.
Acknowledgment
We express our gratitude to Dr. K. Miwa of Arai Laboratory at Nagoya University for the technical advices on the biomedical application. The present study was supported by the Japan Science and Technology Agency (JST) through the Core Research for Evolutional Science and Technology (CREST) program.
References and links
1. A. Ashkin, J. M. Dziedzic, J. E. Bjorkholm, and S. Chu, “Observation of a single-beam gradient force optical trap for dielectric particles,” Opt. Lett. 11(5), 288–290 (1986). [CrossRef] [PubMed]
2. D. G. Grier, “A revolution in optical manipulation,” Nature 424(6950), 810–816 (2003). [CrossRef] [PubMed]
3. A. Ashkin, J. M. Dziedzic, and T. Yamane, “Optical trapping and manipulation of single cells using infrared laser beams,” Nature 330(6150), 769–771 (1987). [CrossRef] [PubMed]
4. K. Svoboda, C. F. Schmidt, D. Branton, and S. M. Block, “Conformation and elasticity of the isolated red blood cell membrane skeleton,” Biophys. J. 63(3), 784–793 (1992). [CrossRef] [PubMed]
5. P. J. H. Bronkhorst, G. J. Streekstra, J. Grimbergen, E. J. Nijhof, J. J. Sixma, and G. J. Brakenhoff, “A new method to study shape recovery of red blood cells using multiple optical trapping,” Biophys. J. 69(5), 1666–1673 (1995). [CrossRef] [PubMed]
6. K. Sasaki, M. Koshioka, H. Misawa, N. Kitamura, and H. Masuhara, “Pattern formation and flow control of fine particles by laser-scanning micromanipulation,” Opt. Lett. 16(19), 1463–1465 (1991). [CrossRef] [PubMed]
7. R. L. Eriksen, P. C. Mogensen, and J. Glückstad, “Multiple-beam optical tweezers generated by the generalized phase-contrast method,” Opt. Lett. 27(4), 267–269 (2002). [CrossRef] [PubMed]
8. D. G. Grier, J. E. Curtis, and B. A. Koss, “Dynamic holographic optical tweezers,” Opt. Commun. 207(1-6), 169–175 (2002). [CrossRef]
9. N. Masuda, T. Ito, T. Tanaka, A. Shiraki, and T. Sugie, “Computer generated holography using a graphics processing unit,” Opt. Express 14(2), 603–608 (2006). [CrossRef] [PubMed]
10. L. Ahrenberg, P. Benzie, M. Magnor, and J. Watson, “Computer generated holography using parallel commodity graphics hardware,” Opt. Express 14(17), 7636–7641 (2006). [CrossRef] [PubMed]
11. M. Reicherter, S. Zwick, T. Haist, C. Kohler, H. Tiziani, and W. Osten, “Fast digital hologram generation and adaptive force measurement in liquid-crystal-display-based holographic tweezers,” Appl. Opt. 45(5), 888–896 (2006). [CrossRef] [PubMed]
12. J. Leach, K. Wulff, G. Sinclair, P. Jordan, J. Courtial, L. Thomson, G. Gibson, K. Karunwi, J. Cooper, Z. J. Laczik, and M. Padgett, “Interactive approach to optical tweezers control,” Appl. Opt. 45(5), 897–903 (2006). [CrossRef] [PubMed]
13. J. A. Grieve, A. Ulcinas, S. Subramanian, G. M. Gibson, M. J. Padgett, D. M. Carberry, and M. J. Miles, “Hands-on with optical tweezers: a multitouch interface for holographic optical trapping,” Opt. Express 17(5), 3595–3602 (2009). [CrossRef] [PubMed]
14. R. W. Bowman, G. Gibson, D. Carberry, L. Picco, M. Miles, and M. J. Padgett, “iTweezers: optical micromanipulation controlled by an Apple iPad,” J. Opt. 13(4), 044002–044004 (2011). [CrossRef]
15. K. Onda and F. Arai, “Robotic approach to multi-beam optical tweezers with computer generated hologram,” in Robotics and Automation (ICRA), 2011 IEEE International Conference (2011), pp. 1825–1830.
16. F. Arai, M. Ogawa, and T. Fukuda, “Indirect manipulation and bilateral control of the microbe by the laser manipulated microtools,” in Intelligent Robots and Systems, 2000. (IROS 2000). Proceedings. 2000 IEEE/RSJ International Conference (2000), pp. 665–670.
17. C. Pacoret, R. Bowman, G. Gibson, S. Haliyo, D. Carberry, A. Bergander, S. Régnier, and M. Padgett, “Touching the microworld with force-feedback optical tweezers,” Opt. Express 17(12), 10259–10264 (2009). [CrossRef] [PubMed]
18. R. J. Anderson and M. W. Spong, “Bilateral control of teleoperators with time-delay,” IEEE Trans. Automat. Contr. 34(5), 494–501 (1989). [CrossRef]
19. N. Mukohzaka, N. Yoshida, H. Toyoda, Y. Kobayashi, and T. Hara, “Diffraction Efficiency Analysis of a Parallel-Aligned Nematic-Liquid-Crystal Spatial Light Modulator,” Appl. Opt. 33(14), 2804–2811 (1994). [CrossRef] [PubMed]
20. M. Polin, K. Ladavac, S. H. Lee, Y. Roichman, and D. G. Grier, “Optimized holographic optical traps,” Opt. Express 13(15), 5831–5845 (2005). [CrossRef] [PubMed]
21. J. A. Davis, D. M. Cottrell, R. A. Lilly, and S. W. Connely, “Multiplexed Phase-Encoded Lenses Written on Spatial Light Modulators,” Opt. Lett. 14(9), 420–422 (1989). [CrossRef] [PubMed]
22. R. W. Gerchberg and W. O. Saxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik (Stuttg.) 35, 237–246 (1972).
23. A. Ashkin, “Forces of a single-beam gradient laser trap on a dielectric sphere in the ray optics regime,” Biophys. J. 61(2), 569–582 (1992). [CrossRef] [PubMed]
24. K. Visscher, S. P. Gross, and S. M. Block, “Construction of multiple-beam optical traps with nanometer-resolution position sensing,” IEEE J. Sel. Top. Quantum Electron. 2(4), 1066–1076 (1996). [CrossRef]
25. J. E. Colgate and J. M. Brown, “Factors affecting the Z-Width of a haptic display,” in Robotics and Automation, 1994. Proceedings., 1994 IEEE International Conference (1994), pp. 3205–3210.
26. O. Bryngdahl, “Geometrical transformations in optics,” J. Opt. Soc. Am. 64(8), 1092–1099 (1974). [CrossRef]
