The output of the Gabor filter bank using the proposed line filter presented at ICRA2014. The image on the left shows 2D US image of the needle in the agar phantom. In order to show the effect of Gabor filter, the orientation angle of the filter (θ) is changed with 1° increments. Segmented images are shown in the middle. Also, segmented pixels are automatically binarized using Otsu’s thresholding method. Note that the needle is inserted with 120°. When filter’s orientation is equal to 120°, needle’s visibility reaches to its maximum.
M. Kaya and O. Bebek, “Needle Localization Using Gabor Filtering in 2D Ultrasound Images”, In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 31 – June 7, 2014, Hong Kong, China, pp. 4881-4886. [Abstract]
In order to localized the needle tip in 2D US images, a Gabor-based needle tip localization method was developed. The method consists of two consecutive stages. In the first stage, needle axis is localized and in the second stage, the needle tip location is estimated. Localized needle tip is tracked using the Kalman filter.
• A simulation environment was used to evaluate the Kalman filter performance error in localization when misalignment between the US probe and needle occur. The result of the simulation is shown in the first part of the video.
• After Kalman filter parameters were tuned using the simulation, proposed needle tracking method was used to track the needle tip while the needle was being inserted into agar and gelatin based phantom. Results are shown in the second part of the video. Video on the left shows the raw needle insertion video. The results of the proposed localization, without filtering applied, is shown in the middle and the result of the proposed tracking method with Kalman filter is shown in the right.
M. Kaya, E. Senel, A. Ahmad, O. Orhan, and O. Bebek, “Real-time Needle Tip Localization in 2D Ultrasound Images for Robotic Biopsies”, In Proceedings of the 17th International Conference on Advanced Robotics (ICAR), pp. 47-52, July 27-31, 2015, Istanbul, Turkey. DOI: 10.1109/ICAR.2015.7251432 [Abstract]
This video shows motion analysis of US frames while the needle is inserted into agar-based phantom. In order to analyze motion between consecutive frames, Lukas-Kanade optical flow method was used. Analyzed motion information was represented as vector field and overlaid onto the current frame.
M. Kaya, E. Senel, A. Ahmad, and O. Bebek, “Visual Tracking of Biopsy Needles in 2D
Ultrasound Images “, Presented at the IEEE International Conference on Robotics and Automation (ICRA), May 31 – June 7, 2016, pp. 4386-4391, Stockholm, Sweden. doi:10.1109/ICRA.2016.7487637 [Abstract]
These videos show needle tip tracking using the sum of squared differences (SSD) and normalized cross correlation (NCC) based visual tracking methods in deformable prostate phantom. The processed video is available on YouTube* and it was affected by the H264 coding artifacts. Results show that needle tip is successfully tracked using SSD and NCC based visual tracking inside the deformable prostate phantom and SSD or NCC based visual tracking can be used to track the needle tip in challenging conditions.
M. Kaya, E. Senel, A. Ahmad, and O. Bebek, “Visual Tracking of Multiple Moving Targets in 2D Ultrasound Guided Robotic Percutaneous Interventions“, Accepted to the IEEE International Conference on Robotics and Automation (ICRA), May 29 – June 3, 2017, Singapour.
This clip appeared in the IEEE International Conference on Robotics and Automation (ICRA) trailer in episode 4 titled “mini/micro robots” (http://trailer.icra2016.org/). This clip shows Ozyegin Biopsy Robot (OBR) which is developed to perform biopsies.
The needle is inserted using OBR and it is controlled using feedback from the US images. Also, 2D US probe was attached to a 6-DOF robotic arm and it is visually served along the needle path to keep the needle tip at the center of the image plane.
M. Kaya, E. Senel, A. Ahmad, and O. Bebek, “Visual Tracking of Biopsy Needles in 2D Ultrasound Images “, Presented at the IEEE International Conference on Robotics and Automation (ICRA), May 31 – June 7, 2016, pp. 4386-4391, Stockholm, Sweden. doi:10.1109/ICRA.2016.7487637 [Abstract]
This video shows needle tip tracking in 2D transverse US images. As mentioned in the related articles, visual tracking is independent of the shape of the needle. After the needle tip is localized once, only motion information between consecutive frames is considered for tracking. Results show that proposed visual tracking method can be used to track the needle tip in both lateral and transverse US images.
M. Kaya, E. Senel, A. Ahmad, and O. Bebek, “Visual Tracking of Biopsy Needles in 2D Ultrasound Images “, Presented at the IEEE International Conference on Robotics and Automation (ICRA), May 31 – June 7, 2016, pp. 4386-4391, Stockholm, Sweden. doi:10.1109/ICRA.2016.7487637 [Abstract]
The video shows three different experimental results.
• In the first experiment, the US probe was attached to 6-DOF KUKA robot and the needle is inserted into the water medium using a 5-DOF needle insertion robot (OBR). Needle insertion is shown in the left. Also, the needle is simultaneously tracked using SSD based visual tracking method in 2D lateral US images to provide position feedback to OBR. The result of tracking is shown on the right.
• In the second experiment, needle loss detection and recovery using NCC are shown. The needle is inserted into agar phantom. During the insertion, misalignment between US probe and the needle occurs. Misalignment is detected using NCC. After needle and US probe alignment, the needle tip is recovered without using a needle localization algorithm and visual tracking continues.
• In the third experiment, the needle tip is visually tracked using SSD based visual tracking in 2D transverse US images.
M. Kaya, E. Senel, A. Ahmad, and O. Bebek, “Visual Tracking of Biopsy Needles in 2D Ultrasound Images “, Presented at the IEEE International Conference on Robotics and Automation (ICRA), May 31 – June 7, 2016, pp. 4386-4391, Stockholm, Sweden. doi:10.1109/ICRA.2016.7487637 [Abstract]
This video shows the results of successful needle tip tracking using visual tracking in ex-vivo chicken meat. The needle tip was localized in the first frame by the user and then the tip was tracked by calculating motion information between consecutive frames.
This video shows motion analysis of deformable prostate phantom while needle is inserted. In order to analyze motion, Lukas-Kanade optical flow method was used. This video also shows that while the needle is inserting, prostate phantom moves due to stick-slip forces. Thus, needle tip and deformable tissue must be tracked simultaneously for successful biopsy. Note, the raw video was downloaded from YouTube*.
This video shows needle tip tracking using the sum of squared differences (SSD) in a deformable prostate video in distilled water. Note, the raw video was downloaded from YouTube*.
This video shows that deformable tissue tracking using thin plate spline motion model while needle is inserting into prostate phantom. Four control points were selected and 2D 15×15 nodes are created to show the deformation. Control points are shown with ‘o’ and ‘+’ indicates nodes. Note, the raw video was downloaded from YouTube*.
This video shows that deformable tissue tracking using thin plate spline motion while the needle is inserted into a prostate phantom. Four multiple key points were selected to track the deformable prostate phantom. Note, the raw video was downloaded from YouTube*.