Abstract: Pedestrian detection and tracking is one of the basic functions of eldercare robot. Aiming at the problem of manual calibration of the initial frame of the image, poor target detection accuracy and difficult to retrack the lost pedestrian, a tracking algorithm based on pedestrian detection and Kernelized correlation filter was proposed. The environment visual information was obtained by the Kinect2 camera. The pedestrian gradient was extracted by the histogram of oriented gradient, and support vector machine was used to train the model to automatically detect pedestrians. The KCF algorithm was used to track pedestrians, and the world coordinates were calculated based on the depth data to drive robot to realize stable tracking of pedestrians. The software system was built under Ubuntu16.04 system and ROS platform, and tested on the Turtlebot mobile platform. The results show that the system can automatically detect pedestrians and track them stably in complex environment, which can effectively improve the mobile and sensing performance of the robot.