Sidewalk Following Robot

This project is a robot which, for extended distances, can follow and map sidewalk networks in a global, UTM coordinate frame while avoiding bikes, pedestrians and small ground vehicles.

Our platform is a Clearpath Jackal, again, utilizing a Nvidia Xavier for perception, 2x Intel Realsense D435 which are frame synced and a Intel NUC for control and planning.

We use Deeplab v3 with a Mobile Net backbone trained on Cityscapes for real time segmentation (25-30fps) running with 16bit floating point mode using TensorRT while doing RGB-D ORB_SLAM2 or DSO depending on the sequence for mapping and pose estimates. There is a strong prior on being on a road from the dataset, so we have to use both road and sidewalk masks to correctly segment the sidewalk (plus CU Boulder sidewalks are very strange compared to the average sidewalk/road). We then do morphological closing and erosion on the resulting segmentation of the sidewalk while dropping any of the mask that is too far away on the horizon to be relevant in the near term feedback controller. We then take a centroid of the resulting sidewalk mask, which we feed into a Kalman filter to smooth the output of the segmentation and deal with frames where we detect no segmentations or just very noisy segmentations. All of this is needed because the network is extremely unreliable on campus because of its expectation of a scene to be viewed from a road and a higher vantage point, neither of which we are doing. From here, we do a simple feedback controller on the heading provided by filtering over our SLAM estimate and our INS fix to change heading to center the centroid in the image to effectively follow the “gap” of the high intensity region of the segmentation surrounded by nontraversable areas. This ends up being useful because humans, bikes and cars are also not traversable, and thus, the robot will actively avoid these other objects as a side effect of this design.

This functions quite well given optimal lighting conditions, however it struggles to segment all traversable area effectively when long shadows (common in the winter in afternoon in Colorado) are drawn across the sidewalk, causing it to incorrectly change direction and sometimes lose the sidewalk completely. It also struggles with sun glare which is not present in the dataset, causing it to lose its segmentation, and further its heading. The kalman filter carries it through some of these, but it can only do so well. This approach also struggles with junctions, because it assumes this is just a very large sidewalk, to explore these branching paths, we would have to implement a control strategy using our map.

This project is for CU Boulders CSCI 7000: Autonomous Vehicle Challange with Professor Heckman in conjunction with the ARPG Lab.