Computer Scientists Build ‘Pedestrian Remover’

The dog stands alone (bottom image) after a UC San Diego pedestrian remover automatically removed the man walking the dog (top image) and filled in the hole with building, grass, curb, and sidewalk.(Source: University of California, San Diego)

The new system, presented at the IEEE International Workshop on Mobile Vision, removes pedestrians and replaces the holes in the images with an approximation of the actual background behind each pedestrian. Scientists pull corresponding background pixels from the image taken right before or right after the image in question. Its inventor is Arturo Flores, a computer science graduate student from the University of California, San Diego. He developed this plan during a project-based computer vision and machine learning class taught by Professor Serge Belongie.

The system is still in its preliminary stages, and it wasn’t even named. Based on Google Street View, a technology featured in Google Maps and Google Earth that provides panoramic views from various positions along many streets in the world. According to Google, they create the views by stitching together overlapping images taken from a moving vehicle.

Recently, Google Street View received some public criticism due to privacy issues; the photos used include many times license plates, personal identifying marks, and even people’s faces. This new technology might partially resolve this issue.

Although Google Street View adds a blurring effect to faces and license plates from its images, other personal elements can be recognizable. In their paper, Flores and Belongie claim that such elements combined with revealing information such as exact geographical location make some pedestrians feel personally identifiable.

The results of the system are surprisingly clean, being relatively “ghost free”, which means that the artifacts caused by the pixel swapping are usually not distracting. However, the pedestrian remover does occasionally produce strange results – like dogs on leashes with no owners, and shoes with feet but nothing else.

Other funny occurrences happen when pedestrians walk in the same direction as a vehicle moving at just the right speed. Under these circumstances, the system struggles to generate background pixels; in

When the automatic pedestrian remover replaced the woman (top image), the umbrella remained (bottom image). (Source: University of California, San Diego)

these cases, the pedestrian may cover up the same spot in multiple frames, foiling the computer

scientists’ pixel-swapping approach to removing pedestrians. Moreover, the pedestrian remover only works in urban settings – where the pixels blocked by people are often “on a dominant planar surface” – which make them simpler to replace.

For example, the system can replace the pixels blocked by a person walking by a mural of horses grazing in a pasture, but it cannot replace the pixels behind a person on a country road walking by actual horses grazing in a pasture, because this background is not predominately flat. Hopefully, later versions will not have these limits.

It is not coincidental that Flores chose a project basing on the freely available Google service; Belongie says he encourages his students to make such projects, since they can tap freely available tools and datasets, and leverage existing software. “This is a cute idea that, and as far as we know, has not been explored,” he said. Flores adds that they are already designing the next step, which will enable the system to remove groups of pedestrians from single images.

TFOT has also covered the European CLASS Project, which tries to teach computers to visually recognize specific objects – an action that might be trivial for humans, but extremely difficult for machines.

For more information about the pedestrian remover, see University of California’s press release.