Intelligent Vehicles that Can See


Autonomous Driving based on AI Deep Learning of Road Profiles from Driving Videos

Driving video is available from in-car camera for road detection and collision avoidance. However, consecutive video frames in a large volume have redundant scene coverage during vehicle motion, which hampers real-time perception in autonomous driving. This work utilizes compact road profiles (RP) and motion profiles (MP) to identify path regions and dynamic objects, which drastically reduces video data to a lower dimension and increases sensing rate. To avoid collision in a close range and navigate a vehicle in middle and far ranges, several RP/MPs are scanned continuously from different depths for vehicle path planning. We train deep network to implement semantic segmentation of RP in the spatial-temporal domain, in which we further propose a temporally shifting memory for online testing. It sequentially segments every incoming line without latency by referring to a temporal window. In streaming-mode, our method generates real-time output of road, roadsides, vehicles, pedestrians, etc. at discrete depths for path planning and speed control. We have experimented our method on naturalistic driving videos under various weather and illumination conditions. It reached the highest efficiency with the least amount of data.


G. Cheng, J. Y. Zheng, “Sequential Semantic Segmentation of Road Profiles for Path and Speed Planning”, IEEE Transaction on Intelligent Transportation Systems, pp. 1-14, 2022. (Video)

G Cheng, JY Zheng, M Kilicarslan, Semantic segmentation of road profiles for efficient sensing in autonomous driving, 2019 IEEE Intelligent Vehicles Symposium (IV), 564-569.



Driving Video Profile (DVP) dataset:


Data Mining of Naturalistic Driving Video (partly supported by Dept. of Transportation, and NIST.DOC)

Nowadays, many vehicles are equipped with a vehicle borne camera system for monitoring drivers’ behavior, accident investigation, road environment assessment, and vehicle safety design. Huge amount of video data is being recorded daily. Analyzing and interpreting these data in an efficient way has become a non-trivial task. As an index of video for quick browsing, this work maps the video into a temporal image of reduced dimension with as much intrinsic information as possible observed on the road. The perspective projection video is converted to a top-view temporal profile that has precise time, motion, and event information during the vehicle driving. Then, we attempt to interpret dynamic events and environment around the vehicle in such a continuous and compact temporal profile. The reduced dimension of the temporal profile allows us to browse the video intuitively and efficiently.


Z. Wang, J. Y. Zheng, Z. Gao, Detecting Vehicle Interactions in Driving Videos via Motion Profiles, IEEE Intelligent Transportation System Conference, 2020, 1-6.

K. Kolcheck, Z. Wang, H. Xu, J. Y. Zheng, Visual Counting of Traffic Flow from a Car via Vehicle Detection and Motion Analysis, Asian Conference on Pattern Recognition, 2019, 529-543. (Best Paper Award on Road and Safety).

G. Cheng, J. Y. Zheng, H. Murase, Sparse Coding of weather and illuminations for ADAS and autonomous driving, IEEE Intelligent Vehicles, 2018, 1-6.

Guo Cheng, Z. Wang, J. Y. Zheng, Modeling weather and illuminations in driving views based on big-video mining, IEEE Trans. Intelligent Vehicles, 3(4), 522-533, 2018.

Mehmet Kilicarslan, Jiang Yu Zheng, Visualizing Driving Video in Temporal Profile, IEEE Intelligent Vehicle Symposim, 2014, 1-7.


J.Y. Zheng, IUPUI Driving Videos and Images in All Weather and Illumination Conditions, CDVL Tech Memo,


Pedestrian Detection from Motion in Driving Video

Pedestrian detection has been intensively studied based on appearances for driving safety. Only a few works have explored between-frame optical flow as one of features for human classification. In this paper, however, a new point of view is taken to watch a longer period for non-smooth movement. We explore the pedestrian detection purely based on motion, which is common and intrinsic for all pedestrians regardless of their shape, color, background, etc. We found unique motion characteristics of humans different from rigid object motion caused by vehicle motion.


M. Kilicarslan and J. Y. Zheng, "DeepStep: Direct Detection of Walking Pedestrian From Motion by a Vehicle Camera," in IEEE Transactions on Intelligent Vehicles, 2022, doi: 10.1109/TIV.2022.3186962. (video)

G. Cheng, J. Y. Zheng, Semantic Segmentation for Pedestrian Detection from Motion in Temporal Domain, International Conference on Pattern Recognition 2020, 1-7. (Video)


M. D. Sulistiyo, Y. Kawanishi, D. Deguchi, I. Ide, T. Hirayama, J. Y. Zheng, H. Murase, Attribute-Aware Loss Function for Accurate Semantic Segmentation Considering the Pedestrian Orientations, IEICE Trans. on Fundamentals of Electronics Communications and Computer Sciences E103.A(1): 231-242, 2020.


M. D. Sulpizio, Y. Kawanishi, D. Deguchi, T. Hirayama, I. Ide, J. Y. Zheng, H. Murase, Attribute-aware semantic segmentation of road scenes for understanding pedestrian orientations, IEEE Intelligent Transportation Systems Conference, 2018, 1-6.

M. Kilicarslan, J. Y. Zheng, K. Raptis, Pedestrian detection from motion, 23th International Conference on Pattern Recognition, 1857-1863, 2016. (video)

Mehmet Kilicarslan, Jiang Yu Zheng, Aied Algarni, Pedestrian detection from non-smooth motion, IEEE Intelligent Vehicles Symposium (IV), 2015, pp. 487-492

Mehmet Kilicarslan, Jiang Yu Zheng, Detecting walking pedestrians from leg motion in driving video, 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), pp. 2924-2929


 Bicyclists Detection in Driving Video (supported by Toyota)

Monocular camera based bicyclist detection in naturalistic driving video is a very challenging problem due to the high variance of the bicyclist appearance and complex background of naturalistic driving environment. In this paper, we propose a two-stage multi-modal bicyclist detection scheme to efficiently detect bicyclists with varied poses for further behavior analysis. A new motion based region of interest (ROI) detection is first applied to the entire video to refine the region for sliding-window detection. Then an efficient integral feature based detector is applied to quickly filter out the negative windows. Finally, the remaining candidate windows are encoded and tested by three pre-learned pose-specific detectors. The experimental results on our TASI 110 car naturalistic driving dataset show the effectiveness and efficiency of the proposed method. The proposed method outperforms the traditional methods.


C. Liu, R. Fujishiro, L. Christopher, J. Y. Zheng, Vehicle-bicyclist dynamic position extracted from naturalistic driving videos, IEEE Transactions on Intelligent Transportation Systems, 18(4), 734-742, 2017.

Kai Yang, Chao Liu, Jiang Yu Zheng, Lauren Christopher, Yaobin Chen, Bicyclist detection in large scale naturalistic driving video, 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), 2015, pp. 1638-1643.


Vehicle Collision Alert based on Visual Motion

This work models various dangerous situations that may happen to a driving vehicle on road in probability, and determines how such events are mapped to the visual field of the camera. Depending on the motion flows detected in the camera, our algorithm will identify the potential dangers and compute the time to collision for alarming. The identification of dangerous events is based on the location-specific motion information modeled in the likelihood probability distributions. The originality of the proposed approach is at the location dependent motion modeling using the knowledge of road environment. This will link the detected motion to the potential danger directly for accident avoidance. The mechanism from visual motion to the dangerous events omits the complex shape recognition so that the system can response without delay.


M. Kilicarslan, J. Y. Zheng, Predict collision by TTC from motion using single camera, IEEE Trans. Intelligent Transportation Systems, 20(2), 522-533, 2019. (video)

M. Kilicarslan, J. Y. Zheng, Direct vehicle collision detection from motion in driving video, IEEE Intelligent Vehicles, 2017, 1558-1564.

M. Kilicarslan, J. Y. Zheng, Bridge motion to collision alarming using driving video, 23th International Conference on Pattern Recognition, 1870-1875, 2016.

Mehmet Kilicarslan, Jiang Yu Zheng, Modeling Potential Dangers in Car Video for Collision Alarming, IEEE International Conference on Vehicle Electronics and safety 2012, 1-6

Mehmet Kilicarslan, Jiang Yu Zheng, Towards Collision Alarming based on Visual Motion. IEEE International Conference on Intelligent Transportation Systems 2012: 1-6

A. Jazayeri, H. Cai, J. Y. Zheng, M. Tuceryan, Vehicle Detection and Tracking in Car Video Based on Motion Model, IEEE Transaction on Intelligent Transportation Systems,  12(2), 583-595, 2011.


Road Appearances and Edge Detection in All-weathers and Illuminations (partly supported by Toyota)

To avoid vehicle running off road, road edge detection is a fundamental function. Current work on road edge detection has not exhaustively tackled all weather and illumination conditions. We first sort the visual appearance of roads based on physical and optical properties under various illuminations. Then, data mining approach is applied to a large driving video set that contains the full spectrum of seasons and weathers to learn the statistical distribution of road edge appearances. The obtained parameters of road environment in color on road structure are used to classify weather in video briefly, and the corresponding algorithm and features are applied for robust road edge detection. To visualize the road appearance as well as evaluate the accuracy of detected road, a compact road profile image is generated to reduce the data to a small fraction of video. Through the exhaustive examination of all weather and illuminations, our road detection methods can locate road edges in good weather, reduce errors in dark illuminations, and report road invisibility in poor illuminations.


G. Cheng, J. Y. Zheng, M. Kilicarslan, Semantic Segmentation of road profiles for efficient sensing in autonomous driving, IEEE Intelligent Vehicle Symposium 2019, 1-6.

Z. Wang, G. Cheng, J. Y. Zheng, Road edge detection in all weather and illumination via driving video mining, IEEE Trans. Intelligent Vehicles, 4(2), 232-243, March 2019.

Guo Cheng, Z. Wang, J. Y. Zheng, Modeling weather and illuminations in driving views based on big-video mining, IEEE Trans. Intelligent Vehicles, 3(4), 522-533, 2018.

Z. Wang, G. Cheng, J. Y. Zheng, All weather road edge identification based on driving video mining. IEEE Intelligent Transportation Systems Conference, 2017.

G. Cheng, Z. Wang, J. Y. Zheng, Big-video mining of road appearances in full spectrums of weather and illuminations, IEEE Intelligent Transportation Systems Conference, 2017.


Night Road and Illuminations (Supported by Toyota)

Pedestrian Automatic Emergency Braking (PAEB) for helping avoiding/mitigating pedestrian crashes has been equipped on some passenger vehicles. Since approximately 70% pedestrian crashes occur in dark conditions, one of the important components in the PAEB evaluation is the development of standard testing at night. The test facility should include representative low-illuminance environment to enable the examination of the sensing and control functions of different PAEB systems. The goal of this research is to characterize and model light source distributions and variations in the low-illuminance environment and determine possible ways to reconstruct such an environment for PAEB evaluation. This paper describes a general method to collect light sources and illuminance information by processing large amount of potential collision locations at night from naturalistic driving video data. This study was conducted in four steps. (1) Gather night driving video collected from Transportation Active Safety Institute (TASI) 110 car naturalistic driving study, particularly emphasizing locations with potential pedestrian collision. (2) Generate temporal video profile as a compact index toward large volumes of video, (3) Identify light fixtures by removing dynamic vehicle head lighting in the profile and stamp them with their Global Positioning System (GPS) coordinates. (4) Find the average distribution and intensity of illuminants by grouping lighting component information around the potential collision locations. The resulting lighting model and setting can be used for lighting reconstruction at PAEB testing site.


Libo Dong, Stanley Chien, Jiang-Yu Zheng, Yaobin Chen, Rini Sherony, Hiroyuki Takahashi, Modeling of Low Illuminance Road Lighting Condition Using Road Temporal Profile, SAE Technical Paper, 2016, pp. 1638-1643


Railway Online

A patrol type of surveillance has been performed everywhere from police city patrol to railway inspection. Different from static cameras or sensors distributed in a space, such surveillance has its benefits of low cost, long distance, and efficiency in detecting infrequent changes. However, the challenges are how to archive daily recorded videos in the limited storage space and how to build a visual representation for quick and convenient access to the archived videos. We tackle the problems by acquiring and visualizing route panoramas of rail scenes. We analyze the relation between train motion and the video sampling and the constraints such as resolution, motion blur and stationary blur etc. to obtain a desirable panoramic image. The route panorama generated is a continuous image with complete and non-redundant scene coverage and compact data size, which can be easily streamed over the network for fast access, maneuver, and automatic retrieval in railway environment monitoring. Then, we visualize the railway scene based on the route panorama rendering for interactive navigation, inspection, and scene indexing.


S. Wang, S. Luo, Y. Huang, J. Y. Zheng, P. Dai, Q. Han, Railroad online: acquiring and visualizing route panoramas of railway scenes. The Visual Computer 30(9), 1045-1057, 2014.

S. Wang, S. Luo, Y. Huang, J.Y. Zheng, P. Dai, Q. Han, Rendering railway scenes in Cyberspace based on route panoramas, International Conference on Cyberworlds (CW), 12-19, 2013.


S. Wang, J. Y. Zheng, Y. Huang, S. Luo, Route panorama acquisition and rendering for high-speed railway monitoring, IEEE International Conference on Multimedia and Exposition, 1-6, 2013.