Surveillance system has been on the increase in recent years due to cheapening of cameras and due to increase in global hostile activities such as terrorist attacks. Multi-view surveillance system has been on the increase due to cheapening of cameras and due to increase in hostile activities such as terrorist attack. Although there has been much research into multi-view surveillance in itself, there hasn’t been much attention to summarizing these data which could assist security personnel by providing frames of security interest such as certain gestures, action or event. A summarizing system for multi-view surveillance environment is implemented. Multi-view gives rise to problem of labeling labels across camera. Labeling for single camera was achieved by utilizing motion information of detected objects. And color information was used to consistently label an object across camera with an assumption that color information of target object was held beforehand. Summarizing using color information over captured dataset over five cameras proved to be successful. The dataset was of challenging one with varying luminance across cameras which made segmentation difficult and consequently labeling.