This study explores the contextual characteristics of the city and the visual perception of human beings, and aims to identify how people see their environment and how to give order between them. This study starts with an introduction of a novel method that extracts complex spatial information of the surrounding environment from an observer through a new spatial representation model, Visible-Space Envelope (VSE). The Visible-Space Envelope (VSE) makes a physical shape of a space visually perceptible for an observer as a three-dimensional spherical physical object. This study shows that the three-dimensional composition of objects in the space where the observer is located can be quantitatively measured by spherical analysis of VSE. Furthermore, this study classifies physical contexts of the city based on the morphological information by tomographic analysis of VSE and the location information of observer’s movement. The method of this study has become possible thanks to mobile sensor data, which is a point cloud dataset including 3D positions of various objects in the real world and location information of observer. What is expected in this study is that it will be possible to quantitatively classify physical contexts of a city in which various elements are mixed. Ultimately, the approach presented here suggests useful evaluation indices for attractive urban design.