In this paper, we propose an all-in-one mobile outdoor augmented reality (AR) framework for a cultural heritage site. The framework was designed to incorporate computer vision-based augmented reality technology and ontology-based data-authoring technology. Through this framework, we clearly explain how to create 3D visual data for camera pose estimation and how to connect AR content with a cultural heritage site. In addition, we suggest a multi-threading camera tracking and offer an estimation model for mobile AR application. Finally, we have confirmed the efficiency and reliability of our framework. Through this vision-based AR framework, seamless AR application for a cultural heritage site can be made.