In this paper, a connectionist model to integrate knowledge-based techniques into neural network approaches for visual pattern classification is presented. We propose a new structure of connectionist model which has rule-following capability as well as instance-based learning capability. Each node of the proposed network is doubly linked by two types of connections: positive connection and negative connection. Such connectionism provides a methodology to construct the classifier from the rule base and allows the expert knowledge to be utilized for the effective learning. For visual pattern classification, we present the techniques for knowledge representation and utilization using the concepts of fuzzy rules and fuzzy relations. We also discuss in this paper some advantageous characteristics of the model: result explanation capability and rule refinement capability. From the experimental results of the handwritten digit classification, the feasibility of the proposed model is evaluated.