The human faculty of distinguishing thousands of faces critically contributes to face identification and our social interactions. While prior studies have revealed the involvement of the fusiform face area (FFA) in the individuation processing of faces, there are also reports supporting that the responses of the FFA is flexible depending on tasks. Here, we investigated whether the specificity of neural responses in the FFA for individual faces depends on the need for individuation using functional magnetic resonance imaging (fMRI). We found that individual face images could be decoded from response patterns of the FFA when individuation was required for the task but not when only categorization according to a common feature such as race or gender was necessary. These results suggest that the specificity of neural responses for individual faces is flexible in the FFA, depending on the behavioral goal of face individuation.