This study investigated the effect of the particle size of silica colloidal suspension (SCS) in SCS-impregnated plain woven fabrics on the ballistic performance of those fabrics. In order to examine the particle size effect, spherical silica particles with average diameters of 100 nm, 300 nm, and 500 nm were used to fabricate SCS. The SCS-impregnated fabrics were subjected to ballistic tests under various boundary conditions. The fabric impregnated with SCS produced using silica particles with an average particle diameter of 100 nm showed better impact performance than those of the SCS-impregnated fabrics containing larger particles and untreated fabrics in terms of impact energy absorption and resistance to blunt trauma, while earlier failure of their primary yarns in the impacted zone under fully clamped conditions was also observed. The results of the drop impact test and yarn pull-out tests indicated that SCS-impregnated fabric with smaller particles exhibits the largest increment of inter-yarn friction at the onset shear strain for shear thickening. It was found that impregnation with SCS affected the interfacial friction of filaments and yarns, which contributes to a larger transfer of the impact energy to interaction of individual yarns in the fabric, resulting in significantly higher energy dissipation with less transverse deformation. Further, the influence of impregnation with SCS on ballistic performance was found to be closely associated with boundary conditions. Thus, the particle size of the SCS and the boundary conditions are the dominant factors that can be manipulated to fully utilize the benefits of SCS impregnated fabrics for flexible body armor.