Devising colloidal nanoparticle assemblies with finely tuned topological parameters is critical to the development of efficient and reliable plasmonic platforms that can enable promising applications, such as surface-enhanced Raman scattering (SERS). Here, we report a facile synthesis strategy for the preparation of stable colloidal clusters of Au nanoparticles (Au NPCs) with well-controlled structural parameters, including the average number and size of constituent nanoparticles and the size of interparticle gaps, in which the galvanic replacement of Ag nanoprisms with controlled amount of Au precursors yielded Au NPCs with maneuvered particle sizes, while the other structural factors were intact. The present approach could allow the precise exploration of the influence of particle size on the SERS activity of nanoparticle assemblies. Notably, the prepared Au NPCs showed different particle-size dependency of their SERS activity along with the change in analyte concentration. Finite difference time domain simulation studies revealed that the experimental results can be correlated with the relative contributions of the magnitude of near-field enhancement and areal density of hot spots in the Au NPCs, which are determined by the size of constituent nanoparticles. This study therefore provides key design guidelines to optimize the plasmonic function of nanostructure assemblies.