Iterative numerical optimization is a ubiquitous tool to design optical nanostructures. However, there can be a significant performance gap between the numerically simulated results, with pristine shapes, and the experimentally measured values, with deformed profiles. We introduce conditional generative adversarial networks (CGAN) into the standard iterative optimization loop to learn process-structure relationships and produce realistic simulation designs based on the fabrication conditions. This ensures that the process-structure mapping is accurate for the specific available equipment and moves the optimization space from the structural parameters (e.g. width, height, and period) to process parameters (e.g. deposition rate and annealing time). We demonstrate this model agnostic optimization platform on the design of a red, green, and blue color filter based on metallic gratings. The generative network can learn complex M-to-N nonlinear process-structure relations, thereby generating simulation profiles similar to the training data over a wide range of fabrication conditions. The CGAN-based optimization resulted in fabrication parameters leading to a realistic design with a higher figure of merit than a standard optimization using pristine structures. This data-driven approach can expedite the design process both by limiting the design search space to a fabrication-accurate subspace and by returning the optimal process parameters automatically upon obtaining the optimal structure design.