Structures with enhanced damping capabilities are of great interest to various engineering applications. The structures are commonly excited by cyclic or general time-dependent loading. In these applications, viscoelastically damped structure are often used to attenuate undesired responses. Most viscoelastic materials exhibit low stiffness for practical applications. Therefore, typical damped structures consist of viscoelastic material and structural, material to have both high damping and desired stiffness. In this paper, a multi-material topology optimization is presented to develop structures involving multiple materials. The density approach is applied to obtain the distribution of viscoelastic and structural material simultaneously. The damping is characterized by viscoelastic dissipation energy of the structures under general time-dependent loading. The amount of dissipated energy is computed by time-domain finite element analysis. Analytic design sensitivity is implemented based on the time-dependent adjoint method. The objective function of the topology optimization problem is to maximize the dissipation energy. The design constraints include the solid phase volume fraction and a local displacement restriction as a stiffness constraint. The proposed method is demonstrated on several examples under different loadings and stiffness constraints.