Wind energy is considered as one of the most promising renewable energy sources due to its abundance and the high energy conversion efficiency of a wind turbine. Because wind flow is complex and highly variable, accurately predicting wind turbine responses such as power and loads speed is challenging. These responses are generally computed by computational wind turbine analysis tools that uses a wind flow as an input. In this study, we propose a machine learning approach to predict wind turbine responses using wind flow data as a direct input. Specifically, we use Multi-Tasks Convolutional Long Short-Term Memory Network to simultaneously predict the energy output and structural load from the target wind turbine while modeling the spatio-temporal structure of the input wind flow. The simulation experiment shows that the proposed model predicts the both outputs within a 5% error.