Accurate recognition of emotions has many applications, but it is challenged by difficulties in collecting emotions in the wild. While naturally occurring emotions are expensive to collect, the inherent bias in their distribution further confounds the issue. The random sampling method frequently employed by researchers fails to overcome these limitations. We propose an adaptive sampling method based on active learning as an alternative to collect emotions with balanced distribution while reducing burdens on users. The effectiveness of adaptive sampling is empirically evaluated with the K-EmoCon, the dataset of continuous emotions and physiological signals collected in the context of naturalistic conversations. The tradeoff between collecting balanced data and querying informative samples is also explored with a parameterized query strategy.