Open-Set Domain Adaptation (OSDA) assumes that a target domain contains unknown classes, which are not discovered in a source domain. Existing domain adversarial learning methods are not suitable for OSDA because distribution matching with unknown classes leads to negative transfer. Previous OSDA methods have focused on matching the source and the target distribution by only utilizing known classes. However, this known-only matching may fail to learn the target-unknown feature space. Therefore, we propose Unknown-Aware Domain Adversarial Learning (UADAL), which aligns the source and the target-known distribution while simultaneously segregating the target-unknown distribution in the feature alignment procedure. We provide theoretical analyses on the optimized state of the proposed unknown-aware feature alignment, so we can guarantee both alignment and segregation theoretically. Empirically, we evaluate UADAL on the benchmark datasets, which shows that UADAL outperforms other methods with better feature alignments by reporting state-of-the-art performances.