Recently, the annihilating filter-based low-rank Hankel matrix (ALOHA) approach was proposed as a powerful image inpainting method. Based on the observation that smoothness or textures within an image patch correspond to sparse spectral components in the frequency domain, ALOHA exploits the existence of annihilating filters and the associated rank-deficient Hankel matrices in an image domain to estimate any missing pixels. By extending this idea, we propose a novel impulse-noise removal algorithm that uses the sparse and low-rank decomposition of a Hankel structured matrix. This method, referred to as the robust ALOHA, is based on the observation that an image corrupted with the impulse noise has intact pixels; consequently, the impulse noise can be modeled as sparse components, whereas the underlying image can still be modeled using a low-rank Hankel structured matrix. To solve the sparse and low-rank matrix decomposition problem, we propose an alternating direction method of multiplier approach, with initial factorized matrices coming from a low-rank matrix-fitting algorithm. To adapt local image statistics that have distinct spectral distributions, the robust ALOHA is applied in a patch-by-patch manner. Experimental results from impulse noise for both single-channel and multichannel color images demonstrate that the robust ALOHA is superior to existing approaches, especially during the reconstruction of complex texture patterns.