An image optimized rolling cache system extracts pixel information and address information of a corresponding pixel from an input image to store the extracted pixel information and the extracted address information, and processes the image by applying a vertical rolling mechanism or a horizontal rolling mechanism using the stored information.