Independence of people in need in their living environment becomes an imperative notion for any society in the years to come. As an approach to achieve “independence,” this paper addresses the problem of building an assistive home environment emphasizing a human-friendly man-machine interaction. From the functional point of view, a house is considered as a large-scale main (parent) robot which is capable of interacting with residents to provide comfortable living conditions such as appropriate humidity, temperature, lighting, safety and security. Also, under the management of the parent robot, the home system includes several subsystem (children) robots playing their inherent roles to provide the residents with independence and happiness. The way of interaction between the resident and the systems is thus a significant aspect in the design of a smart home for the elderly and people with disability. As an engineering approach, we propose to design based on the idea that the robotic systems should be as human-friendly as possible, and the acquired information in interaction, which includes physical, emotional, mental status and intention of the user, should be incorporated into the system to enhance human-friendliness. After briefly reviewing various aspects of the notion of the human-robot interaction, this paper introduces our experiences of realizing techniques used for human-friendly interaction.