With the world driving towards automation and technological advancements, the utilisation of service robots for smart home environments is gaining limelight. However, to be able to perform efficiently in a human-like manner, robots need to integrate tasks like object detection and localization with semantic knowledge representation and activity recognition. Therefore, researchers are increasingly working towards mapping high-level knowledge of the world to the robot’s understanding to produce the desired outcome. The goal is to make the robot capable of making intelligent decisions and carry out its job without human intervention as far as possible. Our work involves making a robot predict activities of daily living of a person living in a smart home environment. Based on the routine of the person, the predictions made by the robot can be useful in making viable choices and adhere to his/her needs. We train an ensemble deep learning architecture to ascertain the need for some robot action and further determine the type of action required. To the best of our knowledge, predicting robot actions based on the semantic nature of the environment using Deep Learning has not been carried out before.