Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective