Remember In Order to Forget