Mrs Filmers and the Smiths

Something has been bothering me about this book. In the scene where Rezia is marking a hat for Mrs Peters, are the Smiths in Mrs Filmers house or at theirs? I thought they were at their place but then septimus realizes he’s alone at “Mrs Filmers sitting room sofa” and he sees her Bread knife.