I know about the little button on the Properties Panel that switches the selection mode between Add and Replace. However, when want to remove objects from a selection set with Shift+Window Select it is always in Replace mode : what is already selected gets un-selected while the objects that are not selected get selected. Frustrating. 

Is there a way to control this behaviour?