How to Deselect a few things from a Selection

How to deselect a few items from a selection? For example in the attached picture below I have selected a few items and then I want to deselect a few of them. When I try to deselect them using dragging mouse pointer by pressing the CTRL key the desired things get deselected but other components get selected. However, I just only exclude a few things from my selection without selecting additional things.

SolidworksSketching