Try holding down the shift key while clicking on the different layers. Or....if you don't mind merging the layers you can do that too.