Speaker switch that daisy chains speakers, to avoid low impedance

At work, we have a stereo and four 4 ohm speakers, and a sub woofer. The amplifier only has one set of speaker outputs and is rated for outputting at 4 ohms. How do we go about connecting the speakers without getting to low an impedance? The speaker switch boxes at the local shop all seem to expand the speaker connections by putting them in parallel, which would bring the load the amplifier sees down to 2 ohms, nominally.

The sub woofer has connectors both for line in levels (250 mV or 775 mV I presume) and for speaker cable. I assume the speaker cable input is high impedance (kilo ohms or higher).

I thought about what the schematics could be for a simple speakers switch box, and below is what I came up with. It is untested: