On a week's holiday in Reykjavik last November, I went swimming every morning outside(!) in the thermal pool at Vesturbaejarlaug.
The pool is in a suburb of Reykjavik and filled with all the local grannies gossiping in the (very) hot tubs and school children doing lengths. There is a sauna and steam room and it's a fabulous way to mix with the locals and get a real taste of Reykjavik life. So refreshing. It's great to walk around outside in just your swimming costume, you don't feel cold, but if you do, the hot tub will cook you up like a lobster.
Hofsvallagata, 107 Reykjavik
+354 411 5150
www.spacity.is/index.php?lang=en
Google map: bit.ly/hj16Fw
Open Mon-Fri 06.30—22.00
Sat-Sun 08.00—22.00
Entry 360ISK adult, 110ISK for 6-18 years, free for seniors and under 6
Sauna 450ISK extra
Entry, towel, swimming costume 850ISK
Entry, towel, swimming costume, sauna 950ISK