Sportswear might refer to a specific sport (like rugby kits or swimwear) or might refer to athleticwear in a broader sense (like Lycra/spandex or long socks). The spectrum of fetishwear is so vast that it regularly crosses over into other genre (rubber wrestling singlets are a good example).

 

Do you have to play a sport in it? No. But you should be able to see the sports reference in whatever it is you're wearing.