go to Dig it Records, 3 rue de la Servette. Behind the Station. The biggest choice in second hand (all styles); also Sounds, 8 rue de l'Ecole de Médecine (plaine de Plainpalais); a lot of new stuff in vinyl, also some second hands. Try also Bongo Joe, 9 place des Augustins. New and second hand.
Dig It Records (music shop, concert), Sounds (music shop) and Bongo Joe (music shop, label and concert/bar). It's also worth checking at Urgence Disk Records (music shop, label and concert/bar at the bottom floor of L'Usine, Place des Volontaires 4), as well as the flea market each Wednesday and Saturday, and every first Sunday of the month (Plaine de Plainpalais), enjoy!