Dani - I keep mine organized just in folders on my hard drive - mostly by artist if I have a lot, like my favorite bands, Barenaked Ladies, U2, etc. I have a folder for them and then in that folder a folder for each album. Those that I only have a few of I organize by Classical, 80's, etc.
If you want a program to help you get organized, my favorite is iTunes. It's a Mac program but also has a Windows version. With it you can rename your files, have it organize them into folders, rip MP3s off your existing CDs, etc.
I've just steadily gone through my mp3 collection and named them like this:
Band (with no 'the') space hyphen space song name
That way they at least get organized by band name even if I don't get them separated into folders:
Corey Hart - I Wear my Sunglasses at Night
Tears for Fears - Everybody wants to Rule the World
REO Speedwagon - Can't Fight This Feeling
Etc
Hope that helps!