okay, the reason they're square is because the line style uses tiny little squares to make the dotted line. Choose the line you want (dotted) and then at the bottom of the menu choose Custom. Change the segment from the square to the ball bearing and you'll get it to be round. It looked dotted until you make it big, where it's apparent that it's in fact just squares.
Unfortunately, I don't see any way to get rid of the line connecting the two. I made each end cap as close as possible, and made one big and the other tiny. So it has the dot, but a tiny post sticking out of it like a lollipop...
hth! And if anyone knows a way to get rid of the connecting lines between caps, please tell me how you did it!