It's an acquired taste.
Some people picked it up when they first saw the First Doctor back in 1963. That wasn't me.
I first saw the Fourth Doctor around 1979 or so and said "who the hell is this guy running around in a scarf while accompanied by a woman in a weird hat"? That was Tom Baker. I saw it on PBS, which was carrying episodes from the BBC.
I stopped watching some time in the Peter Davison era - it got yanked off the air sometime around 1989 - I think by that point PBS wasn't carrying episodes over the pond anymore from the BBC anyway.
And then all of a sudden one day it's revived in 2005, and it's back on the Sci-Fi channel. For a while Sci-Fi was carrying episodes from BBC, and then it stopped. It's a good thing I switched cable systems to get BBC America, now I'm all caught up with the Eleventh Doctor.
Yes, I admit, it's hard to fathom that this show has been running almost continuously (with a long hiatus of 1989-2005) since 1963, with eleven different people playing the same guy. (And unlike soap operas, they have a canonical reason for why he keeps changing faces called "regeneration.")
Hard to explain if you like it why you do - but I do. Yeah, a blue 1960s style police box is the most bizarre of time travel devices. It might not have stood out in 1963 London, but now it does in 2012. Of course the box (the TARDIS) is bigger on the inside than the outside.
The longest running mystery in the program is the program's title. The Doctor has another name which no one knows (which is why they call him Doctor. Doctor ... Who?) Apparently, it seems, this is the oldest mystery in the universe. (He is, after all, a time traveller.) And apparently at some point he's going to be forced to reveal it.
For fans of the show, this is, of course, like crack, and the show producers damn well know it.
You'll probably recognize the show's most notable race of villains - the Daleks. The Doctor has faced them in every incarnation. During the Time War, he even wiped out most of his own race, the Time Lords, in defeating them.

Yeah, I know. What the hell's so fearsome about a roving salt shaker with an eye prong sticking out?
Maybe it's the way they say the word "EX...TER....MINATE!"
This is probably a design some cheap ass props department came up with in 1963 ... and like so many other things, they haven't ever changed them in the show since.