I'm sure it's a combination of both, but who knows what the ratio is.
I read toprated weekly or so to see what's new, and I give each comic a fair shake. About every three months, I go through every page and give each comic a rating. Over time, I'm sure my opinion of what's good/okay/bad has changed, but in general, I don't base my rating decisions on "is this the best comic on the site and therefore deserving of appearing on the first toprated page?" If I think it's good, I'll rate it as such and I don't really care if a comic I think isn't that great is rated higher than one I do or one of mine--humor is all relative. Also, in those instances, I'm certain someone else will downvote it, so I may as well vote it good if I think it's better than okay.
I rarely vote comics bad that I discover first on toprated, but it happens. I have a pretty narrow definition for what I think is bad in the toprated arena, so I probably give more okays to comics that I think aren't good. I think in the first ten or so pages, I average 1-5 okay ratings per page.
Here's what I think when one of my comics appears at the top: "Cool, four people who like my comics liked this one enough to rate it." Then I know that people who don't think I'm the funniest human on the planet will start voting it down. It's okay--I realize I'm not for everyone. Dcom has told me several times that I keep telling the same three jokes over and over and over. Anyway, that's more or less what's happening to other users--their fans put them at the top, then the people who think the user isn't that great knock it down either for pure or impure reasons.