How good do I actually have to be to get into a top graduate program from an average undergraduate program?

So, this is a question that I do not know much about. I might get called on not having done a ton of research on graduate school, but I am really curious about this. I am an undergraduate student of mathematics and I am certainly planning on pursuing a graduate degree. I have looked at the top schools of math and they are certainly what one would expect: MIT, Harvard, Princeton, Yale, etc.

My issue is this: Academically, I was a late bloomer. I did not take high school seriously because it was annoying and elementary. Where I grew up, my school did not offer much in terms of AP classes or other opportunities. So, the only thing I excelled at naturally was math classes. Of course, they were extremely easy and required almost no effort on my part, which is what made me leery in terms of taking college level classes. I figured that I was good at high school classes because they are simple, but college level will be a different story.

I was wrong. In university, the gap was wider. Once I became adjusted, I actually enjoyed mathematics because it was so much more abstract and complex. This caused me to do rather large amounts of independent research in graduate level topics (topology, analytic number theory, analysis, etc). My only issue is that I am not in a competitive environment, really. The students are not top-notch, and don't share the same drive that I have. This is why I feel like being at the top of the class is not really an accomplishment, since I am not doing anything that any undergraduate at MIT, Harvard, Yale, etc, couldn't do.

My question that I pose here is this:

How good do I really need to be?

I have a passion for mathematics and for the most part, I seem to be talented at it. In terms of GPA, I have maintained a 4.0 with little effort just because I typically have already covered the material in the class on my own. However, there is a voice in the back of my head that keeps telling me that I only stand out because I am at a school that is not top tier, ivy league.

What does a graduate student at a top tier school look like? In my mind, I see someone that has been publishing papers since their teens, and was at the level I'm at now coming out of high school (I am a rising senior at the moment). Is this true? Am I being realistic? I am concerned because I would love to apply to some of these schools (and to get in is my dream!) but I honestly don't even know what level I am at in comparison to the rest of their applicants. I would hate to get shot down almost instantly because I am a joke in their eyes.

Sorry if this was a bit of a ramble. The question boils down to the title really, the rest is just elaboration for those that are interested. Thanks!