Does rigor/thoroughness of undergraduate program matter (for graduate/PhD applications)?

I am an undergraduate at a large public research university in the US, finishing my second year. I have done some theorycrafting around my graduation course requirements, and I have realized that it is possible, with some work (and generous CLEP testing), to graduate with a double major, Math BS + Physics BA, by next May. I would be meeting the bare-minimum requirements for both majors (in particular, for math, I will have only taken the 4-semester calculus series, lin. algebra, and the 2-semester abstract algebra series; plus some extraneous ones e.g. game theory).

If I were to stay the extra year, I would comfortably qualify for a double Math BS + Physics BS, and I would have 3-4 more math classes under my belt (perhaps the 2-semester topology series, graph theory, advanced abstract algebra, real analysis, etc). (edit: this option, Physics BS rather than BA, stresses a more thorough physics courseload; 15 hours of upper level physics classes are required for BS over BA)

edit after several answers: Another 4-year option is to get Physics BA + Math BS, leaving my course load open to taking 3-4 math classes per semester. This, hypothetically, could put my post-lin-algebra courses at 2 sem. abstract algebra, 2 sem. topology, 2 sem. intro analysis, graduate level courses of the above, intro complex analysis, intro graph theory, etc

My intent, if I were to graduate next year, is probably to take a year off from academics, and then apply for graduate/PhD programs in math.

Will it be an issue in being considered for these programs, that I have only taken a meager selection of the course offerings in my undergraduate studies? And might it be offset by the fact that I graduated in three years with a double major?

Answer

Your question makes me a little sad. You are obviously a bright and ambitious student, and you have spent almost two years at a US public research university. You are trying to "theorycraft" your future career, which is good. But somehow no one has managed to communicate to you the basic goals of an undergraduate math major and the requirements for graduate study: how did that happen?

To once again answer the title question (the other two answers are completely on point): "Does rigor/thoroughness of undergraduate program matter (for graduate/Phd applications)?" The answer is yes, of course. Not only does it matter, but together with indications of your success in these courses -- course grades together with recommendation letters from your instructors -- it is what matters the most. Your question first proposed graduating early with a double major in physics and math, with coursework that to any math PhD admissions committee member looks extremely minimal. This is a terrible plan for getting into a PhD program in mathematics (not just a top one, but any one that I know).

Let me be clear: I am not scolding you. I am lamenting the total lack of advice you have somehow been subjected to. You are not the first bright, ambitious undergraduate who has asked a question on this site which evinces the misconception that the best students get through undergraduate degrees in as few years as possible. This is quite the opposite of the truth: does anyone know where these misconceptions are coming from?!?

Here is my opinion (as someone who served on the graduate committee of a top 50 PhD program in mathematics for four years): unless you want to work cross-disciplinarily in math and physics, the second undergraduate major in physics is not directly helping you get into a PhD program in mathematics. On the other hand, it seems to be stopping you from taking the deeper and more advanced course offerings that are essential for you to even get consideration at a good (or even decent) PhD program in mathematics. If your postgraduate plans include a career in which knowing a little bit about several different things will be beneficial -- and I can imagine that many careers in business and industry are like this -- then a double major could be desirable. However, for a future PhD student, all else being equal a double major is time taken away from the one thing you're supposed to be devoting much more time and energy to learning.

In your case, you absolutely need the fourth year of your undergraduate program in order to be considered for a top program. Unless you absolutely load yourself up with challenging and graduate-level math courses in that last year (which may not be the best idea for other reasons, e.g. it works against the level of your success in these courses), I would in fact be thinking about what to do after you graduate in four years in order to make your grad school application more competitive. You say that you want to take a year off before going back to grad school. If you still want to do that after a four year degree, then thinking about how to get some additional mathematical training in during that year off might be a good idea.