vision - How do we know the brain flips images projected on the retina back around?

Why do we turn images upside down again rather than dealing with them directly, still vertically rotated after passing through our lens?

I don't see how that would cause any problems, and how we'd ever be able to figure out if we are presented with flipped images after getting used to interacting based on visual input, whether flipped or not.

What am I missing?

Answer

The basis of this question is a common misconception, and unfortunately the accepted answer by @CHM is also based on this common misconception. The misconception is based on the homunculus falacy: the tendency for people to think that the image that lands on the retina is somehow 'assembled' and presented for something (the 'consciousness') to view. This is not the case.

As the comment by @mgkrebbs explains, there is no orientation (up or down) in the brain, there is only neural firing. The information of the visual scene is distributed over the brain, and information does not have physical properties like orientation. Although as @nico pointed out the neurons that process the information do have a spatial structure that mimics that of the retina, this is a topological property (i.e. stimuli that are close on the retina are processes by neurons that are close in V1) and such a topological property does not induce an orientation.

The root of the problem is really that the question "How do we know the brain flips images projected on the retina back around?" is a pseudo-question. Although it is grammatically well-formed, it makes no semantic sense. When the image is 'in the' (i.e. being processed by the) brain it no longer has physical properties like orientation. Thus you cannot ask if it has been flipped or not.