If evolution is not about increased complexity, why does so much complexity evolve?

In my last question I asked why we don't see increased complexity in artificial life simulations of evolution. It seems I had fallen for a common misconception, that evolution was about improvement by increasing complexity. One comment discussing that post read

"... he [David Deutsch] is falling for one of the biggest misconceptions about evolution that you can, that evolution is about improvement. Evolution has simply only ever been about change..."

However, when you look at the history of life you see increases in complexity. You see this increasing complexity evolving over billions of years, suggesting that it requires an explanation.

My question
If evolution is not about increasing complexity then how does so much complexity evolve?

Answer

I think possibly the problem here is the way you're approaching the issue.

You're considering improvement as anything that increases the abilities or complexity of the organism—that isn't necessarily what an improvement is though. The outcome of natural selection is that the organism best equipped to survive/reproduce in a certain environment is the most successful. So, for example, thermophillic archaea do much better in 60°C-plus pools of water than humans do. Our capacity to process information, use tools, etc. doesn't actually confer much advantage in that situation. And there can be downsides to that kind of complexity as well, requiring more energy and longer developmental periods. So, natural selection in 60°C-plus pools of water gives you archaea, and in (presumably) the plains of East Africa, it gives you humans.

The comment you quote mentions sickle-cell anaemia, which is a different example. While there is little benefit to having the sickle-cell anaemia allele in a temperate region, in those regions where malaria is endemic, heterozygosity can provide a survival advantage, and so the allele is maintained in the population. If you're someone living in a malaria-endemic region, and you don't have access to antimalarials, heterozygosity for the sickle-cell anaemia allele is arguably an improvement. It depends entirely on how you define the word.

The fundamental principal of natural selection is that it favours the organism most suited to a particular environment. But, that isn't always the most complex organism. It's important not to confuse human-like with better. It isn't the universal endpoint of evolution to produce an organism similar to us, just the organism most suited to the environment in question.

Also, to briefly address the previous question you asked—you asserted that we must be missing something from the process of evolution because we were unable to simulate it. You also pointed out that (in your opinion) we have sufficient computing power to simulate the kinds of organisms you're referring to. But natural selection is intrinsically linked to the environment it occurs in, so the simulation wouldn't just have to accurately simulate the biological processes of the organism, but also all of the external pressures the organism faces. I'd imagine that, in simulating evolution, that would be the real obstacle.