The Buddha understood the nature of duḥkha. One of the deepest causes of duḥkha is the belief that the world we experience through the senses is the whole of reality. When we believe this, we naturally search inside that same world (that built the duḥkha) for something that will finally end our suffering. So we look for refuge.