"The world" is the domain of human culture and interaction. And it is not singular. We speak of "the art world," "the wide world of sports," "the academic world," and "the world of fine dining." Magazine titles include Runner's World, Swimming World, Guitar World, Fishing World and Bridge World.
So when we hear of hermits "leaving the world," or of monastics or priests "leaving home," it doesn't mean disappearing altogether; it means taking leave of a certain kind of structured human domain with its rules, expectations, fashions, criteria of success and failure, heroes and villains, internal history, and the like. It means attempting to live a life unhindered by these concerns. These persons still engage in any number of activities with other people; they just do it on different terms and on a different schedule.
If such persons strike us as odd, it is perhaps more a measure of our inability to imagine ourselves free of these concerns than it is anything about them. They make us confront our own attachments, and we, seeing them with a heart that is at all receptive, understand how far it is we have yet to go.
Let us hope they never disappear:
With folded hands I beseech
The Conquerors who wish to pass away,
To please remain for countless eons
And not to leave the world in darkness.