"Religion that God our Father accepts as pure and faultless is this: to look after orphans and widows in their distress and to keep oneself from being polluted by the world."
When I lived in Pennsylvania my dad used to take my brother and I down to the river to fish. I can remember those days very vividly. It was there I learned to throw my line in way upstream and let the current carry it past me from left to right. The Susquehanna gathers volume as it passes through our little town of Wyoming, and it carries a lot of debris as well. On good days we avoided snagging old tires or bedsprings hidden beneath the surface. As for the fish, whenever we caught one we generally threw it back because we didn't want strange contaminants infecting us. The fish weren't glowing or anything, they just weren't useful as healthy food.
I think James is referring to something like this when he warns us to "keep oneself from being polluted by the world." If we allow ourselves to become contaminated we will become useless, much like the fish in the Susquehanna. The fish can't do much to avoid the contamination of their surroundings, but we have a supernatural agent in the Holy Spirit that works within us (Gal. 5:16) to avoid the filth of the world. But, beware, lest you rest easy and find yourself drifting downstream and sickly. The Christian life is a fight upstream that requires constant effort and vigilance. The best way I know how to overcome the pollution and current is to be frequent with prayers and devotions. Stay close also to friends that can strengthen your resolve and prick your conscience when they see contamination in you.