I'm a bit suspicious of Swafford's thesis. For one thing, while brutalism is often taken as art meant to "brutalize" or hurt the audience, a quick web search reveals that it's actually art inspired by the (putative) beauty of raw concrete (beton brut). For another, I don't personally find the referenced track unpleasant to listen to or necessarily hostile (although it certainly might have a different "impact" live). It's further worth noting, that the referenced performance receives a sustained ovation from a clearly moved audience, and that many of the comments on the video reference the piece's "lyricism" and "passion."
With that said, artists seek a wide variety of reactions from their audience members. Particularly in modern times, many artists' primary goal is to affect their audience, and not necessarily to charm or enchant them. It doesn't strike me as odd at all that causing pain to an audience --not physical, perhaps, but emotional or purely aesthetic pain --might well be a legitimate goal for an artist to seek. From this point of view, Swafford's contention that the artist doesn't seek to "move" you but to "hurt" you may well be a false dichotomy.
It wouldn't be everyone's cup of tea, of course, but not every artistic experience needs to be for every audience member.