RPGMemes

That's the thing though, it's not invulnerability, it's, for lack of a better term, will realized as magic.

In Moria,

"...Gandalf stood firm. 'You cannot pass,' he said. The orcs stood still, and a dead silence fell. 'I am a servant of the Secret Fire, wielder of the flame of Anor. You cannot pass. The dark fire will not avail you, flame of Udûn. Go back to the Shadow! You cannot pass.'

This is an invocation - You cannot pass, I am a Maia of Eru, wielder of Narya, Morgoth is gone, Balrog, You cannot pass - and he does, indeed, not pass.

At Gondor, the actual fight starts earlier:

For yet another weapon, [..] the Lord of the Dark Tower had: dread and despair. The Nazgul came again, [..] More unbearable they became, not less, at each new cry. At length even the stout-hearted would fling themselves to the ground as the hidden menace passed over them, [..] and they thought no more of war; but only of hiding and of crawling, and of death.

And so,

So it was that Gandalf took command [..] Wherever he came men’s hearts would lift again, and the winged shadows pass from memory. [..] And yet – when they had gone, the shadows closed on men again

And, finally,

"There waiting, silent and still in the space before the Gate, sat Gandalf [..] 'You cannot enter here,’ said Gandalf, and the huge shadow halted. ‘Go back to the abyss prepared for you! Go back! Fall into the nothingness that awaits you and your Master. Go!’" The Witch-King responds: "‘Old fool! This is my hour. Do you not know Death when you see it? Die now and curse in vain!’" and lifts his sword to strike, and then: "Gandalf did not move. [..] a cock crowed. Shrill and clear he crowed, recking nothing of wizardry or war [..] And as if in answer there came from far away another note. Horns, horns, horns. [..] The darkness was breaking too soon, before the date that his Master had set for it: [..] he left the Gate and vanished."

The night assault was driven by the ringwraith terror. Gandalf doesn't confront the Witch King physically, he denies his terror, and denies him entry, and when he, unwisely, forces a confrontation, his whole spell folds like a cheap suit.