Not a kashya, the whole city isn’t patur because they’re not learning.
He was answering two yeshiva bochrim who were asking if they should join the army, and he gave the most basic reason not to, so they can continue learning. His answer applies even if there would be units which would be glatt kosher in ruchniyus.
It would be more instructive had they asked him if they should work or join the army, and described the spiritual danger. But that wasn’t the question they asked.