“Offense is determined by culture, not something we are all intrinsically born with”

Indeed. Which is why symbols historically associated with Nazis and slave owners have become popular with racist types, even if they didn’t start off meaning that.

“It’s quite possible that in 1960, the swastika was still seen as an ancient Asian symbol that a bunch of kooky Germans decided to adopt as their own for a short period of time. “

Extraordinarily unlikely. It’s far more likely that people will have been unaware of its origins and only knew it from its use by the Nazis. Those are people who experienced the war, they did not have the historical distance that we do. Also remember that the Nazi version of the swastika was deliberately oriented differently compared to the Buddhist symbol, an orientation that seems to be shared by the building (though this can be disputed it were indeed deliberate).

“It’s funny that although the Nazis had many other symbols, and the Confederate States of America had many other flags, they are today represented by a single icon in the eyes of the general public”

Most religions and cultures have more than one symbol, but some can be changed by their use more than others. There are other Nazi symbols that existed, of course, but they have not been adopted by modern-day neo-Nazis and white supremacists in the same way. Similarly, although there were certainly other Confederate flags, that particular battle flag was adopted as a symbol of racism long after the US civil war. I’d read up on how it was rallied behind during the civil rights era. These symbols were not picked randomly.

“a typical architect in 1960 would not have batted an eye at a building layout that would today almost certainly draw an instant gasp of horrified disbelief. “

I find it unbelievable that a person who was alive during the war, and went through the post-war discovery of the depths of the atrocities committed during that times would find the symbol less significant than those today, even if that person was a child at the time. A professional architect working for the military less than a generation following the war would at least be aware of its major significance, and was quite possibly a member of the military at the time of the European theatre. Even if he somehow came up with it as a functional design rather than a deliberate tribute, the money you’re concerned about would have been saved if someone had realised how it looks from above. There’s no questioning what it looks like, only a question as to why. The reaction to it would likely be the same back then, if people knew what it looked like from the satellite view

“Anyway, far more people died in captivity under Marxism than under ‘Naziism’ or ‘Confederatism'”

So? Even if you’re correct and not just deliberately trying to conflate a bunch of left-leaning/communist/whatever evens under “Marxism” (what is a “Marxist symbol”, anyway?), that means nothing. Just because you’d personally put Nazis lower on the list of atrocities does not mean that their symbol should not be controversial. The KKK killed less people and co-opted the uniform of Spanish monks, but that doesn’t mean that those robes aren’t still offensive today, even if they are being worn during Semana Santa by the orders who originated them. That may be unfortunate and even wrong, but that’s reality.

“we are not taught to view Marxist symbols in anything close to the same light”

I grew up in the 80s, where Soviet symbolism was certainly something American movies and politicians taught you to fear. Perhaps it’s not as prevalent now because the old Soviet bloc no longer exists, while racist and Nazi symbols continue to be rallied behind? I don’t recall anyone seriously flying the hammer and sickle recently, but I can think of recent events where literal Nazi symbols were used as a rallying point.