Tension with the soviets has been around for a long time. The events that took place before the Olympics show the reasoning for the boycott. In December 1979 Leonid Brezhnev called for an invasion of Afghanistan to lead a communist action. This caused lots of mixed anger and frustration for everyone. This along with other actions done by the Soviets is what led Carter to try and get the Soviets to leave afghanistan. This led to the idea of a boycott which soon became a reality. 

American protesters at the 1980 Winter Olympic Games in Lake Placid

Wally McNamee/CORBIS/Corbis via Getty Images

The exterior of Lenin Stadium, which was the main site of the 1980 Olympics, with a statue of the Soviet Union leader,

An attempt at Diplomacy