Silvera was born in San Diego, California, to Albert and Victoria Silvera.[1][2] He was Jewish,[3] and of Jewish and Italian descent on his paternal side and of Syrian Jewish descent on his maternal side. He was the nephew of former major league pitcher"Subway Sam" Nahem.[4][1] He threw and batted right-handed, stood 6 feet (1.8m) tall and weighed 180 pounds (82kg).
Silvera attended Fairfax High School in Los Angeles.[6] In 1952, as a sophomore outfielder he was named to the All-Western League First Team.[7] In 1953, he batted .500 and received All City honors, was named the Western League Player of the Year, and was named to the All-Western League First Team.[7] In 1954, he batted .367 and received dual All-City and All-Western League Player of the Year honors.[7]