You just can't ask somebody who's never played guard to play guard. He played right tackle his entire college career. It takes some time to adjust your footwork, stance, hand position, etc...
I'm with you. His two best years came at right tackle (yes I'm using pff as a source). He was below average in 2019 as a guard for the Panthers. The only way I see Spencer Brown getting playing time is an injury to Dawkins or Williams.