I'd say "he's a smart guy" assumes facts not in evidence.
But yeah, naming something after James Madison is just the same as calling a football team the Redskins. Please. Back before the Montreal Expos actually moved to Washington and acquired their current team name, there was a lot of speculation about what the team name would or should be. I always argued that, in the spirit of the name of the NFL team, they should be called the Washington Darkies. Would that have been okay with Will? If not, what distinction would have made "Redskins" okay but put "Darkies" out of bounds?
Sorry to seiz