Lefty against lefty should give the advantage to the pitcher. If he didn't allow a homer to any righty then 50 batters shouldn't account for 4 homers to lefties. He just wasn't as effective against lefties as he should have been,or, he was better against righties than he should have been.
The righty/righty thing isn't quite so stark because from the very beginning, righty hitters have to learn to hit righty pitchers because they face so many. However, leftys generally have trouble adjusting to hitting LHP. As a lefty hitter, I can tell you that a lefty with a good slider is kryptonite to a lefty hitter. As a lefty pitcher, I always felt I had the advantage over a left-handed hitter.