Ummm...he was.
"I wish I could tell you that Andy fought the good fight, and the Sisters let him be. I wish I could tell you that - but prison is no fairy-tale world. He never said who did it, but we all knew. Things went on like that for awhile - prison life consists of routine, and then more routine. Every so often, Andy would show up with fresh bruises. The Sisters kept at him - sometimes he was able to fight 'em off, sometimes not." -Red (narrating)
Sure, it wasn't American History X style vivid and explicit, but I thought they made it pretty clear.