Pagano only spent 1 year as DC with the Ravens. In 2010, the Ravens were 3rd in points allowed. In 2011 with Pagano as their new DC, they were 3rd in points allowed.
As for the Colts, Pagano was HC for 6 years...thats far too long a time to offer up the "didn't have the talent to be very good" excuse. The fact that Pagano couldn't construct a defense that was better than 30th in the NFL in years 5 and 6 is frightening. The fact that the Colts defense got immensely better after he left is even a bigger red flag.
As for the Colts, Pagano wasn't the GM, so he's not responsible for the lack of talent on the team. Pagano had basically nothing to work with his first 3 years when he literally only had one defensive player who made a probowl on his roster. Also, the one year they actually had a pretty talented healthy defense (his fifth year in 2014, contrary to your version of events), their defensive was good.
As for the Ravens, total yards allowed and total points allowed are stats for amateurs. Advanced stats and efficiency stats give you a much better picture of how good a defense actually is, and if you look at the advanced stats, they were much better under Pagano in 2011.
My personal favorite advanced stat is PFR's Expected Points Contributed by All Defense. The difference there is glaring:
2010: 58.34 (7th in the NFL)
2011: 133.31 (1st in the NFL by a huge margin)
2012: -10.19 (12th in the NFL)
There are others, too. Yards allowed per defensive play improved under Pagano then regressed when he left.
Same with opponent QB rating:
2010: 76.4 (5th in the NFL)
2011: 68.8 (1st in the NFL)
2012: 80.6 (11th in the NFL)
Same with yards/points allowed per defensive possession:
2010: 27.2/1.46 yards/points per defensive possession
2011: 24.5/1.27 yards/points per defensive possession
2012: 29.0/1.67 yards/points per defensive possession
I could go on and on and on, but the Ravens defensive was overall much better in every defensive efficiency stat under Pagano than they were the year before him or after him.