Another pumping/oversupply question

Macy628

For some background:  My son was born at 34 weeks and wasn't able to breastfeed right away so I had to pump every 3 hours.  We used the milk for his tube feedings and then to supplement with a bottle after he would attempt to breastfeed.  He still gets a bottle with BM and NeoSure (he needs to gain weight) after breastfeeding but has been taking less lately since it seems like he is able to eat more at a session.

I seem to have a really strong let down since he's coughing and pulling away during feedings.  We still have to offer him a bottle with formula mixed with my milk after he breastfeeds.  So I kept up the pumping every 3 hours (mistake).  I was pumping 3 oz after feeding him when we first got home last weekend.  Fast forward to last night and I was completely engorged after 4 hours and pumped 7 ounces after he ate.  What do I do???  

I want to start weaning off the pump but I don't know how to do it without feeling like my boobs are going to blow up.  I started only pumping 5-8 minutes yesterday, but then I still had hard knots that weren't getting out.  I was worried about mastitis (had it several times with my first son).  

I am making way more than he's eating.  Should I try feeding him on just 1 side and pump the other and deal with the side he ate from still feeling full?    I checked out kellymom but I wanted to see if you ladies had any experience or suggestions.  

Thanks for getting through that whole long thing!