For various reasons I felt like I couldn't just ask to go, so I just sat there and got more and more frustrated until I had a little thrash on the couch I was sitting on. Then my ex said, "...oh, I guess it's time for us to go now."
We talked about it after we left and he was angry/annoyed with me for not just saying I wanted to go, and that's understandable... but it's really hurting my feelings how angry he got with me. Like, I was frustrated enough at feeling like I couldn't leave, and then to be yelled at on top of that is making me feel really shitty.
I feel like I have to just let this go so we can move on to other things and not argue about this forever... but it's hard. I feel really crappy.
It's nice that you both want to mutually work your relationship out. It shows a sense of maturity. Talking with your ex sounds silly, but it helps. If he doesn't seem approachable, do it anyways. You might surprise yourself. Some guys just like to be talked to like they're men. He probably felt badly that you weren't having a good time, and took it out on you because inwardly he was kicking himself for not realizing you weren't enjoying yourself.
Just be honest. Let him know that his overreaction caused you to feel terrible, and it wasn't an appropriate response to the situation. Let him know what you were feeling that night. "Honey, I stayed because you were having a good time. I didn't want to ask you to leave if you were having fun. I wasn't upset. I was just happy to spend time with you." If you're straight up honest and in agreement of things, there can't be an argument.
Men appreciate to be treated as such. Sit down, talk to him. In person, not on the phone. I'm sure, that way, you'll be able to work out the snarls and have a more happy and healthy relationship.