Size for sure can make a big difference!
But the good news is that you can stretch to accommodate your bigger partner.
We always encourage trying to dilate a little larger than your partner, so that he will be comfortable. I encourage my patients to measure their partner when erect, and then try to find a dilator or dildo that might be a little wider than they are. I am less concerned about length, the width is the more important.
I would try to do some dilation exercises the week leading up to your next sexual encounter and see if that helps.