My Dentist once told me that letting go is like pulling a tooth. When it is pulled out, you're relieved, but how many times does your tongue run itself over the spot where the tooth once was? Probably a hundred times a day. Just because it wasn't hurting you doesn't mean you did not notice it. It leaves a gap and sometimes you see yourself missing it terribly. It's going to heal. but it takes time.
Should you have kept the tooth? No, because it was causing you so much pain.
Therefore, move on and let go.