It sounds like you are using story points incorrectly. Story points are relative to each other for this team only, and nothing else. For the very first sprint you have nothing to compare points to so you have to make educated guesses. After that, it should become easier and easier to be consistent in sizing.
With each sprint except the first, pick a previous story that seemed to be sized approximately right and is vaguely "average" for however you want to define that. If you don't have any stories that were properly estimated, pick a "typical" story from the past and decide what points it should have had using your 20/20 hindsight. Whatever story you pick becomes your new standard.
Next, pick a new story and simply ask yourselves if this is considerably easier, about the same, or considerably harder than this previous story. Don't think at all about how much time it takes to complete the story. Don't think "well, this will take two more hours", think "this is half as hard" or "this is twice as hard". That's all you have to do.
If you've estimated something as 5 points, and it took 4 hours longer than the previous 5 point story, don't worry about it. If it took a few hours less, don't worry about it. Those are all 5 point stories. You can start worrying about it when you size a story as 5 but it takes as much time as most 8's.
When you become better at recognizing similar sized stories, estimation becomes easier and easier. You will be off by a lot for a few stories, but you'll be short on some and long on the others, and it will mostly even out.
TL;DR: it's important to remember that the points mean nothing, and velocity is an indicator. What matters is consistency. Virtually every 5 point story should take more time than a 3, and less time than an 8, but if one 5 point story takes 8 hours and another takes 12 or even 16, that's probably ok as long as it's still less than an 8 point story.