You don't rely exclusively on point estimates for sprint planning anyway. During planning, your velocity can help you to quickly "propose" a sprint plan and/or assist with your sanity check before making the final commitment; but, no matter how much velocity data you have, the team ultimately needs to commit to stories, not points.
Bear in mind, 5 x 1-point stories do not equal 1 x 5-point story for exceptionally small values of 1. They are only roughly equivalent for the purpose of long-term planning.
And that's where the primary value of the points comes into play: when you need to guesstimate when an epic or release can be completed, or whether you have too much on your plate for a scheduled release-date.
That said, a successful team will eventually find itself planning about the same number of points each sprint. But, there will be variances based on the particular stories in that sprint. A successful team can articulate during planning why some 5-point stories don't actually offset all 5 of the initially proposed 1-point stories ...