If you walk up to a human on the street and ask "How big was a T-rex?" the answers would fluctuate even though majority of humans know what a T-rex is, how big it kind of was, but nobody really knows for certain - because we have NO relative scale to baseline from.
That's the cognitive behavior you're trying to figure out with forecasting and many methodologies spin cycles with "I've got it!..i have the secret to accurate forecasting!" snake oil to the masses. When you actually do forecast you're actually saying outloud "I will ALLOW x days/hours/points for that to complete" - its in a sense creating a "timebox" for that event to be carried out within.
For me, Points is just shifting the boundaries, at the end of the day unless you're in a team that is happy to say "*Well we have 3 weeks per sprint, and thumb suck...i figure we should shoot for 30 points to complete in that cycle! who's with me!*" and thats as deep as you go in forecast modeling - fine! ..as realistically you're just setting an arbitary budget and that's it. You're also then in retrospective looking at the work completed with a sense of "holy crap, we did 33pts that sprint, that was pretty cool" and not much can be done about that. You can use velocity to determine mid-sprint you're getting bang for your budget buck by asking outloud "Have we hit 15pts yet? will we" but then your back to adding relative time to the equation aren't you?" but the danger here is you're now using Velocity to measure productivity not capacity which from what I understand kicks the Reactive Release Management (story points) in the head..
The point system is almost too clever to not notice that you still attach relative time to the equation, everything from your agreed "sprint cycles" to your daily standups in which you enact some hidden rule around duration + complexity = "Max is taking to long with that task" innate gut feeling team code red moment?
The human brain cant forecast because it involves a lot of working memory mixed with long/short term recall, so its like asking a novice math student to do fractions in their head not on paper.. It's why other industries never agree on a forecast and constantly validate forecasts in relative time (eg geologist never stop forecast modelling until that cubic meter has been dug out of the ground and then its "done").
I'd say Point system works if you're not forecasting. You're agreeing to a chunk of work that's based on a sub-chunking algorithm but that's really your closest approach to forecasting as possible. In fact you're release management would look for natural breaks in the "backlog" queue that fit around theme(s) (ie in Silverlight we Product managers would wait until after they complete their backlog and piece together the themes we initially set. We never knew what the engineering team were doing specifically we just had a basic outline. We'd then take that body of work and build our marketing event around it (Microsoft Mix))
When you start locking down velocity expectations inside sprint cycles that rely on velocity + time, you're back to forecasting estimations again only this time you're worse off because you're playing the "it depends game" ... More importantly you're also killing potential for team growth / career growth as well.
The tax you pay for Points vs Time is with points you need to look for alternative measurement formulas to track onjob skill development / mentoring or developer behavior.
As in you will still need look at a "median developer" as your ideal person to attach skill/effort with, you can then baseline other developers with that person to determine how they are fairing in their ongoing growth within your team. It also highlights situations where the "fast" developers are carrying most of the water but are getting bored or worse they are working longer hours and no recognition / reward because of competing deadlines etc. Standups don't unearth this in reality, they are really there to detect bad smells within the team per say, as in "that person is struggling, lets help"
Next also comes the "carry over" stories, stories that don't get chunked into that sprint cycle but then spill over to the next sprint cycle. Which then can easily create a knock-on effect if you're factoring in time, but the moment you do factor in relative time..again, you just regressed back to "time based forecasting/estimation" and again the point system is just muddying the waters.
If you go points you have ignore time completely and i mean completely as the moment you let time creep in you're gaming the idea / methodology.
Having travelled around the world as an Evangelist, I saw a lot of teams swear their hand on whatever they hold dear that they have cracked the Agile Forecast code... but I always clicked my tongue, smiled and walked away with the thought "yeah...you almost did, but that mistress we call 'time'... she's just cruel..."