TL;DR
Story points are estimates of the effort for the team as a whole to move a story from "started" to the "definition of done." If stories don't involve multiple skill sets or involve multiple people, you probably have a task rather than a story.
Story Points Aren't Man-Hour Estimates
Story points are a measure of relative effort:
If FOO is a baseline 2-point story
And BAR is roughly twice as much work
Then BAR is a 3-point or 5-point story.
In Scrum, one typically sees the Product Backlog estimated in story points, although opinion is often divided about whether individual tasks should be estimated on the Sprint Backlog. When estimating Sprint Backlog tasks, one can certainly use ideal hours rather than story points, but even so the estimates are generally for wall-clock time rather than man-hours.
Scrum Teams Use Self-Organized Time-Boxes
Man-hour estimates are usually a "project smell" that a team is not fully self-organizing. An effective Scrum team is agile precisely because it enables teams to shift resources from task to task as needed within each sprint.
If the team embraces pair programming, or if a story requires multiple team members to swarm over specific tasks, then the team should be factoring those things into both their story-point estimates and into their Sprint Planning. During Sprint Planning, the Scrum team peels each story off the Product Backlog, estimates it, and then tries to determine if the story will fit into the current sprint.
Team effort required to reach the "definition of done" tends to be a more accurate tool for this type of assessment than measuring man-hours. However, the team should use whatever metrics lead to the most success for the project.
Generally speaking, the key is to let the task performers (e.g. the development team) do the estimation. Man-hour estimates tend to be a budgeting tool imposed on the team's estimation process from outside, rather than providing an effective estimation technique for a time-boxed process. Scrum is based on time-boxing, not the 100% utilization fallacy, so it's probably best to avoid techniques designed to measure utilization right from the start.
Each team and each project are different, so your mileage may certainly vary.