The product manager doesn't manage the team, but the product backlog. Having any hierarchical link will make things harder.
I also suggest you to remove yearly evaluations from your company. They are individual, and Scrum is focused on a team rather than individuals.
There is no hierarchical link between the Scrum Master and the rest of the team. The team is self managed.
The ScrumMaster ensure that the scrum process is properly implemented and help the team on a daily basis.
The team reports to the team during the daily stand ups.
The team reports to the product owner during the sprint review.
Simple and effective.