There is nothing about story points or scrum that makes assigning story points to something impossible or even difficult, unless you have massively altered the definitions within your own process. Story points are an estimate based on complexity, with only a vague relation to time. In most times story points eventually become more time based which makes assigning them to any arbitrary story or issue easier. Second, there is the misunderstanding of what is meant by a sprint being able to deliver a shippable product at the end of every sprint. Stories and sprints are intended to deliver units of value to the product owner, this is commonly misinterpreted as working code for every story. In the case of hypothesis driven development the yes or no is the valuable part, stories should work towards how to get that yes/no, and what to do with a yes/no. There isn't a need to write or estimate stories to fully develop and implement a hypothesis until you've tested it, and at that point you should have sufficient insight to deliver reasonable estimates. It's even possible and useful to have stories that have an output of other stories.
It sounds like you are in a common pitfall of having a separate requirements gathering, getting a bunch of things written up, and now trying to throw it over the wall to the development team. This is a waterfall mentality that is hard to break, a preferable alternative is to have stories that involve parts of your development team to help take the idea and turn it into stories and requirements. If you already have a lot of things the best solution is to write it up in a logical story breakdown (as a first pass) and use some backlog grooming time to restructure and flesh out the initial stories.
As for how a product owner would prioritize these items, it becomes straightforward once they get written up as features/epics/stories. It becomes a simple question of is knowing if hypothesis A is worth pursuing is more or less important than new feature B or bug fix C. If hypothesis D is a complete unknown on how hard designing an following a test will be, then make a story to find out and prioritize that. Adding "what do I want to know" to sprint planning processes can be a helpful way to get a lot of valuable user stories that aren't really development work, but are often more useful.