If you don't know what you are doing, then the last thing you should do is write a requirements spec, as all that will give you is a requirements spec written by someone who doesn't know what they are doing.
Maybe someone else knows what they are doing - if so, they should write the spec. If not, you have to learn what needs to be done. And for this it makes little difference whether its a new activity that literally noone knows how to do, or there theoretically is someone who could tell you, but you don't have the budget to pay them to spend time doing so.
The problem with requirements specification as as a learning activity is that it doesn't give any feedback as to whether what you have specified is right or wrong. There is no such thing as a requirements compiler or test tool that comes back and says 'this is wrong, it makes no sense, doesn't solve the problem, or noone will like it'.
Exploratory prototyping is one technique that suits some people as a means of learning enough about a domain to write a useful requirements spec. But you shouldn't assume it will be the best in any particular case without considering alternatives. And you certainly shouldn't be expecting to get much use out of any software produced as a byproduct of that learning process, any more than you should expect to be able to sell the notes you made in class for some fraction of the price of a university education.