We have a big set of survey questions, which we distribute to users online. We collect submission from users and based on their submission, we decide the answers for those questions. Now, each user gets a set of questions, say N. For each submission given by user, he/she gets some points(cost c). Now, there is a upper limit on questions such that if a question receives more that k submissions, it will be disabled from next time assignment.
Now the problem is, sometimes, users take questions and don't submit it for a long time. So, we allocate same question to many users (say m, m > k), and after sometime everyone submits the solution, so instead of required k submissions, we get m submission. Which in the end increases the total cost attached to survey. Is there any algorithm to solve things like this.
We thought of doing a 'soft reservation' system, where we will keep track of each question's assignment count, so that if we have given k users the same question, we won't give it to other users. But, it has side-effects which includes timer check for each question whether its submission has come or not. Which is kinda dirty.
Other which looks closely related is assignment-problem, hungarian-algorithm, but its solution isn't what we are looking for. Please help.
There is another clause of random selection of questions. We want to give differently selected set of questions to each user. So now, if I have 50 users and 1000 questions and a question's max submission (k) is 5, how can I achieve result in minimum cost spent.