During our sprint planning meetings, we use a planning poker deck to reach a consensus on estimates, with the cards indicating how many days we estimate to need. We then use these estimates combined with our estimate for how many days we can work this sprint to determine what items we will do this sprint and which ones are for the next sprint. We do this because our developers are rather specialized and some tasks are preferably handled by certain developers. Because of this, estimating effort gives a clearer picture compared to estimating complexity. It's not pure agile, but it works for us.
During this, we find that for certain stories, 3 days is slightly too optimistic, but 5 days is too far in the other direction. In effect, we sometimes miss a card that reads 4. When this is the consensus, we do write down 4, jokingly saying that "we will be thrown out of the Agile party".
Joking aside, Is there a specific reason as to why Planning Poker doesn't have an option for 4 story points/days/whatever? I know that it originated from the Fibonacci sequence, but still, 4 is a low number, which means that as an estimate of either complexity or effort, it's still reasonably accurate.