Context: Finding company to do the job

The use case of an application I want to write is that a user will describe a task to be performed by some company, e.g.,

I want to move 10 boxes (30x40x50cm) of books and clothes from London to Bristol next week. How much would that cost?

and the application will find candidate companies that could do the task. In this case it would need to categorize the task as for removal/transportation companies.

Problem: incorrect categorization

I tried a number of online services for text categorization but results are discouraging. uClassify classified my text as "Home", "Games", and "Arts"; Textimate.me --- as "Science & Environment"; Textwise returned "Business/Consumer_Goods", "Services/Clothing", and "Arts/Design/Fashion".

The possible reasons for incorrect categorization:

  • task description is very short and there is no evident characteristics in the text to differentiate what is more important for the user: boxes, clothes or maybe moving them,
  • classifiers trained on other types of texts,
  • classifiers trained with respect to a different taxonomy.

How would you tackle such a problem?

Potential solution: harvest domain-specific corpus

Since the problem is quite specific to my domain I think about using supervised machine learning algorithm, that would learn with respect to already categorized corpus of such tasks. The corpus can be harvested from sites with tasks already categorized by customers (variation of e-bay). I could also pre-process text before machine learning: extract only verbs ("move") and nouns originating from verbs ("translation"), as they denote what has to be done.

This requires quite a lof of work, so I'm curious whether this is the right direction.

  • 1
    Can you tell us a little bit more about what you've tried? Also, it would be greatly appreciated if you could edit your question with appropriate formatting so that it's easy to understand the problem your facing.
    – Jim G.
    Jan 6, 2013 at 21:58
  • Jim, thank you for your feedback. Please, let me know if the question is more clear after my update.
    – dzieciou
    Jan 6, 2013 at 22:20
  • 1
    Awesome. As it turns out, this is a terrific question for Programmers.SE. Thanks.
    – Jim G.
    Jan 6, 2013 at 22:27

1 Answer 1


I think your problem entails a mini-schlep:

There are great startup ideas lying around unexploited right under our noses. One reason we don't see them is a phenomenon I call schlep blindness. Schlep was originally a Yiddish word but has passed into general use in the US. It means a tedious, unpleasant task.

No one likes schleps, but hackers especially dislike them. Most hackers who start startups wish they could do it by just writing some clever software, putting it on a server somewhere, and watching the money roll in—without ever having to talk to users, or negotiate with other companies, or deal with other people's broken code. Maybe that's possible, but I haven't seen it.

You've identified one element in your business plan that will be difficult to solve with only software. This is a great sign!

One of the many things we do at Y Combinator is teach hackers about the inevitability of schleps. No, you can't start a startup by just writing code.

That scariness makes ambitious ideas doubly valuable. In addition to their intrinsic value, they're like undervalued stocks in the sense that there's less demand for them among founders. If you pick an ambitious idea, you'll have less competition, because everyone else will have been frightened off by the challenges involved.

So here's my advice:

  1. Write code that does a reasonable job of scanning the user's input for keywords. When implementing this piece, be mindful of the Pareto Principle (or 80-20 rule).
  2. In your keyword matching code, allow for a default case that does not match on any keywords. On the screen, apologize to the user that you could not find a match, but encourage them to return soon because you are constantly improving the application.
  3. In your keyword matching code, log the incoming keywords and the match that it found in a data store.
  4. Review this data store frequently. Perhaps thrice daily on working days, once daily on non-working days.
  5. The data should give you clues about how you can improve your algorithm, so pursue a strategy of continuous improvement.
  6. In your spare time, learn more about computational linguistics and how its concepts can help you better solve your business problem.
  • +1 That's a very practical answer and does not blocks my work. What do you mean by Pareto Principle in this particular task?
    – dzieciou
    Jan 8, 2013 at 18:49
  • 1
    @dzieciou: I mean focus first on getting 80% of your use cases correct. Then focus on getting the remaining 20% correct in an iterative fashion.
    – Jim G.
    Jan 8, 2013 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.