Should I put unit testing stuffs in a separate repository, not in the same repository as the programming library? So I reference the programming library as submodule. But most open source projects that I have seen do not organize the projects like what I mention above. Can anyone explain which approach is better?
You should put the unit tests in the same repository because otherwise someone has to answer to the question "Where are the tests?" every time the project is handed over from one person to another. References to other repositories tend to get invalid over time when repositories are relocated and people change from one version control system to another.
Just keep the tests close to the code.
You want the version of the tests to match the version of the code, so that implies you keep the tests in the same repository as the code. That goes for automatic code generation scripts, build scripts, etc. too.