1

I am currently working a data science team, and we write small utilities all the time to consolidate data, extract data, etc (an example is go through a folder of excel workbooks, convert all the sheets into csv files, and consolidate those sheets that have the same column headers, adding columns for the source locations). Not all of the utilities are even in the same language (we have powershell, bash, python, R, SQL, etc.)

The number of these types of utilities is growing, and while we can store all of the code in git, we don't have good way to make these utilities discoverable, document them in consistent place/way, or even know what they are supporting.

Is there a good solution to this problem?

  • 2
    Every useful utility or open-source project I've ever found in a Google Search also had a companion web page that describes the project, it's purpose and how to use it. – Robert Harvey May 27 '18 at 3:59
  • Is there a system that I can stand up that will have a place to put all that information (something like an internal PyPI that supports multiple languages)? The best I can do without that is code in git, and hopefully stable links to releases in TFS. – soandos May 27 '18 at 4:22
  • How many utilitites are we talking about? Maybe it's time to start consolidating some of these libraries into fewer, larger libraries with specific themes. – Robert Harvey May 27 '18 at 4:39
6

As I see it, the key problem you are trying to solve is an avoidance of duplication of effort. I.e., once a problem has been solved you would like to not solve it again. In my opinion, no existing approach both solves that problem perfectly and doesn't require a fair amount of time to set up correctly. That said, there are plenty of partial solutions with varying trade-offs, and I'll mention a few of them.

  1. Refactor utilities into components of higher-level utility modules. Scripting languages like Python can be used as a glue and provide an interface to all the utilities your team makes. The documentation can then be monolingual, and it might be advantageous to auto-generate a lot of it from the docstrings and other source documentation. At the expense of having to code an interface, your team then has a consistent way to search and view all their utilities.

  2. It might not be feasible to require every dev platform to support a common language, to duplicate some efforts by developing an interface, or to require an entire library of utilities to be loaded when only one is desired. If the utilities are treated more as snippets to be dropped into place in a larger pipeline, what you really need is a strong way to search those snippets.

    • As Robert Harvey said, most useful projects have a companion web page. If web pages are not your team's strong suit, nearly anyone can set up a wiki. The primary disadvantage of this approach for a large number of tools is that the organizational structure must be explicitly outlined. The onus is on the creator of the documentation to make sure tools are easy to find, and that effort is akin to the organizational effort in refactoring utilities into larger modules. To some extent, documentation will lag behind capabilities.

    • There are a variety of software tools for creating Personal Knowledge Bases. A local linux advocate Noah Chelliah recently suggested osTicket as an ideal tool for this, and it's what he uses for his company. It still doesn't quite do everything I want in a PKB, but it supports code snippets in any language and has excellent search functionality along with stellar permissions handling. Your documentation to some extent could then consist of placing any new code in the knowledge base. The internal search features would allow easy retrieval of that information. They offer a self-hosted option if FOSS is important to your team.

  3. To some extent, not all small utilities need to be included in the team's documentation, so long as relevant pieces are appropriately documented in their appropriate projects. This will reveal my Python bias, but I recently made use of two pseudo-inverse operations int(`t`.encode('hex'), 16) and eval(('%x' % t).decode('hex')) in uncovering private information from CodeFights. Attempting to search for how to solve that kind of problem in any documentation system will likely take longer than just spitting out the one-liner every time I need it. Many steps in a data science pipeline like batch-processing excel files can similarly be done in a line or three with minimal effort, and it is far more important to have each pipeline well documented than to have each utility readily accessible.

  4. The data science flavor to your question adds a couple of interesting points. The FOSS project "Cookiecutter Data Science" is designed for Python, but they make a couple of good points pertaining to more diverse teams and the documentation of individual projects.

    • Your original data sources should be immutable. The subsequent analysis should be a Directed Acyclic Graph. With this approach, familiar utilities like Make can form a canonical documentation for your data transformations in any given pipeline. For sufficiently complex operations, Make might not be reasonable (see the GHC for an example of Make gone wrong), but I'd be willing to stake quite a bit on Make sufficing for your use case. As an added bonus, makefiles completely document the dependency chain for producing your datasets and models. Combined with some comments in that makefile, very little extra developer effort needs to go into documenting a pipeline beyond what was needed to produce it in the first place. The file can be easily parsed to yield a more useful view of that documentation (graph theoretic models and imaging are what I prefer, but straight XML summaries or other forms of mark(up/down) aren't terrible).

    • Tests should apply to data and models, not just to code. If you're assuming no NaN values, add a test for that assumption. If you're assuming at least 70% non-empty cells, add a test for that assumption. If a predictive model achieved 90% accuracy in development, add a sanity check to test subsequent models for at least some reasonable bound (85% for example) and also the same model when run on production data. The two instances can differ substantially and without warning, and knowing when that happens is important.

As a general rule, I'm a fan of automating things like documentation. Generating machine-readable comments takes hardly any more time than generating good human-readable comments. With suitable comments in place, your source files can be automatically parsed for utility functions and their behavior. From that, a skeleton wiki can be produced. With a little text analytics, you can even come up with a hierarchical model for how those utilities could be organized. Then, generating documentation is no more complicated for your developers than submitting their code to version control in the first place. If anything in that document seems out of place the small changes needed can be identified and performed manually, but at the end of the day you are left with a searchable document (set of documents) that corresponds with an appropriate project structure, and your developers can spend their time doing what they do best -- developing. The Sphinx project is commonly used for this in Python. It doesn't have support for auto-generating docs from source out of the box in other languages, but it does support parsing rst files into web documentation, and the language-specific tooling to generate rst files doesn't take a ton of time. If you wanted to automatically add the docs to osTicket or another platform instead of create a full-fledged web page for the project, that would be fine too. Automation is your friend though.

1

Look what github has done to solve this problem:

Every utility/repository

  • has a readme.md and
  • a list of keywords/tag and
  • a wiki
  • stored in the repository and
  • visible via a http-server
  • that can be searched by a searchengine.

you can find a utility/repository through

  • a search engine
  • through combining the keywords (i.e. Android + exif)

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.