Shell Script Design Patteren: Source a library file VS Call different files?

Question

We discuss about POSIX compliant shell script here.

While we are writing more and more shell scripts, we build some helper functions to reuse the codes.

We are considering putting a few helper functions (around 10 functions of the same category) into a library file (e.g. string_lib.sh) or splitting each functions into individual script files (e.g. str_replace, str_cmp, etc.) to be called directly.

Due to the potential problem of variable name collision, we lean towards splitting each functions into individual script files. But there will be more than a hundred of script files, which is hard to manage.

We understand in some cases, we do not have a choice to choose between these two.

• If we need to pass (several) variables from functions back to the caller, we must use a library file and sources it, as they need to share the same variable scope. (Using text file to pass variable is not preferred in most cases.)

• If we would just like to avoid one script file being too long, split it into different sections and source them sequentially. In this case they usually need some shared variables (e.g. setting variables).

In other cases, these would be the pros and cons comparison.

Grouping helper functions into library files:

• (o) The library file will be sourced (by . string_lib.sh).
• (+) Same category of functions written in same file. Easy to manage.
• (-) Possible of variable name collision.
• (-) Need to avoid double inclusion, or mutual recursive inclusion which will cause infinite loop.
• (-) Too many inclusion may slower the script. If only a few functions are used within a library file, other function definitions are redundant.

Split functions into individual script files:

• (o) The script files will be called directly, just like Linux programs.
• (+) No need to source (i.e. .) before use. Just put all the scripts into the $PATH directory.
• (+) Unnecessary functions are not sourced. Possibly faster loading.
• (+) Called script is in a different shell, the sub-shell. No worry of variable name collision. (Similar concept as Java, C, etc.)
• (-) One helper function will have one script file. There will be many script files. This is harder to manage.

There is another SO answer about bash design patterns and best practices. Especially Stefano Borini's Modularization section is inspiring. Its import function is smartly written. Do take a look when you need to source script files. However in that SO post, it did not compare between grouping functions into a library file versus splitting functions into individual files.

https://stackoverflow.com/a/739034/1884546

Are there any guidelines or recommended design patterns of shell scripts?

Answer 1

First piece of advice is to echo Stefano Borini: "don't."

Some languages offer good support for core software engineering concepts without the need of supporting 1970's sh syntax. Prefer python over bash wherever you can, or similar high level languages.

---

Ok, we're writing some shell scripts. Do yourself a favor, and stick to just bash, so that's all you test against and all you use in production. Much simpler than worrying about endless portability details.

Use shellcheck.

Spell it source rather than . dot, since only bash matters to you. It makes telephone conversations about the code go much more smoothly. You will be onboarding new engineers, so you may as well make the documentation easy to read and to pronounce.

source a single utility file, which optionally sources additional library files, as long as they are "popular" and not "too big", too time consuming to frequently parse them.

Take advantage of functions as much as you can, as they are lighter weight than forking off a bash child. Consider adopting some brief distinctive prefix for names you're putting into the global namespace.

---

With for example perl you might opt for use strict;, making the language become more picky in the hope of getting a helpful diagnostic for common errors like variable name typos.

Consider starting each script with:

set -e -u -o pipefail

The -u option will cause echo $typo to error rather than interpolating an empty string for unset variable.

The other two cause "bad status" $? to immediately terminate the script. So for example this sequence won't trash log files in current directory:

set -e

cd /some/where/nonexistant/typo

rm -r *log*

because failed cd causes exit without attempting to run rm.

Consider setting those three options at top of a central utility script so that application scripts don't have to remember to do it. (This would be Breaking Behavior for any scripts you already have that rely on the old behavior.)