At home I use Linux and have for more than a decade. At work we use Windows and so I find PowerShell more familiar. I am a Linux/Unix fanboy at heart, but I've had greater success grokking the PowerShell pipeline. The model of pipelining objects is useful because when it comes right down to it, in data processing we are often processing entities (things that have key/value pairs, i.e. attributes).

Clearly this is something Unix has been doing forever. I suspect tools like sed and awk may play a part, but they're less intuitive to me. In Unix you're piping raw text through the stream. If you wanted to pipe operations over your personal address book you have to first consider how conducive the format of the incoming data is to pipeline processing. I suspect that a record would appear on a single line. I don't know if there's a community-favored data format/protocol for writing records that are easily piped between Unix programs. Obviously, the format dictates what programs you use and how you use them (e.g. how you filter, map, and mapcat).

In PowerShell:

$input | ? { $_["contact_type"] -eq "Professional" } ;filtering
$input | % { $_["state_code"].ToUpperCase() }        ;mapping
$input | % { $_["phone_numbers"] -split ";" }        ;mapcatting
$input | % { $_["contact_type"] = "Friend"  }        ;mutating an attribute

;or when preferring to avoid mutation of the source data
$input | % { $h = $_.Clone(); $h["Surname"] = $_["Surname"].ToLower(); $h} 

I'm sure I could spend the day figuring out how I might accomplish this in Unix, but I am interested in something that is idiomatic and equal to the PowerShell legibility. Choosing the right data format (CSV, XML, etc.) impacts the legibility of our pipeline logic. It would feel awkward if the logic kept the line-of-text abstraction intact. I would prefer some abstraction where the logic brings greater focus to the keys and values (i.e. the feel of "objects") than to the lines of text.

In Unix, if you could store your record data in any format what format would you pick in order to facilitate clear and concise pipeline processing? Bash counterparts for the above examples would be appreciated.


When bash and core utils are not enough, I turn to Perl. You can store your data in a text file, in XML, CSV, real database, or you have several serialisation modules available.

Once your contacts are in an array, you can easily apply all the operations (I don't know PowerShell, so they might not be exactly equivalent):

my @professionals = grep 'Professional' eq $_->{contanct_type}, @contacts;
my @upper         = map uc $_->{state_code}, @contacts;
my @numbers       = map { split /;/, $_->{phone_numbers} } @contacts;
$_->{contact_type} = 'Friend' for @contacts;

The same applies to Python or Ruby.

  • 1
    Any time I've got a shell script longer than 10 lines, I look to switching to perl (or python or ruby if you're one of those people). Typically that comes with a simplification of the data and a performance boost (because you're not launching processes all over the place to do specific tasks and shoving the data in a pipe). – user40980 Apr 1 '15 at 15:42
  • Hmm. If the sentiment is to not use bash then I completely understand since I could write very clean Ruby code; however, the shell is supposed to be tightly intertwined with the console programs available on the OS. When you move out of shell and into a scripting language you're in different territory. You're not piping a la Unix. – Mario T. Lanza Apr 1 '15 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.