Approach for keeping track of all a company's scripts/products/data

Question

Sorry that this is a general question, but I believe this is a good resource to ask this sort of advice.

I would consider myself a junior developer. Most of my coding career has been working for a small company (5-10 people) where we develop a lot of GIS products. We have hundreds of products, scripts, and data sources.

We mostly just know where everything is when we need it based on memory. However, as time has gone on, sometimes we forget and it takes forever to find which script makes which product, or which scripts use a specific data source if that data source gets updated.

I'm just wondering what the general convention is in the software development world to keep track of everything. Is it common to have a manifest of sorts on a local machine that carries short descriptions of each script, or is there some beloved software out there to handle this issue? Any advice would be greatly appreciated!

Answer 1

Its difficult to know your exact situation but I think the answer here is that people automate the running of the builds (products) and encode the connections between what might usually be a database/api/website but in your case is products, scripts and data. You need a kind of dev-ops for GIS

You should definitely use source control, but take it a step further and add some sort of xml or json document which holds the meta data linking the product, script and data.

publish your data and scripts to an internal git repo, ftp or web site so you can reference it by url

maybe something like:

"product A" : [
{
   "script" : "git://internalgithub/script1" 
   "data"  : "ftp://internalftp/dataQ"
},
{
   "script" : "git://internalgithub/script99" 
   "data"  : "ftp://internalftp/dataZ"
}
]

The have a build script which takes these files and "creates" the product?

If you can automate whatever the tasks you do with these products then you can collect those automation steps together and publish them on a website, add searches, tree diagrams or whatever.

Each step you automate reduces the number of things to remember and a central place to publish them allows you to store your things where you can find and reference them

Answer 2

This is part of configuration management. It’s a discipline to control the state of a system from its hardware to its software and how it interacts with other systems. It maintains a library to document what was used to build the system. How it works. And what changes have been made to it. In some shops it’s a title and a full time job.

Such a library would include every line of code regardless of language and every piece of 3rd party software used to create the system. Also the requirements, the design documents, the code reviews…

A configuration managers job is to ensure all such things exist, are organized, and retrievable on demand.