0

we have a legacy project, on Windows. It goes on from 1990-s.

Until very recently it was not backed by any version control, today it is moved into Git.

The question is all the prior snapshots. As of now, they are spread on network shared folders, with folder names usually suggesting the part of the project, the date or product version, and often the specific programmer, from whose development machine it was copied.

Needless ot say, it takes much of disk space, and mostly on many copies of exactly same files, and today when Covid-19 makes many of us work from home - that is becoming a problem.

Those legacy sources are rarely needed, but sometimes one just to look for a specific identifier (variable, function, file, etc) or even a statement - to find when and why it was introduced.

Copying them onto my home machine would be slow and would occupy too much of my local disk. Same for other developers. Even if i would Zip (or PPMd) files on server - it would be still big blob to copy over RDP, and i would not be able to search for the text in that archive without speed impact, or without unpacking it first - which will take a huge hit on local disk space on every developer machine.

I can not move them into existing Git repository - when i settled for my own convenicenceit i did not even knew about those archives, less so had access to them.

Anyway, Git is not a solution here. Git (or other DVCS) will do data de-duplication and will greatly decrease disk space required, indeed. But Git mantains a "one true current snapshot" model, which is useful for development, but not for "industrial archeology" (yes, i can settle several repositories and switch them to different branches - but that defeats the idea). I just can not run the text search through ALL the snapshots stored inside Git repo, within all their branches and revisions/commits, with the same ease i can run grep (or a similar GUI tool) over thousands of mostly plain text files, even on a shared network folder.

Additionally, i do not have information to recover the "real" scheme of all the "branches" that were happenning with people i never met. While guessing timestamps of every folder is doable even if time consuming, inventing some formally correct relationship of snapshots like "from John" and "from Mary" is not.

Additionally, putting files into Git will loose their timestamps metadata. While it is not needed much for the sources that were in DVCS from get go, for pre-VCS sources that has a lot of meaning. Even for mere guessing which modules were alive and which were derelicts that people just could not practically delete away without VCS.

So, how people do manage situations like this?

I think i need some system that:

  • runs on Windows
  • deduplicates files
  • maintains original paths, names and timestamps
  • has useful and easy to learn FTS without restricting me to one "current" snapshot out of many stored
  • preferably FTS to be available as usual "Search text in files" with no practical performance hit when compared to just storing files and folders "as is" (so, Zip archive is not an option, and it has no deduplicaiton too), so zero learning curve and freedom to use any grep-like and diff-like tools of choice one might like.
  • does not require formal relationships from one snapshot to another, though optional timeline relations would be good bonus. It would be okay to have no concept of snapshots at all - it would be no worse than what it is today.
  • can be easily and efficiently copied over the relatively slow network (so, if there is some natively de-duplicating filesystem for Windows Servers - it would not help much, it would be problematic to copy the data over network without re-duplication and there will probably be no such filesystem on non-Server development machines)
  • low maintenance burden
  • preferably detects and fixes or at least warns about random data degradation events (bit flips on HDD/SSD, etc).
  • no need to modify data after initial storing it, actually it better be strictly read-only

I think hypothetically it can mostly be implemented on top of NTFS 5 symbolic links, and perhaps someone had already done it?

Or maybe i just fail to think out of the box and see a different workflow, easy to learn and use.

  • Your real problem is finding out what is what - only when you know that you can think about how to store them in any organized way. – Jan Doggen May 15 at 16:16
  • @JanDoggen it is semi-organised now, by folder paths and names. Making more formal (strict and regular) organization would be problematic. And maybe not always needed, too. Non-precise give-or-take few months information is usually succint. – Arioch May 15 at 17:25
  • 1
    You've got a pretty sizable wishlist and don't seem willing to go for partial solutions. Maybe you painted yourself into a corner and need to reconsider some requirements which may be contradictory in practice. As they stand, the requirements look very specific to your special case, and it's very unlikely that someone has already a solution for it that you can just use. – Hans-Martin Mosner May 16 at 5:57
  • @Hans-MartinMosner they are specific of course to my personal idea of comfort, but they seem (in my eyes of course) as pretty natural for a case of middle-size inherited project and itssemi-organized snapshots. I can't believe this our situaiton is unique. Afterall, Git itself was born due to much more specific set of requirements and a much harder problem. – Arioch May 16 at 13:50
1

I would recommend using GIT anyway.

  • not because its fast.
  • not because it makes life easier
  • not because you can make an actual timeline

but because that data will rot in the file system as more and more commits are made to the git repo, and fewer and fewer people are even aware of these snapshots.

If you need fast text searches, then you are going to need them on git too. That's a separate issue. Look into directly contributing to git itself to stabilise/improve the functionality, or find a way to make your own through another solution - there are more than a few full-text search databases out there.

Data degredation should primarily be the responsibility of the hardware SAN you are running on. If you don't have a SAN buy a copy of unraid, or setup freenas. They will also provide suitable wrappers to host the git server, and search engine you want.

Meta-data is tricky, its not necessarily reliable. Both due to how much the file system actually stores, and that so many OS level operations will update it. If there is any meta-data you need preserved it is best to directly encode this into the path, or into a separate meta-data file.

All the other requirements are trivially handled by git itself.

| improve this answer | |
0

The question is all the prior snapshots.

git grep pattern $(git rev-list d70e..HEAD)

Introduction to git grep

runs on Windows

Git for Windows

If that's not enough I humbly suggest you need an enterprise search solution.

| improve this answer | |
  • 1
    Git Grep has problems outlined in comments at stackoverflow.com/a/2929502/976391 - performance problems with scaling, problems with non-Linux OSes, problems with overflowing command line limits. And specifically for our situation, it will impose a learning curve for bash-jitsu and git-jitsu, will need much more boilerplate actions for loading two files into WinMerge or other GUI files comparator with syntax highlighting, for quick peeping into adjacent files in the same "commit", and will need making up strict directed-graph relations between arbitrary folders. – Arioch May 15 at 17:41
  • unix.stackexchange.com/a/493750/185008 git rev-list --all | xargs git grep "excited too" ..... is very, very slow on the Unix history repository – Arioch May 15 at 17:45
  • Enterprise search systems have another set of problems. They do not STORE they only retireve and index, so will help nothing about the disk size. They also seem to be hardly integrated into the "digging up source" workflow too, how can i quickly launch WinMerge over a subfolder in two different snapshots? They overshoot in "variety of sources" and underperform in the specific needs... – Arioch May 15 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.