5

I have a workflow wherein I walk recursively through a bunch of directories, then for each file perform some action. I'm using goroutines to walk each directory, and also to process each file. The problem I have is keeping track of when the work is "finished" as each directory spawns a new goroutine and there is an unknown number of subdirectories and files.

The solution I'm using at the moment involves creating a secondary channel to act as a counter - before spawning a new goroutine we +1 the counter, and when the goroutine completes it -1's the counter, and when the counter reaches 0 we know we're done. It basically does this:

func scanFile(filePath os.FileInfo, result chan string, counter chan int) {
    result <- "result of operation"
    counter <- -1
}

func scanDir(dir string, result chan string, counter chan int) {
    for _, f := range (/*files in dir*/) {
        counter <- 1
        if f.IsDir() {
            go scanFile(f, result, counter)
        } else {
            go calcCrc32(f, result, counter)
        }
    }
    counter <- -1
}

func main() {
    results := make(chan string)
    counter := make(chan int, 1) // need a buffered channel due to next line
    counter <- 1
    go scanDir("some dir", results, counter)

    countValue := 0
    for {
        select {
        case result := <- results:
            fmt.Printf("got result")
        case delta := <- counter:
            countValue += delta
            if countValue == 0 {
                fmt.Println("all done")
                return
            }
        }
    }
}

This seems to work pretty well, but it feels like an absolutely terrible hack. Is there a design pattern/idiom in go that would achieve this in a better fashion?

P.S. I think I could simplify this a lot by not using goroutines for the directory scanning, and only use them at the file level, but the question still holds - how does one manage the lifetimes of "recursive" goroutines?

1

2 Answers 2

3

From the docs for sync.WaitGroup:

A WaitGroup waits for a collection of goroutines to finish. The main goroutine calls Add to set the number of goroutines to wait for. Then each of the goroutines runs and calls Done when finished. At the same time, Wait can be used to block until all goroutines have finished.

1

I have, in the past, used something like this in Go, for this exact case:

func ProcessFile(fileName string, wg *sync.WaitGroup) {
   defer wg.Done()
   // Do things
}

func ProcessDirectory(dirName string, wg *sync.WaitGroup) {
  defer wg.Done()
  // Do things
  switch {
  case isFile(name): 
    wg.Add(1)
    go processFile(name, wg)
  case isDir(name):
    wg.Add(1)
    go processDirectory(name, wg)
}

It can be implemented using a state-keeping go-routine to count up and down, but a sync.WaitGroup is a much simpler solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.