Continue my Gowhere journey. I need to read data from a CSV file—a common task in programming, dealing with file.

A sample content with 3 columns ID, Effort, and Title.

ID,Effort,Title
"94503","4","Bug: The line was not aligned correctly"
"97018","12","Implement a cool feature"
"97595","1","Document an incident"

The file size is small so I do not have to worry too much about the performance at this point. Let’s see what requires to perform the task in Go.

In general, here are steps to read and parse a CSV file

1. Open the file
2. Read everything or line by line
3. Parse the text into desired outcome format
4. Close the file. When to close depends on how you read the file

Open a file

Go supplies a built-in package "os" to work with the file system. Opening a file will return a pointer to the file—if no error, otherwise an error, a normal pattern in Go.

There are 3 required steps

1. Open the file by os.Open(file)
2. Check for error before usage
3. Close the file by using a defer function. This is important otherwise the file is locked

    // Need the os package to read open a file
    // f is a pointer to File object (*File)
    f, err := os.Open(file)
    // Check for error before usage
    if err != nil {
        writeLog(err)
        return nil
    }
    // Close file by a defer function
    defer func() {
        er := f.Close()
        if er != nil {
            writeLog(er)
        }
    }()

Read file

Depending on the file size and the type of applications, developers can choose either read all contents at once or line by line. I always prefer line by line.

    // This is a cool concept. Given that we have a pointer to the file opened by the os.
    // A scanner is created and scan line by line
    b := bufio.NewScanner(f)

    stats := make(map[string]int64)
    for b.Scan() {
        line := b.Text()
        parts := strings.Split(line, ",")
        effortText := strings.Trim(parts[1], "\"")
        if effortText == "" {
            continue
        }

        effort, err := strconv.ParseInt(effortText, 0, 0)
        if err != nil {
            writeLog(err)
            continue
        }
        id := strings.Trim(parts[0], "\"")
        stats[id] = effort
        fmt.Printf("%s - %d\n", id, effort)
    }

Go supplies bufio package to help manipulating IO (file). The first time I heard about the Scanner concept. It hits my mind immediately, kind of "wow that makes sense and cool".

After holding a point to a Scanner:

1. Call Scan() to loop throw the buffer data
2. Call Text() to access the current line. If the file is opened in binary mode, use a different method
3. Parse the line to meet your logic

Proceed data

For common operations on a string, strings package is there. For conversion from a string to various data types—int, float, …—strconv package is there. They are all built-in packages.

Close the file

Handle by the defer function, which is called when the caller function exists.

    // Close file by a defer function
    defer func() {
        er := f.Close()
        if er != nil {
            writeLog(er)
        }
    }()

Gowhere? Some steps further—append arrays, hash table, regular expression, and for each loop.

After playing around with HTTP, I got the JSON data from the API. I wanted to analyze the data—to display the total hours spent on each work item.

A typical record has this structure

type WorkItem struct {
    FromDateTime time.Time
    ToDateTime   time.Time
    Detail       string
}

And an actual work item looks like

{
    "FromDateTime" : "2019-11-29 02:05:00 +0000 UTC",
    "ToDateTime": "2019-11-29 03:19:00 +0000 UTC",
    "Detail" : "Work Item 12345: Implement a cool feature"
}

The "Work Item 12345" has many records. The detail field are not the same except they contain the number 12345 as the work item ID. I want to display the sum of time spent for Work Item 12345. So the algorithm is pretty simple

1. For each record, extract the work item id from the detail field
2. Calculate the difference in hours between FromDateTime and ToDateTime
3. Sum the difference with the existing value—if not exist, create a new one with the time spent zero

Note: If I am writing in C#, I can finish the implementation quickly with Linq support.

The expected result of a work item is below

{
    "WorkItemId" : "12345",
    "WorkItemName" : "Work Item 12345: Implement a cool feature",
    "TimeSpent" : time_spent_in_hours
}

Stats structure to hold the analysis result of a work item

// Stats ...
type Stats struct {
    WorkItemId    string
    WorkItemName  string
    TimeSpent float64
}

Let’s write some code and explore

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "regexp"
    "time"
)

func main() {
    // Assuming that I have a list of IDs for a team.
    // The getWorkItemRecords will an array of WorkItem of a member.
    var records = make([]WorkItem, 0)
    for _, id := range team {
        r := getWorkItemRecords(id)
        // Discussion about appending 2 slices: https://stackoverflow.com/questions/16248241/concatenate-two-slices-in-go

        // This is how to append 2 arrays
        records = append(records, r...)
    }

    // A dictionary (hash table) with
    // key: WorkItemID (or name if cannot find the ID) - simply a string
    // value: total time spent
    // More detail about map here: https://blog.golang.org/go-maps-in-action
    statsMap := make(map[string]Stats)

    // Regular expression to extract ID (all numeric characters)
    workItemIdExp := regexp.MustCompile("[\\d]+")

    var id string

    for _, r := range records {
        timeSpent := r.ToDateTime.Sub(r.FromDateTime).Hours()
        if timeSpent < 0 {
            // The record does not have an end time
            continue
        }

        id = workItemIdExp.FindString(r.Detail)

        if id == "" {
            id = r.Detail
        }

        ts, exist := statsMap[id]

        if !exist {
            ts = Stats{id, r.Detail, 0}
        }
        ts.TimeSpent += timeSpent
        statsMap[id] = ts
    }

    var workingHours float64 = 0
    for key, value := range statsMap {
        workingHours += value.TimeSpent
        fmt.Printf("%s (%s) %f\n", key, value.WorkItemName, value.TimeSpent)
    }

    fmt.Printf("Working hours: %f\n", workingHours)
}

What are my further steps from this exercise?

1. Append 2 arrays (slices) with "…" syntax—append(records, r...)
2. Hash table (dictionary like) with map—map[string]int means a dictionary with key is a string and value is an integer
3. Regular expression with regexp package—regexp.MustCompile("[\\d]+")
4. For each loop with the range—for _, r := range records

A happy weekend!