FIX: Go Runtime : goroutine stack exceeds 1000000000-byte limit fatal error stack overflow
I was writing a CLI program to process some data. I need to call the same function again and again until I finish processing all data records. The number of records is 40,572,219 which is so big.
I wrote the function like this.
func dbAutoProcessData(db *sql.DB, inId int) {
rows, err := db.Query(`SELECT * FROM tbl WHERE info LIKE '%something%' AND id > ? LIMIT 1;`, inId)
if err != nil {
log.Fatal(err)
}
// add it manually before re-calling the func
//defer rows.Close()
var id, uid int
var info string
for rows.Next() {
if err := rows.Scan(&id, &uid, &info); err != nil {
log.Fatal(err)
}
}
// some data processing
updatedInfo := info
_, err = db.Exec(`UPDATE tbl SET info = ? WHERE id = ?;`, updatedInfo, id)
if err != nil {
log.Fatal(err)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
rows.Close() // close rows first
// re-run the same function
dbAutoProcessData(db, id)
}
I ran this program for hours. I saw it stopped with this error.
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020512380 stack=[0xc020512000, 0xc040512000]
fatal error: stack overflow
with more stack trace for more information.
Basically, I was calling the same function from inside itself in a recursive way (recursion). And the stack is filled with the previous function signatures and it overflowed. To avoid exceeding maximum stack size of goroutine, we need to rewrite the code in iterative way.
Fortunately, we do not need to call it recursively. it can be written iteratively in an infinite for loop like this.
var id int = 1
for id > 0 {
id = dbAutoProcessData(db, id)
time.Sleep(5 * time.Millisecond)
}
I created a for loop. I call the function which gets the id and run then return a new id. Then sleeps for 5 ms. Then the loop re-run with the new id value. If the id is bigger than zero, the record is in db. If the id is 0, so there is no match record and the loop should stop.
The function dbAutoProcessData() after refactoring is like this.
func dbAutoProcessData(db *sql.DB, inId int) int {
rows, err := db.Query(`SELECT * FROM tbl WHERE info LIKE '%something%' AND id > ? LIMIT 1;`, inId)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var id, uid int
var info string
for rows.Next() {
if err := rows.Scan(&id, &uid, &info); err != nil {
log.Fatal(err)
}
}
// some data processing
updatedInfo := info
_, err = db.Exec(`UPDATE tbl SET info = ? WHERE id = ?;`, updatedInfo, id)
if err != nil {
log.Fatal(err)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
return id
}
The changes in brief are:
- the function return the last processed record id
- use conventional
deferstatement. no need to put the code manually at the end of function
I faced this error and just documented me fixing it in this post. I hope this post helps you with your programming journey. If you want to get notified about new posts, follow me on YouTube , Twitter (x) , LinkedIn , Facebook , and GitHub .
