Need Code Review by sheilatruong96 · Pull Request #9 · blics18/SendGrid

sheilatruong96 · 2017-11-18T03:24:53Z

These are the things we need code review with:

server.go
1. Added crossCheck within the checkBF endpoint.
2. Changed the global bloom filter variable to type struct.
server_test.go
1. Added in TestUnsubClear to test the clearBF endpoint.
client.go
1. Changed Populate() to grab data from MySQL and call the populateBF endpoint to populate the bloom filter.
runClient.go
1. Able to test the functionality of the bloom filter by calling functions inside of client.go.
database.go
1. Contains functions that generate random data and populate MySQL.
README
1. Removed "Install" statements because we're using govend to vendor Go package dependencies

We changed the file structure as well (where each file goes) and we were wondering if this is fine?

jimmyjames85 · 2017-11-20T21:07:04Z

-## How to test:
-
-  1. Change into directory of server_test.go
-  2. In the terminal, type "go test"


I tried running this and it prints error to the console several times, but than it passes which confused me at first. I thought it didn't work, probably because I don't have the same mysql setup as you. Maybe don't print error? or maybe it is passing but shouldn't be

% go test Unable to make connection Error from Database ConnectioneFeOnJkMqw@aol.com is not in the database Unable to make connection Error from Database ConnectionhzSfMqs@aol.com is not in the database PASS ok github.com/blics18/SendGrid/server 0.117s

Also I would consider writing and committing a createTables.sql file that generates the schema.

jimmyjames85 · 2017-11-20T21:11:06Z

-
-func createTable(numTable int, db *sql.DB) error {
-	stmt := fmt.Sprintf(`CREATE TABLE IF NOT EXISTS User%02d (
-		id int NOT NULL AUTO_INCREMENT,


instead of creating the tables inside your service, you might want to take advantage of docker which you could configure to spin up a mysql instance and then populate it every time... e.g. https://github.com/jimmyjames85/bpmonitor

jimmyjames85 · 2017-11-20T21:18:39Z

-
-	p := MakeRandomUsers(numUsers, numEmails)
-
-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")


Instead of hardcoding the username and password, consider passing them in as an environment variable that could retrieve via os.Getenv("DB_HOST") or often we use https://github.com/kelseyhightower/envconfig

jimmyjames85 · 2017-11-20T21:29:09Z

-}
-
-func crossCheck(UserID *int, Email string) bool {
-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")


Here db is a connection pool. To gain the benefits of having a pool it is best to keep this open all the time...for the lifetime of the app. You could than access it as part of the server. e.g.

func (bf *boomFilter) crossCheck(UserID *int, Email string) bool { rows, err := bf.db.Query(stmt) ..... }

jimmyjames85 · 2017-11-20T21:29:25Z

-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")
-	if err != nil {
-		fmt.Println("Failed to get handle")
-		db.Close()


return from the function

jimmyjames85 · 2017-11-20T21:29:32Z

-	err = db.Ping()
-	if err != nil {
-		fmt.Println("Unable to make connection")
-		db.Close()


return from the function and bubble up the error

jimmyjames85 · 2017-11-20T21:37:40Z

-	}
-}
-
-func crossCheck(UserID *int, Email string) bool {


What if UserID is nil. We should check for that...

Would we have to check for this still? Because in check before UserID gets passed to crossCheck, we already checked if it is nil

jimmyjames85 · 2017-11-20T21:42:16Z

-
-	const numTables int = 5
-
-	stmt := fmt.Sprintf("SELECT uid, email FROM User%02d WHERE uid=%d AND email='%s'", (*UserID)%numTables, *UserID, Email)


This can get dangerous, trying interpolate parameters with Sprintf. To prevent sql injection you could do something like this

stmt := fmt.Sprintf("SELECT uid, email FROM User02 WHERE uid=? AND email=?") rows, err := db.Query(stmt, *UserID, Email)

Now to dynamically determine the table name User02 based on the userid, I think you will have to use Sprintf. However you have control of the UserID and numTables which are both int. You do not have control over email that may include a single quote ' which can ruin entire query.

https://www.calhoun.io/what-is-sql-injection-and-how-do-i-avoid-it-in-go/

https://xkcd.com/327/

jimmyjames85 · 2017-11-20T21:42:35Z

-		db.Close()
-	}
-
-	const numTables int = 5


Would be nice if this was configurable instead of hard coded...

jimmyjames85 · 2017-11-20T21:43:21Z

-}
-
-func main() {
-	port := ":8082"


Would be nice if this were configurable instead of hardcoded.

jimmyjames85 · 2017-11-20T21:59:32Z

-	filter *bloom.BloomFilter
-}
-
-func createBloomFilter() *bloomFilter {


typically this would be an exported function and instead of create it is idiomatic to use new

type BloomFilter struct { filter *bloom.BloomFilter // this still is "private" or not exported } func NewBloomFilter() *BloomFilter{ ... }

jimmyjames85 · 2017-11-20T22:04:41Z

-	}
-}
-
-func crossCheck(UserID *int, Email string) bool {


This function needs to bubble up the errors it encounters. Consider changing this functions return signature

func crossCheck(UserID *int, Email string) (bool, error) {...}

This way if there is an error (e.g. with the database connection) then this would return an error, instead of false which may or may not be the case.

jimmyjames85 · 2017-11-20T22:07:06Z

-	rows, err := db.Query(stmt)
-	if err != nil {
-		fmt.Printf("Error from Database Connection")
-		return false


Returning false says the email is not in the database. But we don't know at this point. All we know is there was an error with communicating with the database

jimmyjames85 · 2017-11-20T22:07:45Z

-	const numTables int = 5
-
-	stmt := fmt.Sprintf("SELECT uid, email FROM User%02d WHERE uid=%d AND email='%s'", (*UserID)%numTables, *UserID, Email)
-	rows, err := db.Query(stmt)


consider using db.QueryRow()

jimmyjames85 · 2017-11-20T22:11:57Z

-		return false
-	}
-
-	return rows.Next()


Eventually we need to call rows.Close() in order to return the connection back to the db conn pool. Take a look at pg 6-11 The_Ultimate_Guide_To_Building_Database-Driven_Apps_with_Go.pdf It's a quick read and it explains what is going behind the scenes pretty well.

In any case consider

ret := rows.Next() rows.Close() return ret

jimmyjames85 · 2017-11-20T22:20:29Z

-	fmt.Println("Starting Client")
-
-	// populate the MySQL Database
-	db := client.PopulateDB()


The flow of this seems odd. It looks like client.PopulateDB creates a database object and doesn't close it, but we close it here. I think we should have another function func NewDB() (*sql.DB, error) that creates the db conn pool. Then if there is no error, you can pass db variable to PopulateDB, and DropTables functions.

jimmyjames85 · 2017-11-20T22:20:59Z

-	defer db.Close()
-
-	// populate the Bloom Filter from values in the MySQL Database
-	client.Populate()


This returns an error. We should check it.

jimmyjames85 · 2017-11-20T22:21:32Z

-	client.Populate()
-
-	// to check if values are in the Bloom Filter. Note: Remember to replace the value of b and the userID in Check to what they are in the MySQL DB.
-	// b := []string{"NmNTsOQJOl@aol.com"}


dead code ?

jimmyjames85 · 2017-11-20T22:24:05Z

-	return db
-}
-
-func DropTables(db *sql.DB) error {


This is kinda scary...Why do we want to do this? The database is the only source of truth we have. For our use case, the tables hold the unsubscribe emails, and we def don't want to remove them.

jimmyjames85 · 2017-11-20T22:25:47Z

-const numTables int = 5
-
-func createTable(numTable int, db *sql.DB) error {
-	stmt := fmt.Sprintf(`CREATE TABLE IF NOT EXISTS User%02d (


The table names are User01, User02, ... which makes it seem like we only have 5 users. I think we want them to be Unsub01, Unsub02, ...

jimmyjames85 · 2017-11-20T22:29:05Z

-
-	defer stmtHandle.Close()
-
-	for i := 0; i < len(usr.Email); i++ {


you could also use range https://gobyexample.com/range

jimmyjames85 · 2017-11-20T22:30:42Z

-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")
-	if err != nil {
-		fmt.Printf("Failed to get handle\n")
-		db.Close()


If there was an err on sql.Open, we are not guaranteed that db is not nil. I.e. db.Close will also error, and possibly it might be a nil pointer err. Instead we should bubble up the error and exit the function.

jimmyjames85 · 2017-11-20T22:34:48Z

-	err = db.Ping()
-	if err != nil {
-		fmt.Println(err)
-		db.Close()


Here db.Close is the right thing todo, however we should bubble up the error and exit the function. Otherwise on line 75 insertToTables will attempt to connect to a closed connection.

jimmyjames85 · 2017-11-20T22:35:33Z

-		err := createTable(i, db)
-		if err != nil {
-			fmt.Println(err)
-			db.Close()


bubble up err, and exit function... also consider defer db.Close(), immediately after creating the db successfully

jimmyjames85 · 2017-11-20T22:36:59Z

-	body, err := ioutil.ReadAll(r.Body)
-	if err != nil {
-		w.WriteHeader(http.StatusBadRequest)
-		w.Write([]byte("Could not read the body of the request"))


return from function

jimmyjames85 · 2017-11-20T22:37:44Z

-
-func (bf *bloomFilter) populateBF(w http.ResponseWriter, r *http.Request) {
-	if r.Body == nil {
-		w.Write([]byte("Could not read the body of the request"))


this will register as http.StatusOK. You need to write the header you want here, and if it is an error case, I would return from the function

jimmyjames85 · 2017-11-20T22:38:53Z

-	body, err := ioutil.ReadAll(r.Body)
-	if err != nil {
-		w.WriteHeader(http.StatusBadRequest)
-		w.Write([]byte("Could not read the body of the request"))


return from function

jimmyjames85 · 2017-11-20T22:52:14Z

-	stmt := fmt.Sprintf("SELECT TABLE_NAME AS tableName FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA='UserStructs'")
-	rows, err := db.Query(stmt)
-	if err != nil {
-		fmt.Println("Error from Database Connection")


bubble up error

jimmyjames85 · 2017-11-20T22:52:21Z

-	err = db.Ping()
-	if err != nil {
-		fmt.Println(err)
-		db.Close()


bubble up error

jimmyjames85 · 2017-11-20T22:52:29Z

-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")
-	if err != nil {
-		fmt.Printf("Failed to get handle\n")
-		db.Close()


bubble up error

jimmyjames85 · 2017-11-20T22:52:56Z

-		stmt := fmt.Sprintf("SELECT uid, email FROM UserStructs.%s", tableName)
-		rows, err := db.Query(stmt)
-		if err != nil {
-			fmt.Printf("Error from Database Connection")


bubble up error

jimmyjames85 · 2017-11-20T22:53:59Z

-			} else {
-				userMap[&id] = []string{email}
-			}
-		}


need to call rows.Close to return the connection back to the db conn pool.

jimmyjames85 · 2017-11-20T22:57:52Z

-	Email  []string
-}
-
-func Check(userID int, emails []string) error {


Where is this function being called? Also should it return a list of users that are in the bf?

jimmyjames85 · 2017-11-20T22:58:27Z

-	return nil
-}
-
-func Clear() error {


Again where is this being called?

jimmyjames85 · 2017-11-20T22:59:23Z

-}
-
-func Populate() error {
-	db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")


consider passing in the credentials as arguments instead of hard coding them

sheilatruong96 added 9 commits November 17, 2017 19:10

Delete vendor.yml

254033e

Delete runClient.go

5460252

Delete README.md

3e73523

Delete LICENSE

82bf4b8

Delete server.go

476a092

Delete server_test.go

d9b6892

Delete client.go

b3feae1

Delete database.go

f8e7cc5

Delete generateEmail.go

de52109

jimmyjames85 reviewed Nov 20, 2017

View reviewed changes

Comment thread client/client.go

err = db.Ping()

if err != nil {

fmt.Println(err)

db.Close()

jimmyjames85 Nov 20, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bubble up error

jimmyjames85 reviewed Nov 20, 2017

View reviewed changes

Comment thread client/client.go

return nil

}

func Clear() error {

jimmyjames85 Nov 20, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again where is this being called?

jimmyjames85 reviewed Nov 20, 2017

View reviewed changes


		p := MakeRandomUsers(numUsers, numEmails)

		db, err := sql.Open("mysql", "root:SendGrid@tcp(localhost:3306)/UserStructs")


		const numTables int = 5

		stmt := fmt.Sprintf("SELECT uid, email FROM User%02d WHERE uid=%d AND email='%s'", (UserID)%numTables, UserID, Email)


		defer stmtHandle.Close()

		for i := 0; i < len(usr.Email); i++ {

Conversation

sheilatruong96 commented Nov 18, 2017 • edited by dppham1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmyjames85 Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sheilatruong96 commented Nov 18, 2017 •

edited by dppham1

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading

jimmyjames85 Nov 20, 2017 •

edited

Loading