Create and Run Go service in Docker

Create Go service the easy way, For Windows, Linux, MacOS and Docker.

Why would you want to create a service? 🔗

With Go programming language you can use the final executable the same way on Windows, Linux and MacOS (by generating proper executable). But you cannot install this executable as a service using standard system tools like sc.exe, systemd or launchd. By using this approach you will know, how to make your software executable installable with those system tools and make it run as a service, when system starts.

As a bonus, this approach also works for Docker containers.

Prepare the project 🔗

I assume you have some knowledge of Go language, and programming in general. But you can use this article/post as an easy preview what — and how — can be done with Go programming language.

Let’s get it started.

Start up Goland and create a new project using Go modules. Name the project first_service and hit Create.

In the project window, create new Go File and name it main.go. Then select type of Simple application.

You should see a file containing…

Start writing the code Add two lines below package main, that contains information about this service, we are creating. Service name and it’s description.

No we will add an external module, that allows us to run this program as a service. Add another line of code, just below the package main. This import line will have (for now) a red squiggle line below itself. This means, that this module is not imported.

Import the module Place your cursor on this red squiggly line and hit ALT+ENTER (or OPTION+ENTER if you are a mac user). Select Sync dependencies … Goland will download this module from internet and combine it with your code.

But that red squiggle line doesn’t dissapeared. Why is that?

The answer is simple. We imported the module, but we are not using it now. The red squiggle line tells us this exact information: we are not using this imported module. So let’s use it.

Use the module Add new line of code type program struct{}, just above the main function. This line of code contains a structure. Structure that holds your service. That red squiggly line is still there.

To get rid of it, we need to add some code into that main() function. Code, that will almost create and start you service. Code below.

Explain the module I think, we need to pause here and go through the code.

Line starting with serviceConfig adds your name and description to the service. When you run this service in the future you will know what’s the name of the service in Windows, Linux or MacOS service properties.

You can add an information about a version, for example like this: const serviceName = “Medium service, version 2021.1.2.

Goland will automatically add new import line for fmt as this is internal Go package, not an external one.

Lines below will almost create — service.New()— and — s.Run() — your service.

Implement module methods You see, we have two new red squiggle lines. The first one is in s, err := service.New(prg, serviceConfig) and the second one in err = s.Run(). To get rid of them, place your cursor in that prg word, hit ALT+ENTER and select Implement missing methods.

Goland will create those two missing methods. Code below is what you’ll see. Two new methods: Start() and Stop().

Update the code in those two methods with code below, so we can see — when the service is running — what the service is doing. Those lines fmt.Println() will print on the screen, that your service has started and/or ended. But there is another red squiggly line under go p.run().

Adding production code Again: put your cursor on that go p.run() line, hit ALT+ENTER and choose Create method run. This will create a new method, that will hold our production code.

To make a simple production code simulation, update that method, so it looks like code below. This code will print an information, that your service is running, every second. When you add that time.Sleep() method, Goland will automatically import time package for you.

Test the code We arrived at the end. Your code should look like below. Now press your right mouse button in that main.go file and select Run go build main.go. Goland will run your code.

When you start your code, let it run for a while and then stop it (by pressing red stop square button). You’ll see something like this.

Medium service started Service is running Service is running Service is running Service is running Service is running Medium service stoppedProcess finished with exit code 0 Congratulations.

Your service is starting, running and stopping the right way. And as a bonus you can install this small program as a service under your system of choice.

But, for now, it will still stop immediately. To make it stop after production code is done — after that 1 second simulation work is done — please check improving the service article.

If you want to know up-front, how to create an executable, jump directly into this article.

Summary By using this approach you know how to…

work with GoLand create service, that runs on Windows, Linux and MacOS create service, that you can install on all those three systems create service, that is prepared to run in Docker container

What does it mean “the proper way”? You made your Go program to run as a service and there is a production code that takes about 5 seconds to finish all the work. You don’t want to stop the service in the middle of the work, don’t you?

This article expands the code we made before, so the service will stop only after this production code is done.

That means: whatever is going on in your production code won’t be scrambled, when someone stops your service. Of course, there are many ways, how to do this. This here is my style of doing things.

Update the code As we like to know the state of the service in every instance of time, when code is running, we will add two boolean variables, that show us their statuses, for example using fmt.Println().

Add variables Those two variables we will call serviceIsRunning and programIsRunning. We start with adding those two variables just below the import section. There will add third variable of type sync.Mutex. This variable takes responsibility of synchronisation, when writing or reading to/from those two boolean variables (when two lines of code want to update the variable serviceIsRunning in the same exact time, mutex allows only one line of code to make the change and then invite the second line of code to make its change).

In this particular example we won’t be writing to one variable from two different sections of code, so this sync.Mutex is not necessary now, but we will use it.

As you see, all three variables are underlined with red. The reason is again the same: we are not using those variables anywhere.

Update the methods At first, we start with serviceIsRunning.

Update the Start() method with code below. Initially, the variable serviceIsRunning is set to false. After the service started, we set it to true. Before setting, we lock this serviceIsRunning variable with writingSync.Lock(), so any other future line in the program (that wants to write to the same variable) has to wait until this writingSync unlocks back our serviceIsRunning variable. Now, when our service starts, our variable serviceIsRunning is set properly.

We do something similar with the Stop() method. No more talking, the code is below and the idea is the same.

Now it’s time to play with the programIsRunning variable. We move to the run() method. At first, we want to make sure, that our production code will run only if the service is already running. See code below, nothing special. Adding one condition: the production code runs only when variable serviceIsRunning is true.

You can now test your code. It will work the same as before, so it looks like we make no progress, but believe me, we do make a progress.

Update production code To see the effect in work, update the time.Sleep() method, so it will wait for 2 second (you can try 5 second to see the effect better).

We play again with that mutex variable and, for a first time, with the programIsRunning variable. Code below shows the changes.

This code takes about two seconds to finish. In the beginning, we set programIsRunning to true (so everywhere in our code we can check if this program is running). And after this loop is done, we set this variable programIsRunning again to false.

After all this is done, you can check if the program runs. And it runs, but… still no effect.

Final fine tuning Let’s get back to our Stop() method. Update the code with a new loop. This new loop simply waits for the programIsRunning variable to be false (that means: it waits for the production code to finish itself).

And only then the loop will end the Stop() method.

Final test Run the code now (if you see no effect, increase the time.Sleep() in run() method to 5 seconds).

Hopefully you’ll see the effect. When you try to stop the program (simulating someone stops the service), the program waits for the production code to end properly, and only after that, the service will shut itself down.

For example: if you increase that sleeping in run() method to 10 second, you will see much more Medium service stopping … lines. Of course, that depends on when you stop the program.

Summary After reading this article you know, how to …

make your service run and end properly know, in every part of the code, what is the state of the service

Why should you use Go as a web server? That is a valid question. We have Apache, Nginx, IIS and bunch of other web servers. You can write web server in Java, Node, PHP or whatever else.

So why use Go?

Because creating Go web server is simple. Because Go web server is fast. Because you can run your web server a a single executable. Because you can run it on all three major systems the same way. Because you can run it easily in Docker. And because you can use one technology— Go — for almost everything, so you don’t need to learn anything else.

That is why you should use Go as a web server. And this article will teach you, how to do it.

Prepare our project The goal of today’s article is to improve the service to make it a simple web-server service. Service, that will serve CSS, Javascript and HMTL files (and of course you can add fonts, data and other files, as you like).

Create directories Begin with creating three new directories(folders) in the main project directory. You can see the result on the left side of the screenshot: three new directories: css, html and js.

Add new line to your import block: github.com/julienschmidt/httprouter. And again, press ALT+ENTER and import this module.

Side note: you can use Go’s built-in routing and/or you can use many others routers. I tend to use julienschmidt’s httprouter for three reasons: 1) it is simple to use 2) it is really fast 3) another part has a SSE functionality, which I use a lot (more on that in another article)

Updating the code in main.go After importing the module we will focus on that run() function. Please delete all the code that you have here from before, and insert two new lines.

First line will create router instance (variable) and the second line will start this web server, running as service, on port 81. You can now test running from Goland, just to make sure, everything is working properly.

You will notice, that serviceIsRunning variable has red underline, because we are not using it now. But we will.

Side note: you can use almost any other number for port. If you omit the number like this http.ListenAndServe(“:”, router), or even like this http.ListenAndServe(“”, router), the web server will start on default port 80

To have a better information about starting your webserver service, update the run() function to display an error message, if any.

Here is a screenshot to show you, what you should see, if a port is in use. So, in my code I need to use different port. 82, for example.

To serve files to user, we need to add two other lines. One line is for serving css files from that css directory we created. The second one is serving javascript files.

Maybe you are asking, why we are not adding another line for serving html files. And of course, you can add this line in here, so you can the access all html files in a browser, like http://localhost:81/homepage.html . But we will use different approach, that allows us to modify that html file, before we send it to the browser.

Adding new files Now create three files: homepage.css, homepage.js and homepage.html. And create them in proper directories. Also create fourth file homepage.go in the main project folder, like you see on the screenshot.

Side note: I prefer this approach (more files, same name, proper directories), because when looking for bugs on a specific page (like homepage.html), I can immediately go for all “partner” files, homepage.go, homepage.css and homepage.js files in this case. And I do not care about any other files, because “by name” they are for another page.

Update homepage.html file, so it looks like below. Add two lines, that imports css and javascript from proper directories, change the title, add two new h1 headings and one button, so we can then test, that everything is working.

Update homepage.css file, just to be sure that the page loads it. Of course, you can play with css like you want. In this example I made it simple like this:

Update homepage.js file, again, to be sure, that it will be loaded and it will work.

Code below will grab for a button with id test-button and link it with a function that will display alert message. This function listens (hence the addEventListener) for a type click. So when you click, an alert with text Button clicked appears. Plain and simple.

Update main.go file, so finally we can serve that homepage.html to the browser. Update the run() function with new line. To make it simple, is says: when you open http://localhost:81 in the browser, it will call the serveHomepage function.

Side note: you can play with this and make it for example like this router.GET( “/homepage/”, displayPage). Of course, now you have to open http://localhost:81/homepage to call a function displayPage

Serving html file to user Because that serveHomepage is underlined with red, we need to create that function. Need to mention, this function does not have a () at the end. Navigate to that serveHomepage, press ALT+ENTER and select Create function… This will create empty function and it will create it in main.go. Please move it to homepage.go, so that file will look like this.

Catching the error Now try run the service and I think you will see an error. This error says that serveHomepage function is undefined. In other words, the code doesn’t know, where to find it.

The reason for this error is simple. By default, when you are starting your program using Goland, it will start the project in the file mode. We need to change it to directory mode, or, for better to package mode. This can be done by clicking on go build main.go in the upper left in Goland.

Here, select Edit Configurations.

Change to Package and click OK. Now, when you run the program, everything should be working. But we are not finished yet, that homepage.html is still not serving to the browser.

Completing the code Final work is made by editing serveHomepage() function in homepage.go. At first, we will add back our programIsRunning variable, see code below.

Then we will add a new structure. This structure contains just one string, that will hold generated time. Time in this file will be linked with that {{.Time}} in that homepage.html file.

Update the serveHomepage() function with new variable and change that property Time to actual time (as string): those are the two lines in the middle.

Then add two new lines. First line parses the homepage.html file. Second line loads that homepage HomePagestructure and updates Time field.

Side note: you need to use capital T, otherwise it will not work

Because {{.Time}} is present in that html file, it will replace it with time.Now().String(), in this case, when parsing that homepage.html.

You see few more red squiggly lines, so place cursor onto template.Must, press ALT+ENTER, and import html/template.

Side note: we use {{.Time}} as an example here, but you can chose variable name as you want and parse program version, database size, … everything you can imagine. You can even parse data for tables.

To be sure you have everything in place, this is the final homepage.go.

Final test Now you can run your web-server service. Run it and open browser on http://localhost:81 .

You should see something like on screenshot below (notice I need to run it on port 82).

Congratulations. You have you web server running as a service. It is successfully serving proper css and javascript files. It properly parses html file and add time, that was generated in the backend in go.

And of course, the button is working too.

Nice work.

One more thing To make sure that your web server doesn’t accidentally stop when in the process of serving file to your browser, you can simulate this by adding time.Sleep() method to your code. Then restart your program, open the browser and stop the program. You need to wait for about 10 seconds — simulating work — before the page will be served to your browser.

And as you see in Goland, the program will wait for this, and only then it will stop itself.

Summary Now you know how to …

create a web server and run it as a service run this web server on a specific port (and easily change it) serve different types of files (html, css, js, …) add information to html file, before it is send to browser

part: creating the service part: improving the service part: upgrade for web part: you are reading it right now part: simple javascript frontend functionality part: frontend — backend communication part: server-side logging part: run it all in docker part: adding database container part: database — service communication Addition 1: functions, methods, pointers and interfaces Addition 2: websocket communication Addition 3: socket communication

What does it mean, SSE? Why should I use it? SSE stands for Server Sent Events, standardised push technology, that is very easy to use.

If you want to update something on a web page, you can use SSE. Be it sending a time, table, colour, notification badge, …

In this article, we will use a simple example, where you’ll see the power of SSE.

Prepare the project Start with adding the right module, that has this functionality already built-in on the server side of program.

We will use julienschmidt’s repository again, this time it is module from github.com/julienschmidt/sse. So add the new module and import it. Your import block of code should look like code below.

Adding SSE functionality Then navigate to run() method and add a new line of code just under your router := httprouter.New(). This line of code will create a new streamer called timer. Ignore the red squiggly line as we will immediately use that streamer.

Implement SSE on server side Add another two lines of code: router.Handler(“GET”, “/time”, timer) and go streamTime(timer), so finally your run() method should look like the code below. The first new line will register a handle for this streamer and the second line will run new goroutine that will stream time, until the service is stopped.

Create that streamTime() function with ALT+ENTER and move it to your homepage.go file. Update this function with code below. This piece of code will stream actual time, every millisecond, for as long, as your service is running.

Implement SSE on client side Now we will add a new element to your homepage.html file, that will display that streamed time. It is the element with id actual-time: .

And finally add few new lines to your homepage.js file. Those few lines access that /time handle we defined in router.Handler(“GET”, “/time”, timer). And every time — in this case every millisecond — it will update that html element with actual time.

Test, how fast can SSE work Now you can test it, but don’t forget to remove that time.Sleep() method from your serveHomepage() function. You should see your SSE working like a charm. Time is refreshed every millisecond.

Now you see the power of SSE.

Attention! But please, this is just as an example, how to send data from backend to frontend. Do not use SSE for sending time every millisecond. See screenshot below, how it looks like, when measuring transferred amount of data for this every millisecond streamed time.

Summary SSE technology streams data only one way. From server to client. And it is not wise to use it for showing millisecond time on your webpage, you can do this in javascript completely on client side.

I specifically used sending time to show you, how much data and how quickly can be transferred from server to client, but in real applications, I use it for notifications and updating data of every kind.

To summarise it, after reading this article, you know how to …

implement SSE on server side in Go implement SSE in client side in HTML and Javascript what are SSE advantages (fast and simple to use) what are SSE disadvantages (server to client only) Real — life example To give you an example, this is, how I am using SSE.

This is a screenshot from big LCD panel, hanging in one production hall in some factory. SSE is used for updating almost every piece of data on this screenshot. Colours, texts, percentages and — in this case — even for time (because customer wanted to have that time synchronised on every panel).

Neat feature Very neat feature of using SSE is something I call “continuity”. When you stop the service, that is sending all the data using SSE, but leave the page opened, you immediately see, that the data on the page are not refreshing.

In a case of this LCD panel, you see no new updates… but the page is still on. And in our example with time… the time stops running. But when you start the service again, you see the time is running again. Check it for yourself. And in case of LCD panel, it will automatically refresh with new data.

P.S.: You can see the full code for that LCD panel here( https://github.com/petrjahoda/display_webservice ), but that service is just a part of a project, so ignore everything that is not about SSE :-)

Why use vanilla Javascript for your frontend? Because vanilla Javascript is the frontend language. Of course you can use Typescript, Dart and other technologies, but Javascript is the basis.

From a different point of view, you can use Angular, React, Vue and another million frameworks. But they all run using Javascript under the hood.

By using vanilla Javascript you learn, understand and use the technology as it was designed. There are ton of tutorials and bootcamps over the internet and if you want my recommendation, I find this guide really enough, to start using Javascript right away.

Dive right into updating the code Open your homepage.html file and focus on those three lines, containing tag . Update , and add new line. Those three lines will display three separate informations:

time, when page was generated actual server — side time, using SSE time, when button was pressed (new functionality) Then switch to your homepage.js and change line, that updates actual-time object. We are adding just some text information, so we will immediately know, which time is which, and that it is working.

Adding new Javascript functionality At this point of time, we updated text information on the web page and now we are adding that javascript functionality, as mentioned in the beginning of this article. Add new line, that will access element with id button-time.

And then update function, which — until now — displays alert message. After this change, it will display actual time.

Run the program and you will see something like the screenshot below.

Now we have three separate functionalities working on one simple web page:

page is updated with actual date-time, before it is generated and sent to user — this functionality is using Go templates page is updated with “live” date-time from Go backend, using SSE functionality page is updated with time when button was clicked, using vanilla Javascript And did you notice that the page is not refreshing at all? Just those times, either using SSE or via button?

Fine — tuning the time format But we still have some work to do, because those three date — times have really odd formatting and updating time every millisecond is wasting resources. We will change that now.

Open you homepage.go file and update that sleeping time in streamTime() function, so it will stream time about every second. Also, update line, that is sending actual time. Add a formatting method.

Side-note: Goland has a neat feature for formatting date-time instances. Navigate into that formate function and press m, Goland will show you possibilities for string m.

Then update one line in serveHomepage() function.

Now we have both date-times, that originates in Go formatted properly. Finally we will format date-time for that button clicking. Navigate to homepage.js and update the listener function to look like below.

Side note: I specifically chosed the same formatting to look good and the same, but you can play with Go date-time formatting and you can also play with Javascript date-time formatting as you like. Just to give you an example for different Javascript formatting.

new Date().toLocaleDateString(’en-US’, {year: ’numeric’, month: ‘2-digit’, day: ‘2-digit’}); There are many source for formatting date-time, here are two links. First one for Go, second one for Javascript.

Go: Format a time or date The parameter describes the format of a time value. It should be the magical reference date formatted the same way as… programming.guide

Date.prototype.toLocaleDateString() The toLocaleDateString() method returns a string with a language sensitive representation of the date portion of this… developer.mozilla.org

Time to test it all We arrived at the end of today’s article. Screenshot below is the final result you should see. All three date-time instances are working properly and with the same formatting.

Summary As you can see we don’t have to use any framework yet. No Angular, no React, no nothing else. We are updating data on homepage with plain and simple Javascript. I hope I have convinced you to invest time to learn Javascript and to learn only vanilla Javascript. You can always learn more technologies, but it is wise to use the basis first.

Even with this little in this article, now you know how to …

load javascript file (you knew that before) access an element on the web page add a listener to that element activate that listener change/update that element when listener is activated

Is it hard to program a communication between frontend and backend? I can only answer in case of Go and vanilla Javascript. And in this case the answer is: it is as easy as riding a bike.

If you want to be convinced, keep reading. In few minutes you will see and understand that you can do it by yourself. You will send data from webpage to server and back, without even refreshing the page itself.

Prepare before coding At first, update your homepage.css, so all buttons looks the same.

Then, update homepage.html. Add lines starting with those two lines and ending with heading with id=“ask-time”. From the user point-of-view, we got three new things on the page: a label, an input field and a button. The user cannot see that fourth with id=”ask-time” as it has no text assigned yet.

You should see the look of the webpage on a screenshot below.

Adding the functionality to frontend Now switch to homepage.js and start with adding three new constants. (Before it, get rid of that one-an-only console.log(time) we left here from before.)

One will access input element, second will access new button and third will acces for now invisible element, which will display data from backend.

Then add new event listener onto that askButton, that will update that invisible element, for now just to make sure that everything is working.

Please run the code, so we can see, that everything is working properly, as on an screenshot below. Click the button to check everything is working.

Stay in your homepage.js and update that function, so it looks like below. Go through the code by yourself and then come back here for a little explanation.

Line starting with let data creates a variable, that contains json-style data with two “fields”: Name and Time. I purposely took these two variables, but you can ad more then two. First variable access that input with id=”name”, second variable is actual datime.

Then we are calling fetch() function. This is javascript internal function.

This function calls /get_time (not implemented in backend yet), sets up proper headers, wrap that data and by using POST method is sends everything to backend.

If everything goes well, then() is called. Here we acces the response data, then unwrap JSON data inside and log() it to the console.

If it goes south, we log() the error to the console.

First test Try it now. Open up your console in browser’s developer tools and hopefully you will see two errors. The first one indicates that the request for /get_time ended badly with an error and you see that error: 404 Not Found. This is understandable, as we did not implement nothing in backend yet. Second error shows you, that the javascript cannot unwrap data, as there are none.

Adding the functionality to backend Now we are going to implement that /get_time in the backend. Switch to main.go and add new line that contains definition of that new method: router.POST(“/get_time”, getTime). Again, press ALT+ENTER on that getTime() function and create it. Then move that created function to homepage.go file.

Before going back to javascript, we will work only in this homepage.go file. To access that JSON structure we are sending from javascript frontend, we need to make a new structure with exactly the same names. Structure that’ll hold that javascript frontend’s data. We’ll name this structure TimeDataInput with two fields: Name and Time (exactly the same names as in data variable in homepage.js)

Update that getTime() function with code below. First we need to create a variable of type TimeDataInput and then we read the body of the request and unwrap its content into this variable. If there is a problem, we will print this problem onto the screen. If not, we will print those data fields, that were received from javascript frontend.

Second test Try to run the code now, open the browser, add here your name and press that new button. In Goland console you’ll see your name and time of your request.

Medium service started Streaming time started Petr 27/12/2020, 08:56:50 Implementing the response for Javascript Next, create another structure, now for outgoing data (response back to Javascript). You see four fields in this structure: Result, Time, Text and Duration. You understand in a minute, why those four and of course, you can name them as you line and you can add as much as you want.

Handling the error But first, we will handle that error, so user can see, what went bad, in case of error.

Update the code as below. As you can see, this code simply prints information about error to console on the server, then creates new TimeDataOutput variable, set the Result to nok and Text to some information. Then wrap this structure into JSON and sends it back to web page.

We omit handling error, when sending data back to the webpage, by using _, to make the code shorter, but it makes sense to handle this error in real code.

Handle the happy path Then we add code for a success path, called happy path in go terminology. Navigate to those two fmt.Println() function and add code after them. We will simulate some work here for one second: again that time.Sleep() method and then send back proper information about that work. Result is ok in this case, Text is as you like, Time is again actual time formatted properly, and Duration takes advantage of that timer variable. Everything is wrapped into JSON and sent back to the webpage.

Third test Run it now, open up your console in the browser’s developer tools, insert your name, press the button and after a work is done — that one second — you will see the result from backend

Side note: I tend to use timers a lot in my code, because I want to measure and log almost every function, in term of it’s duration. It helps me to debug the code after the code is in the production. In those logs, I can immediately see how long everything took and I can immediately see, when something is wrong = takes too much time (for example some database heavy lifting). So in case of real production code, I will move that timer := time.Now() in the beginning of the function and at the end of the function, just before the return, I will add fmt.Println(“processing takes: “ + time.Since(timer).String()), to see how long does it take to run a specific function, or method.

Finalising the communication Back to our homepage.js file. Add one line, just after that console.log() function.

Now, we are done. We are able to communicate with the backend Go code from our webpage, from the javascript frontend. We can send anything we want and we can receive back another anything we want.

Final test Run the code and try it for yourself. Screenshot below is what you should see.

Real — world usage in production code Note, that in this case, there is no need to refresh the page after user makes his input and presses the button.

I use this functionality for almost every communication I need to: user input, logging user web page behaviour back on the server (will be covered in next article), downloading data for charts, downloading data for tables, simple sending data between users, …

Also, when there are more components on the web page — for example charts — I download data for all components separately. The reason is simple: let’s say there are four charts on the web page and one chart has somewhat complicated data. That means that processing those data takes not negligible time. From the user’s point-of-view, three charts will appear immediately and that fourth chart takes time. The user will — of course — report this “problem”, but his report will be much more specific. The user will tell you that one specific chart loads slowly and the user will tell you which one. You then check it in your logs (remember using that timer before) and then you can focus on that one and only function and make it faster (for example cache some data in case of database loading, or something like that), so that fourth chart will load in proper time.

Summary I hope those few minutes with this article helped you understand, how can you add a simple communication technique between Javascript frontend and — int his case — with Go backend.

From previous article you know how to initiate some action from backend using SSE, now you know, how to initiate some action from frontend.

As usual, we finalise this article with a little summary, I am convinced that now you know, how to…

send data/information from Javascript frontend send those data on demand (click button for example) send data from user input respond back from Go backend, with another data/information do all this without whole webpage refresh

Why to write your own logging? There are ton of loggers in the Go world. Fast, easy, complicated, slow, … you name it.

The reason, why there are so manys is simple: everyone one of us is different and every one of us have different demands. So every one of us see logs as something specific for him, because I, You, Him, Her,… will be those one’s who will go through those logs in case something went wrong. And you want those logs to be the way you want. So it makes sense to make logging your personal thing.

But I advise you: try bunch of them.

Zap, Logrus, … , search and try. You can even use Go’s built — in log() functionality.

This article will show you, how — and why — you can built your own logging functionality. Functionality, that will act the same on Windows, Linux and MacOS and even when running in Docker.

Jump right in with creating log.go file In the root of the project, create new file, called log.go. In that file create a new function called logInfo(), code below. This function does nothing more now, than prints some information to the console.

To add logging to file, add a new function appendDataToLog(), then create this function using ALT+ENTER.

Add a huge block of code below, that will add functionality and handle all open-write-close file errors. You can go through the code, but there is nothing special here.

Because we want to write into log directory, if you add some logging now and try it, it will log an error to the console, just like his:

2020–12–29 08:40:39.072 [MAIN] — ERR — Cannot open file: open log/MAIN 2020–12–29.log: no such file or directory How to log into proper directory We will add a simple check for this log directory to the beginning of or program, just to make sure, that this directory always exists, when our production code is running.

Navigate to main.go, to the beginning of the run() method, and add here a new function called logDirectoryCheck(). Create this function and move it to the log.go, just to have all about logging in one file. Because Windows and Unix/Bsd based systems handle things differently, and we want to make it work on all systems, there will be more code, than we expect. Code below.

As you can see, this is a horrible, long function, not readable at all. We will make it more readable and we will use another Goland functionality. Select everything starting with var dir string and ending with } just before logDirectory := filepath.Join(dir, “log”). Then navigate to Extract Method, just like on screenshot below (or use a proper shortcut). After that Goland will extract all this code and it is up to you to name the new function. We name it getActualDirectory() as it gets the proper directory, no matter your system.

Then, select the rest of the code, until the end of the function and extract it to a new function called createLogDirectory(). Code below shows you, how your function should look like. You immediately see that this function does (unfortunately) two things, get actual directory and create log directory in it.

You can omit a lot of code, if running only on one specific system, but now you have better understanding, what does it take, to make it work (almost) everywhere.

Test the basic logging Time to test it. Navigate to your run() method in main.go and add a new line logInfo(“RUN”, “Program is running”), just after logDirectoryCheck(). Run the software. You will see two things. New directory log is created and a new file inside it is created. What is neat, by specifying reference you can modify the name of the file.

So you can have separate log files for every function, or for every file (log files for log.go, log files for main.go, …), or … just like you want.

And exactly the same log information is logged into the console. That will be used for Docker logging in the future.

Medium service started 2020–12–29 09:15:00.561 [MAIN] — INF — Log directory created 2020–12–29 09:15:00.561 [RUN] — INF — Program is running Streaming time started Medium service stoppedProcess finished with exit code 0 Now update streamTime(), so it looks like below. Just a simple logging added, to know the function started.

Logging errors Before updating the rest of homepage.go add a new function logError() into log.go, to log errors (to see them immediately in log files).

Side note: you can play with this as you want. For example, I always log error to .err files and ALSO to.log files. Sometimes I log by name of files (main.go is logging to main.log, homepage.go is logging to homepage.log, …), sometimes by function names and sometimes completely different.

Updating the rest code Update serveHomepage() with three new lines. Two are in the beginning of the function, one line at the end. This will log that someone asks for this page and how long does it take to handle the request.

Update getTime() function similar way. Please note, that I removed some not necessary code here (like that simulating of work).

Final test We did it. We have logging from backend and also from frontend.

Don’t you believe? Try clicking that Ask for data button. If you do it,by cascade getTime() function is called as a result and all this is logged to file.

You can now improve the code: create just one simple Go function called logFrontend(), that will just log the data, nothing else. And — in javascript code — add calling this function wherever you want. On every button, on hovering over something… just everywhere you want.

Bonus feature As a bonus, we will implement deleting old log files, to make sure your disc will not run out of space. Navigate to your main.go file and update your run() method: add new function go deleteOldLogFiles(48 * time.Hour), that will run in its own goroutine. Create this function and move it to log.go.

Update this function with code below. This function will run every hour, until service is stopped. It will scan log directory for all files older than those 48 hours and delete them.

It is using time of file’s last modification file.ModTime() and because new files are created for every new day, this last modification time stays the same for older files, because no new data is written to them.

Another bonus feature :-) We will add just another neat trick I use when making a web server, because I want to know, who asked for that page, who asked for that time, who clicked that button, who…

And by who, I mean IP Address of a computer. This is done very easily, see example below.

And this is what you’ll see in the log

2020-12-29 09:41:39.068 MAIN INF Get time function called from 192.168.86.21:62587 Side note: in case you see something like Get time function called from [::1]:57133, this means the function was called from localhost.

You can even use this ip address as a reference when logging data, so your files are named after that ip address.

Summary You now know, how to make your own logger, that can

log to different files change names of those files on — the — fly log using different levels of logging (we did Info and Error) log from backend log from frontend, using the same functionality use the same functionality on all major systems delete old log files automatically

Why use Docker at all? Of course, you can create Go executable for every system and you can run it where-ever you want. That’s true.

By using Docker, you can create a neat deploy system for yourself. And you do not care, what system actually runs on the final machine. You only want the machine to run Docker. You can easily stop and start multiple programs (multiple services if you like this terminology). You can easily change the port for Go web server without changing the Go code itself. You can run multiple different databases at the same time.

You can do a lot using Docker and I advise you to invest time to learn it.

For those of you not familiar with Docker, I recommend Bret Fisher’s Docker Mastery. This helped me to move from “what the hell is Docker” to “great, what else can I use Docker for”.

And for those of you already familiar with Docker, but want to know more, this talk can give you much more understanding about containers at all.

Prepare the project Open Goland and create new file called Dockerfile in your project directory. Insert code below. Those command will create empty image using FROM scratch and then copy content from css, html, js and linux directory.

But wait, there is no linux directory yet. That reason is, we did not build our software. We did not make any executable.

Create Go executable There are numerous ways, how to build our software… how to create this executable. One way is to use Goland’s built-in functionality.

Create a new configuration by copying the first one. And in this new configuration change four things:

Name… this name will be used for the name of the executable and has to match with CMD [“/medium_service”] , so we use medium_service, Goland will automatically add_linux to filename Select output directory (linux as we will use linux containers) Remove tick on Run after build Set environment to GOOS=linux Side-note: by defining GOOS you can build executable for any system and any architecture that is available.

Shrinking the executable by 73% When you run this configuration, you immediately see that a new executable appears in this linux directory. This executable is about 9.6MB and we will do two tricks to make it much more smaller. First add two flags-ldflags=”-s -w” to Go tool arguments, as on screenshot below.

Run this configuration again and you can see, that the executable is about 6.9MB, reduction about 28%. Second trick is to use any executable packers, that are available. I use UPX and I use it also in production with no impact at all. With commandupx medium_service_linux the size of the executable will shrink to about 2.6MB, reduction of 73%.

Create Go executable with script Another way, I prefer to use, is to use a script, be it a bash or powershell script. As I am using apple device, so below you can see my bash script. This script is called create.sh and is placed in the project directory. It does numerous things.

At first it will get name of the current working directory by using name=${PWD##*/}, then it will update all modules (not necessary, but I like to have latest versions of all modules), then it will build our executable using GOOS=linux go build -ldflags=”-s -w” -o linux/”$name”, then move into that linux folder, pack the executable with UPX and move out.

After all this is done, it will finally do something with docker. By using docker rmi -f petrjahoda/”$name”:latest it will remove any previous image already created before, by using docker build -t petrjahoda/”$name”:latest . it will create image. This dot means it will search for Dockerfile in a directory, where this command is executed. And finally it will push this image to docker hub using docker push petrjahoda/”$name”:latest. To make this final command working you need to have your own repository working and you have to be logged in to Docker on your machine.

Side-note: Don’t forget to chmod +x create.sh and remove that previous executable with _linux at the end.

Running the script When you run this script you see numerous things going on (see below). After everything is done, you can find your image in your docker hub repository. In my case it is here.

go: golang.org/x/sys upgrade => v0.0.0–20201223074533–0d417f636930 Ultimate Packer for eXecutables Copyright © 1996–2020 UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020File size Ratio Format Name — — — — — — — — — — — — — — — — — — — — — — — — - 6901760 -> 2630120 38.11% linux/amd64 medium_servicePacked 1 file. Error: No such image: petrjahoda/medium_service:latest [+] Building 0.4s (8/8) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 135B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build context 0.2s => => transferring context: 5.26MB 0.1s => [1/4] COPY /css /css 0.0s => [2/4] COPY /html html 0.0s => [3/4] COPY /js js 0.0s => [4/4] COPY /linux / 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:1df9b9adbc1167cb1fcb1fc66d74cf33964c35530dc04fa4f5434e917bc50a53 0.0s => => naming to docker.io/petrjahoda/medium_service:latest 0.0s The push refers to repository [docker.io/petrjahoda/medium_service] bc5cf960a3a7: Pushed 6eec7c2c219c: Pushed 2038e7a8e302: Pushed 95541ba72fa1: Pushed latest: digest: sha256:4584ba804f99d001a57d84bc85b6624eb8f9ab2b8edc9e226e43e670a0ce51b3 size: 1149 Test running Go service as Docker container Looks like we did everything right. Time to test it.

Open up your terminal, or console, or use Goland’s built-in terminal and run docker run — name medium_service -p 90:81 petrjahoda/medium_service:latest. This will run a new container with name medium_service, mapping internal port 81 to external port 90 and using already created image with latest tag. When you run the command (in my example I had to use that port 82, because I used it in my application) you see logs from your application.

Open up your browser and navigate to http://localhost:90 . Your web page has to be running. If you try to click on Ask for data button, you immediately see, that it was successfully logged.

Make Docker container run in the background To make this docker container running all the time, end it for now using CTRL+C and remove it using docker rm medium_service. Now run it again, this time with added -d parameter: docker run — name medium_service -p 90:81 -d petrjahoda/medium_service:latest. This will run the container and detach from it. So now your console is free to use or close and you have your web server working in the background.

You can check your container working by using docker ps -a that will show all your not removed containers, or by using docker stats (see image below) that will monitor your running containers in realtime. In my case I have more running containers, medium_service is on the top.

CONGRATULATIONS. You have your own web service running in a Docker container. Every part of the software is running as it should and as a bonus, by using FROM scratch we make our container very secure (but on the other side you cannot from example ping FROM INSIDE the container).

By using docker image ls you can see our image is about 5.26MB whole (Go executable and all other files).

Bonus feature Below is my create.sh file I use in production for every software. There are two different things. At first you see, it runs ./update and at the end there are three more docker commands.

The reason for this is, that I like to have a correct version of my software and do not have to think about it. So in every software in main.go I have a constant const version = “2020.4.3.14” and when the service is starting, i like to use something like logInfo(“MAIN”, serviceName+” [“+version+”] starting…”).

This ./update does two things. Update this constant and update those last three lines of code in create.sh, containing version.

The result is, that the latest docker image is always really the latest and there are more images as my software is built with proper version number. For example, version 2020.4.3.13 is combined like this:

2020 is year 4 is fourth quarter of the year 3 is third month in this quarter 13 is the day of the month You can see using this approach for example here. Also that update executable is in the project directory.

Summary This article helped you to know, how to …

create Go executable using two different methods create Docker image and push it into Docker Hub run Docker image with different parameters run Docker image in the background (and check logs) shrink Go executable by ~ 73% add some automated versioning as bonus feature

Is it a good idea, to run database in Docker? From the developer point of view… is is the best thing you can do. You can run multiple different databases on multiple ports so you can very easily develop your software against all of them. Running MySQL, MariaDB, PostgreSQL and SQL Server all at once? No problem.

Also… have you ever tried to uninstall SQL Server? I don’t know how about you, but on my machine, there was always something left. Using Docker you can very easily run, start, restart, stop and delete database as you like, and be sure that nothing was left.

From the user point of view… why not. And by this I mean using Docker in production. Our company is using MySQL and PostgreSQL in Docker for about three years on about 50 different machines, without any problems. Databases are huge, communication is huge and everything is working without problems on Linux and Windows machines.

As a bonus you can make your database secure, so it’s not accessible from outside Docker. More on that in the article.

My recommendation is: when running in production, prefer Linux machines. The reason is, that you have your database data stored directly on the hard drive, but in case of Windows, they are stored inside a virtual machine.

Run PostgreSQL in container This article will help you to add PostgreSQL container, so we can (in the final article) make our web service communicate with database tables. If you have your medium_service container running, stop it with docker stop medium_service and remove it with docker rm medium_service.

Start with docker-compose.yml file Create directory medium in your project directory. In this directory create new file called docker-compose.yml. This file will contain configuration for both containers (service and database), so you don’t have to run each container separately. Update this file as below (don’t forget to change the name in case of your medium_service container, mine uses, of course, my docker hub account petr_jahoda).

Explanation of docker-compose.yml file To make this file readable for you let’s go through it. Code below creates internal network called medium.

Code below creates a persistent volume, called db, that will be mapped to our database data, so our data won’t be erased when restarting, stopping and even removing database container (handy in case of updates).

Code below will run our medium_service in created medium network. Container name is also set to medium_service. This container is created from petrjahoda/medium_service:latest image (use your docker hub image name, or you can of course you this image). Docker can handle logs for you, so technically we can remove logging to file in our program, but we will leave it here for now. Configuration is set up to log max 10 files with size of 10MB. Container depends on medium_database running, it’s internal port 81 is mapped to external port 90, so from outside — from your computer, or from another computer — you access this web service on port 90 (we did this already in previous article). Restart set to always means, that after computer restart or docker restart, this container will start running automatically.

Code below for database container doesn’t have lines that map database port from inside to outside. This means that our database cannot be accessed from outside, only from inside, from another container in the same medium network. In this case only from running medium_service container. This can be a neat security trick, because no one can access our database from outside. If you want to have this container accessible, simply add two lines about port and use something like ‘6543:5432’(Internal postgres port 5432 is mapped to external port 6543). There are another lines, that were not used in medium_service. Two lines are mapping our database data to that created volume db using ‘db:/var/lib/postgresql/data’. And two final lines are setting password for default postgres user. If you forgot to set this password, your container will keep restarting.

Run the docker-compose.yml file (database and web server) Save the file and navigate to already created medium directory. In here, run command docker-compose up -d. Hopefully you will see something like below. Docker will create our network called medium_medium (it uses directory name as a prefix), then volume called medium_db and then creates our database and service container.

Now, if your run docker ps -a , you’ll see two containers running.

Access the database container from another container In second part of this article we will access our database from within our already created and running medium_service. Switch to main.go and after go streamTime(timer) add new go checkDatabase(), then create this function.

Create a new file called database.go in project directory and move checkDatabase() to this file. Finally add two new imports and one constant, so database.go looks like code below.

Side note: as you can see, we are accessing host medium_database as this is the name of our database container.

Then update this checkDatabase() function with code below.

Final test Finally we are going to test everything. docker-compose down your running containers and remove volume using docker volume rm medium_db.

Now we have everything cleaned up.

Update our medium_service container with added code using create.sh. Then navigate into your medium directory and run docker-compose up -d again. It will use this updated medium_service container. To see, if everything is working run docker logs medium_service.

Why is there an error? Hopefully you will see an error, like this one, on my machine. Connection to database is refused.

➜ medium git:(Chapter_9) ✗ docker logs -f medium_service Medium service started 2021–01–02 09:33:47.389 [MAIN] — INF — Log directory created 2021–01–02 09:33:47.390 [RUN] — INF — Program is running 2021–01–02 09:33:47.390 [MAIN] — INF — Streaming time started2021/01/02 09:33:47 /Users/petrjahoda/GolandProjects/medium_service/database.go:11 [error] failed to initialize database, got error failed to connect to host=medium_database user=postgres database=postgres: dial error (dial tcp 172.31.0.2:5432: connect: connection refused) 2021–01–02 09:33:47.392 [MAIN] — ERR — Problem opening database: failed to connect to host=medium_database user=postgres database=postgres: dial error (dial tcp 172.31.0.2:5432: connect: connection refused) The reason for this error is simple: because we are running both containers from scratch, it takes a little time, before default postgres database in medium_database is initiated the first time. And because our medium_service started really fast, before the database was initiated, hence we see that error.

Run it all again To check this again, after a database is initiated, run docker restart medium_service and then go for the logs again. Hopefully you will see that this time the database is available.

2021–01–02 09:38:41.112 [MAIN] — INF — Database available! Because our database is now running and its volume is already set up and filled with data, you can now test, that after restarting both containers, the database will be immediately available. Stop both containers using docker-compose down and then run them again with docker-compose up -d. Then go for the logs and you will see that the database is available at run time.

To finally check everything is running smooth, run docker stats and you’ll see the result:

Summary By now, you have a better understanding, how to …

set up more docker containers in one file run this file using one command access database only from the medium_service container set up the database to make it not accessible from outside make containers start automatically set database container to store data persistently

Access database running in Docker Let’s say you have running two (or more) Docker containers. One of those containers holds running database. And you want to access this database from all of those non-database containers.

This article will show you, how to do it. And it will also show you, how to create database and tables directly from Go code.

Prepare the project Open up your project again and navigate to database.go file. Add a new constant here.

This constant holds connection string to our new medium database (not yet created).

This article contains five major steps:

creating database directly from code creating table directly from code adding simple write operation adding simple read operation display read data on the web page Create database directly from Go code Our function checkDatabase() does only one thing now: it checks, if the PostgreSQL database is accessible or not.

Update this function with code below. At first, the function will check if the main database postgres is available, this stay the same. If it is available, there is a second check for availability — this time for database named medium. If this medium database is not available, it tries to create it. If it is available (already created), it simply log this information.

First test If you now build your medium_service docker image and run docker-compose up -d you will notice that you have got an error first. It is the same error as we ran in the previous article: medium_service simply runs much faster than medium_database initiated its database.

To fix this, simply restart this service using docker restart medium_service and show logs docker logs -f medium_service. Now you can see the result.

Service started Service checked for postgres database… available Service checked for medium database… not available Service created this database If you restart the service and go for logs again, you see, that this time, even this medium database is already created (and it should be). First step is done.

Creating tables directly from Go code We need to start with creating structure for our new table in database.go file. Add a new structure right below already existing two constants. This structure will be used for mapping data between Go and database. We will name this table ButtonRecords and it will have 6colums:

gorm.Model creates four very useful columns (see them at the screenshot in the bonus section) Name creates column of text type Time creates column of timestamp type Now add code, that will handle creating the table and updating the table (if already created).

Updating could be handy, if you have a newer version of your tables with — for example — new columns. This second part of code will automatically add those new columns into your already existing table.

Second test Now docker-compose down your running containers. Then update your medium_service image. Leave that volume medium_db with already created database and run it all again with docker-compose up -d. This time you will see, that both database checks are ok as they should be. And then the code checks for the table and creates it, if not found. Like in this instance.

If you restart you medium_service and check for logs again, this time you will see, that the table already exists, so only update will be processed. So our second step is done.

Adding read and write operations Navigate to your homepage.go file and into getTime() function.

We will add here a little functionality: when user clicks that webpage button, our code will create new record in the already created table. And then reads all records from this table and sends them back to webpage. Then we will show this data in a HTML table.

Update the getTime() function as below. As you can see, we deleted half the previous code and added new code. Because we want to handle the errors first, this new code tries to open the database and in case of error it sends this error information back to the webpage.

Side note: you can use numerous Go modules for database communication. I tried them almost all and end up with GORM, which is easy to use, has readable code, can access all major databases and — my experience — is really fast.

Now we will add this read and write functionality. All errors are now handled, so we can add code to the rest of the function — this is our happy path.

Add two new lines, that creates a new variable with a name record, fill its fields from data, that came from our webpage, when user clicks that button.

And that’s it, writing data to database was easy.

Reading data from database is easy too. Add those three lines. First lines creates variable named records, second line reads all records from table ButtonRecords and third line simply print some useful information.

We also need to update our structure for out — coming data, so we can send those records back. Update the TimeDataOutput structure as below (just one line added).

And then add code for the rest of the function. There is one new line responseData.Records = records. This line add our database records to our response back to the webpage.

Third test And once again… docker-compose down everything, create your medium_service and docker-compose up -d everything back.

Open you browser, fire up the page and check for medium_service logs. Screenshot below is what you should see, after you press the button.

And the more time you press the button, the more records should be read from the database. Just like on screenshot below.

And we also need to check, if everything is going back to the browser. If you have you console opened in the browser, you should see that after a button is clicked and data received back, there is an object with an array of data. Those are the data from the database.

We did step 3 and 4.

Finalising the project We are almost at the end. We will add a final functionality: display data from database in a simple table on the page, after user clicks that button.

Update your homepage.html file with new lines, that will hold those returned data:

Then navigate into homepage.js and add one new line to the beginning of the file: const table = document.getElementById(“table”) so we can access that html table.

Then update your response function with code below. This code will access those Records and add one record after another to our table.

Also add time.Sleep(2*time.Second) to the beginning of your checkDatabase() function so we don’t need to restart the medium_service container again, when we run everything from scratch.

Final test Again, docker-compose down everything, remove volume using docker rm medium_db, create our medium_service again and docker-compose up -d everything up.

Those two seconds are enough for the medium_database to create its base postgres database, so our medium database is created on the very first run.

Hopefully you should see something like below. Every time you press the button, new record is saved — in the background — to the database. Immediately after that, all records are downloaded from database and displayed as a table in the browser.

Summary We arrived at the end and now you know, how to …

write to database running in different docker container read from this database send those data back to the browser and show them to user