This is a text-only version of the following page on https://raymii.org:
---
Title : Get serial port data on the web with live updating
Author : Remy van Elst
Date : 28-09-2019
URL : https://raymii.org/s/blog/Get_serial_port_data_on_the_web_with_live_updating.html
Format : Markdown/HTML
---
Recently I was asked to build a test setup to measure the accuracy of a few
loadcells (weight sensors) in a frame. Someone other than me would execute the
test and required access to the measured data, preferably live and without
needing knowledge of ssh or linux. The weight sensors can be read with an
openscale board from sparkfun via a serial connection which the board emulates
over USB (via an FTDI chip), via that same serial connection you can also
calibrate them or change settings like amount of decimals. As a quick solution I
created a webpage that uses a small amount of javascript to live update the
readings from the board. screen is used to capture the output of the serial
connection to a file which the webpage displays and updates. In this article
I'll talk about how I put various bits of software together to get the output of
the serial connection onto a nice webpage with live updating.
![the hardware setup][1]
The picture above shows the hardware setup. The blurred part is the actual
hardware part with the load cells in it. You can see one cell on the left. The
red boards are the Sparkfun Openscale boards, connected via a USB hub to a
[Variscite VAR-SOM-MX6][3] board running Yocto Linux (openembedded). I built a
webserver package for the board (nginx) since normally it runs nothing except
the main application.
The choice for this linux board was just because that was available. It could
just as well be a regular PC or Raspberry Pi, anything that can run screen and a
webserver.
The tests themselves will be done by someone else, who can navigate to a webpage
to check the current readings. Below is a screenshot of the webpage:
![the webpage][2]
<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days. </a><br><br> </p>
If the test results are good enough, those load cells might be used in the
future, but not with the sparkfun openscale boards. The cells will then just be
connected to the regular machine and we'll write our own software to use the
sensors.
### Hardware setup
The hardware I used consisted of common off the shelf parts:
- [5 Sparkfun OpenScale][4] boards
- [5 Double-ended beam load cells][5]
- 6 port USB hub
- Linux board to connect it all up (could be a Raspberry Pi)
Connecting it all is not a hard job either:
- Screw in the load cells on the desired places.
- Wire them into the screw terminal on the sparkfun board
- Connect the sparkfun boards to the USB hub
- Connect the USB hub to the linux computer
I didn't require a powered USB hub since the linux board itself gave enough
power but your results can vary. Try less sensors or a powered USB hub if the
serial connection is flaky.
### Software setup
This is where stuff gets interesting. The goal of this setup was to be easy for
a non linux user to use. Someone who will test the accuracy of the sensors, what
happens when you do X, how does the setup respons when Y happens, etc. That
person is not interested in the software side of things, they just require the
measurements.
The time I got allocated to build this was not much, that's why this setup is so
simple. With more time, I might have added graphs, log archiving or a piece of
custom software to read the sensors without the sparkfun board. But, it had to
be done quickly, which required me to think about what tools were at hand for
this job.
The essential thing the setup needed to do was to provide log data in an easily
accessible format. During testing you don't want to click five times or type
commands, you just want to test. The setup should provide that.
The sparkfun openscale boards barf their output right on the (emulated) serial
port. What if we could present that data on a webpage so that the tester just
needed a laptop or tablet during their testing. What if we could make that data
live update using modern fancy web techniques? (Remember I'm doing embedded C,
not javascript).
#### screen
Screen is a linux terminal multiplexer. You can have one terminal and use screen
to get more. Handy when you're doing work via SSH, you can reconnect to the
screen session and continue on if for whatever reason the connection dropped. Or
you can run your IRC client, it won't disconnect when you log off.
As a byproduct of earlier times you can also connect to a serial port. Handy for
sending `AT` commands to your modem or connecting to your actual physical
terminal. We're going to use that to connect to the USB serial device in the
sparkfun board.
Screen supports logging of the session to a text file and allows you to insert
timestamps in that log. Perfect for what we want. That log can then be read out
via the webpage we're creating later on.
Start screen with the following commandline options:
screen -dmS sensor1 -L -Logfile sensor1.txt
/dev/serial/by-id/usb-FTDI_FT231X_USB_UART_XXXX
Replace the actual serial device by the path to your device. The sparkfun board
all have a unique serial number, so I can match them to the actual sensors.
- `-dmS`: start screen in detached mode and name this session `sensor1`
- `-L`: log the output to a file
- `-Logfile`: change the filename of the log from the default to `sensor1.txt`
The output of the sensor will now be in text file `sensor1.txt`. If you use
`tail -f sensor1.txt` you'll notice that it only updates every 10 seconds,
whereas the openscale board outputs every 2 seconds. Luck has it that screen
allows you to set the flush rate of the log. You can either use a `.screenrc`
file, or send it to the running session as a screen command. My compiled version
of screen has some quirks due to it being an older version, so I cannot use the
`.screenrc` route. Using the `-X` option I can send a command to a running
screen session.
Note that this is not a shell command that will be executed, it is a screen
command. One you could also give inside the running screen session by pressing
`CTRL+A :` (colon).
The [command to set the flush rate][6] is: `logfile flush seconds`, replacing
seconds by the number of seconds.
To send it to a running screen session, in my case, `sensor1`, use the following
command:
screen -S sensor1 -X logfile flush 2
No quotes around the options. If you `tail` again, you'll see the file being
updated every 2 seconds.
I put a construction like this in a shell script for all 5 sensors, the script
is autostarting at boot of the linux machine via `systemd`. That means that all
sensors will be logged to a file named `sensorX.txt` where X is the sensor
number.
With the logging of the serial port data set up we can continue on to create the
webpage.
#### Fancy pancy javascript webpage
As you might have guessed from the section heading, I don't code javascript
often. My piece of cake is C++ and C for small embedded platforms. If the code
looks simple to you then that's the reason. No frameworks, no `nodejs`, heck, if
I could get away with a `<meta http-equiv="refresh" content="2">` I'd use that.
But, modern day users expect more than that, so why not.
The code you'll see below uses bootstrap css for styling. Hosted bootstrap that
is, just because I'm lazy.
The javascript part is split up into two parts. One is to actually do the XHR
(ajax) request on a file and update the DOM (replace the current contents of an
html element with the new content) and one is to reload the five files using
that first function. The automagic reloading is set with a timer via a
`setInterval`.
The first function to reload the data:
function reloadData(url, html_id)
{
var req = new XMLHttpRequest();
var now = new Date();
req.open("GET", url + "?" + now.getTime(), false);
req.onreadystatechange = function ()
{
if(req.readyState === 4)
{
if(req.status === 200 || req.status == 0)
{
dataDiv = document.getElementById(html_id);
dataDiv.innerHTML = req.responseText.split("\n").reverse().slice(0, 1000).join("\n");
}
}
}
req.send(null);
}
The function accepts an URL and a HTML element ID to replace. A timestamp is
appended to every request so that it is not cached by the browser. If the
request fails, nothing happens. The `.onreadystatechange` contains the function
we want to execute every time the readyState attribute changes of the request.
There is no filtering or escaping of contents, the html attribute is blatantly
replaced.
In the status files the data is set with the newest lines on the bottom. For the
webpage, I want those on top, since you're not scrolling when doing the testing.
You just want to look. Therefore, the file line order is reversed with this bit:
.split("\n").reverse().slice(0, 1000).join("\n");
It also limits the log to at max 1000 entries. Not a great seperation of
concerns, but for this stage, workable.
The second part is a wrapper function to call the `reloadData` on the five files
we want:
function reloadFiles()
{
reloadData("sensor1.txt", "sensor1");
reloadData("sensor2.txt", "sensor2");
reloadData("sensor3.txt", "sensor3");
reloadData("sensor4.txt", "sensor4");
reloadData("sensor5.txt", "sensor5");
}
The last part is this:
setInterval(reloadFiles, 1000);
Which will execute the wrapper function every second.
The HTML part of the page contains some styling and columns for a header, the
actual data is put in four of these blocks:
<div class="col-md-auto">
<h2>Sensor 1</h2>
<pre>
<div id="sensor1" align="center">
<p>Loading Sensor 1 Data...</p>
</div>
</pre>
</div>
The text `Loading Sensor 1 Data...` will disappear the first time the function succeeds.
Bare-bones javascript that gets the job done. Maybe not the best, but this whole
setup was put together in two hours and that includes wiring up all the
hardware, which took most of the time.
[1]: /s/inc/img/logging_setup.jpg
[2]: /s/inc/img/logging_webpage.png
[3]: /s/inc/img/var-som-mx6.png
[4]: https://www.sparkfun.com/products/13261
[5]: https://en.wikipedia.org/wiki/Load_cell
[6]: https://www.gnu.org/software/screen/manual/html_node/Log.html
---
License:
All the text on this website is free as in freedom unless stated otherwise.
This means you can use it in any way you want, you can copy it, change it
the way you like and republish it, as long as you release the (modified)
content under the same license to give others the same freedoms you've got
and place my name and a link to this site with the article as source.
This site uses Google Analytics for statistics and Google Adwords for
advertisements. You are tracked and Google knows everything about you.
Use an adblocker like ublock-origin if you don't want it.
All the code on this website is licensed under the GNU GPL v3 license
unless already licensed under a license which does not allows this form
of licensing or if another license is stated on that page / in that software:
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Just to be clear, the information on this website is for meant for educational
purposes and you use it at your own risk. I do not take responsibility if you
screw something up. Use common sense, do not 'rm -rf /' as root for example.
If you have any questions then do not hesitate to contact me.
See https://raymii.org/s/static/About.html for details.