Wednesday, February 19, 2020

Temperature adventures with (RaspPI) ESP32 and MQTT.

 Guest speaker today. Glenn has a farm and has contributed before. Farm automation is especially useful for keeping the work down somewhat and getting information. Here is Glenn's latest project:
---------------------------------------------- 
A while back I began looking at convenient, inexpensive temperature measuring devices. Now the reason behind this is very simple. Going out the back door of our place we have what we call the boot room. We live on a ranch, so needless to say we separate our ‘barn’ clothes from our ‘house’ clothes. The barn clothes hang in the boot room. Well before changing into our barn clothes we would like to know what the exterior temperature is. We don’t have an exterior temperature display near the boot room. Soooo… this is where the ESP32 comes in.

I first started this project as a raspberry pi based project. You see I have a raspberry pi 3 B+ in the horse barn and it measures the temperature of both my workshop and the stall area using DS18B20s. Now just about any of the raspberry pi’s from the 2B on up, or the zero would have worked. The raspberry pi zero W would not have worked in my case here as the device is housed inside a Leviton Structured Wiring panel and there is no wifi within the barn. If you are doing this in a location that has Wifi available then the Pi zero W could certainly fit the bill.

I had adapted code from Dave’s other posts to fulfill the need but that code was originally base on Sqlite and wasn’t feeding to my new MQTT docker container. Time for a change but thanks Dave for the original code!!

This project really contains three parts:
1. Temperature measurement
2. MQTT testing and coding
3. ESP32 implementation.

Temperature measurement.


The DS18B20 is probably the best contribution to the temperature measurement in the history of IoT and home automation. I’m not going to go into depth on this as Dave has several posting on it and they are a wealth of knowledge. These are connected to the raspberry pi as shown in this drawing:
Note: I am only using two Sensors at the moment not the three shown.
Full disclosure here. I am a big fan of Python and Python modules. Have been for a long time. So, when I start a new project I go on a search to see what new modules are out there. I’m an engineer, not a developer. I’m not the guy that writes a lot of code day in and day out.

In this case I came across a module that has really captured my imagination. It is called:
rpi-temperature-mqtt.

It was originally written by a fellow who calls himself HackerCowboy. Here is his Git repository link, https://github.com/hackercowboy/rpi-temperature-mqtt.

I grabbed the code and started playing with it.

First you must install the module onto you device (raspberry pi or ESP32).
pip install rpi-temperature-mqtt

All of the configuration is done in a file you create called config.json. Here is my config.json from the barn:
{
    "mqtt_client_id": "barn",
    "mqtt_host": "10.10.XX.XX",
    "mqtt_port": "1883",
    "wait_update": "60",
    "verbose": "true",
    "sources": [
        {
          "serial": "28-0000055aae0e",
          "topic": "workshop"
        },
         {
           “serial”: “28-0000054de0b5”,
           “topic”: “boxstalls”
          }
      ]
}

As you can see I have two DS18B20s that I am sampling. The code supports multiple devices. The other item of note is that the code provides its own scheduler capability. You set the time period between sampling of the collection of devices with the line:

“wait_update”: “60”,

This says to wait 60 seconds between sampling another set.
It also provides a delay between devices. That line is:

“wait_process”: “10”,

In this case it will wait 10 seconds between samples from the two DS18S20s.
Now to the section labelled “sources”. For each device you will need to add two lines.

“serial”: “your device serial number”,
“topic”: “what you call it in MQTT”

Note that each section must end in a comma after the topic statement. That is except for the last section where the comma is left off.

The other item I use in the config.json file is the verbose option. It lets me see exactly what the module is sending and shows me when the connection is made:

NOTE:

Before you start using rpi-temperature-mqtt you MUST initialize the Device Templates. This is a new change in the later versions of Raspian, etc. If you do not do this rpi-temperature-mqtt WILL NOT WORK. So you must add the following instructions to boot/config.txt

# Add the device tree to initialize w1-gpio.
dtoverlay=w1-gpio

Save it and reboot. If you forget you can manually start it by issuing the command:

sudo dtoverlay w1-gpio

Remember if you don’t do one of these the rpi-thermostat-mqtt code will error and say  it can’t find w1-master…..

MQTTfx:

I use a Ubuntu desktop machine in my lab. As well I have a separate server that runs all my docker containers. So for testing purposes I log into my test device (be they raspiberry pi or ESP320) via SSH and control them.
I found that in order to really see what is going on an MQTT broker monitor is really useful. Enter MQTTfx. This piece of code will run on a variety of platforms and gives you the ability to subscribe to a broker (local or remote) and then publish or subscribe to the topic of interest.

The first thing we need to do is create a connection profile. Select the Broker Status tab. Then from the top Extras tab, edit the connection profile:
  

  
· Enter your new Profile name at the top.
· Enter the ip address of your Broker.
· Enter the port number of your Broker.
· Leave the rest as presented.
· Click Apply, then OK.



You will be returned to the main screen. Now make sure Broker status is selected. From the profile pull down window select your newly created profile and then click on CONNECT:


Once you connect you should see this screen which connects you to the Mosquitto Broker:



Now click on Subscribe.

At this point you need to start the rpi-temperature-mqtt software running on your device by issuing this command:

rpi-temperature-mqtt config.json

Make sure that you do this from the subdirectory that contains config.json.

Now when you switch back to the MQTTfx screen and you select your topic(s) you should see something like this:


Note I’ve selected to view the workshop topic from the pull down. The lower right window displays the last reported value.

Ok, so this post is getting a little long so I’ll end it here. Next we’ll move to the ESP32 and talk about micopython, rpi-temperature-mqtt, and Hazzuh32 (ESP 32) from Adafruit.

Cheers.
Glenn.


Sunday, January 5, 2020

Introduce End Devices to a Network and Introduce Problems

As I've touted for years now I have a network of XBees that I (basically) run my house with. I monitor room temperatures in key areas, control my pool, monitor my power, etc; all without a bunch of wires strung around the house. For the first year I ran a network transparent (AT) mode that broadcast to all devices and all of them listened to what was going on....that didn't work well.

I described what the problem turned out to be back then <link> and moved to a more directed network using API mode to control the traffic level and increase the speed of throughput. That all worked really well. Then I created the room temperature monitors. They were created as battery operated devices since I wanted to put them in places where there was no power; XBee routers use too much power for such an application, so they had to be XBee End Devices.

That was the beginning of a long time problem that I simply couldn't find, and has driven me nuts a few times before finally getting the entire mess to work again.

The symptom was that a device would leave or get kicked off the network and then simply refuse to join back in. I'd go for a few days and a sensor would leave and no amount of resets, power cycles, slaps or flights across the room would get it back on the network. Hell, I even programmed another XBee and put it in the same place and it wouldn't work. Then, seemingly at random, it would connect and start working all by itself.

Months of watching the XBee traffic after adding a ton of logging to almost every device in the house led me to nothing at all. Reading every blog and question remotely related on the web told me nothing. I was completely baffled by this problem.

Some of the things I tried were to automatically reset the network to force it to reform using the NR=1 command. This dumps all the routing tables and everything rejoins. This would work, but if it happened to often, the entire network would go down and I had to intercede at each device to get it back up.

Hook up a tablet using an OTG cable hooked to the device that most often failed and monitoring the activity for hours hoping to get a clue what was going on. This was cool because it allowed me to learn how to watch a device using something that wasn't a laptop running the entire Arduino IDE. I could plug into an active device and watch what was being logged without resetting the device. This is a nice thing to have available, but it didn't help find the problem.

I had the device reset itself, issue it's own NR=0 command to clear the local tables, reset the XBee, just about anything I could think of and nothing helped. I could have ignored it if I was only reporting temperatures, but two of the sensors were serving as the temperature sensors for my air conditioning system.

It really sucks when the cooling stops at 110F outside and the house heats up. It sucks about equally when the heater sticks on and the temperature goes up to 90+ inside on a cold day. Power bill didn't like that much either. I didn't want to break down and go back to the old method of measuring temperature, the sensors made the house much more comfortable.

I finally got a hint from a question asked about the XBee end devices not being able to rejoin a network. Seems the XBee routers have a table of 12 entries reserved for end devices that they can parent. That's cool, but I don't have 12 end devices. I still read the device tables on the XBees looking for what the heck was going on though. Then I found it.

I had relatives visit during Christmas and they brought their cell phones. The folk (my kids and grandkids) are ALWAYS on their phone. Either talking, playing games, texts, whatever; their eyes and hands are literally glued to the phone. The increased RF and WIFI traffic saturated my house and the network struggled trying to get packets through the interference that comes with low power RF activities. Devices disconnected and couldn't rejoin, packets got lost in the ocean of packets from all the devices, it was a total mess. I dug in again to see if I could get a clue.

I actually found the problem. What was happening to me is that the XBee end devices have to check in periodically to maintain their connection. If you wait too long, the table is purged to conserve the device table space for end devices. The time allowed is set by parameters on the XBee router and the end device needs to check in often enough not to get purged. I was using a 2 minute timeout on the temperature sensor and the default on the router.

To make things worse, I was using hardware control of the sleep period, and not correctly handling the interaction of the Arduino and the XBee conversation.

A couple of corrections such that I would send the temperature message, ask the XBee to go to sleep, then WAIT until it actually went to sleep before sleeping the Arduino made things much better. I allowed the end device to exhaust the stored messages that the XBee router parent was holding by just waiting until they all came in. The final item was to extend the XBee router timeout to way higher than necessary for a couple of missed transactions (like a full day) took care of the problem of it not being able to rejoin.

I was actually preventing it from rejoining by sleeping the device too quickly; it just couldn't get back in before I told it to shut down.

Why don't other people have this problem? I think they do sometimes, but didn't spend the time it took me to chase it down. I spent months watching and trying things before I stumbled on it mostly by accident looking at the tables because of some other problem someone else had with their network.

My network is humming along with only an occasional missed message. The extended awake time hasn't seemed to be a problem with the battery life either. The XBee trying to rejoin was a heavier load on the battery than the extra time the receiver is on. Transmit takes more power than receive, and I only transmit one message every two minutes, so the tiny overhead of the acknowledge packets isn't noticeable.

It's been seven full days of bliss because all the sensors and control systems are working perfectly. The network even has more capacity available for even more XBees. This is really tempting because my indoor freezer has a failing thermostat. Stupid thermostats on freezers are expensive and I already monitor the temperature inside it. It may be time to take complete control of the freezer. I wouldn't even consider that with the devices acting the way they were.

I know, in the scheme of things this short a period of time doesn't actually prove the problem is gone. But, the instant clearing of problems that had culminated with the increased number of cell phones pretty much convinces me I have it taken care of.

Maybe I can start thinking about something else now.