Monday, May 11, 2015

Yet Another Update to the AcuRite 5N1 Weather Sensor

The previous post on this project is here <link>

Just when you thought I'd finally shut up about the AcuRite weather sensor I decide to post again. I promise, at some point I'll actually be satisfied with this device, but not quite yet. Over the months I've been working with this I've received comments and emails from folk that tried it and discovered things. One reader had an especially interesting situation; he had a neighbor that had a sensor within range. This meant that the code would pick them both up and present data from both sensors. To add insult to injury, when he tried changing his station to channel C, he found out that rtl_433 had a bug that prevented his receiving data on that channel. He must have been really annoyed at that point.

Since wind and temperature matter to placement, this was causing him grief and he set off to fix it. He isolated the sensor ID, and while he was at it, decoded the battery level, channel and found a better way to read the RF that allowed channel C to work.

During his trials he also found that the checksum was allowing bad data into the decoding and would present unusual readings. I've seen this over time as well. Things like a wind gust of 147 mph, a temperature spike of 30 or so degrees for no reason, the usual outlier readings, but with weather, it's hard to tell what an outlier really is.

I totally stole his ideas and some of his code and incorporated into my weather station,; I haven't been running it too long, but it looks like the vast majority of the outlier readings are gone. There may be some turn up, but so far it's been good.

First, inspection of the data showed that the bytes holding the various readings were all even parity. I added code to check the parity and discard the message if it didn't pass. Note that I only check the readings bytes because the first, second and last bytes are sensor, status, and checksum; these don't incorporate parity. In the c code I add the decoded sensor id, battery level, and channel to the JSON string I output, then check for the proper values in the python code that receives it. I did it this way because some folk may want to catch multiple sensors and average or compare them; especially if they can see a neighbors device for free.

I also noticed at my house that I receive a couple of devices from other folk that are not AcuRite devices. These things can cause problems as well, especially since they could easily be using the same checksum algorithm.

All the changes meant that I had to get the very latest version of rtl_433 from the original github repository. This was required because the rtl_433 folk made a new decoding method available that works better with the AcuRite data stream. That decoder wasn't there in January, but it is now. So, I tried again to fork it and failed miserably, and wound up just updating my copy of rtl_433. But, another good thing happened as well. The folk that wrote rtl_sdr, the actual radio software have improved it a lot and it installed first time, and I got to remove that from my copy.

Now, you get rtl_sdr, install it, build it, and then get my rtl_433, install it and build it. That will give you the version I'm using with all the changes to support the newest findings. My python code for the weather station I run is in GitHub as well. The specific stuff I use is:

rtl_sdr <link>
rtl_433 <link>
weather station <link>

The reason I linked to a particular commit in GitHub is so you can have exactly the same code to work with. The two rtl projects are in flux and could change dramatically at any time. You can always get the latest version if you think it will serve you better.

The latest version of rtl_433 not only has the new decoder software, it also allows you to specify the device you expect to receive. That makes it nice for me since I only have the AcuRite device I'm interested in. I changed my start up command to support this feature:

rtl_433 -f 433.915e6 -R9 2>> saveweather.log | saveweather.py  >> saveweather.log 2>&1

The '-R9' specifies the 5N1 device. Notice how I save the stderr output to a file? That helps me understand how many errors I'm getting and could help me fix a problem. Right now I'm also using the -D for debug option as well to get more information. Try '--help' on rtl_433 to see the other options it has.

I owe this latest version to Pcjunky (yes, I know his name, but I ain't gonna post it without his permission). He took my code and beat on it until it worked to his satisfaction. Thank's a lot.

Now, I want to show you the parity routine I ran across. Here's the code:

static int acuriteParity(uint8_t v){
    // returns 1 if parity odd, 0 if parity even
    v ^= v >> 4;
    v &= 0xf;
    return (0x6996 >> v) & 1;
}

This calculates the parity for a byte in around five cpu operations. It's an extremely clever implementation that I leave the explanation as an exercise for the student.

Have fun.


54 comments:

  1. I have downloaded your latest code from the above links and I have some questions.

    1. On line 137 of the "saveweather.py" script it says "get the values out of the houserc file". I am not able to find that file.

    2. Is it possible for you to post the schema for the database?

    I finnally got my Raspi connected to the SDR dongle through a Belkin powered USB hub so now am working on getting the data out and into a MySql database.

    Thanks.

    PS. As usual I find your blog posts very helpful on my above quest.

    ReplyDelete
    Replies
    1. The houserc file is where I have all my keys for the cloud providers, database passwords, etc. So, I don't put it in GitHub. I describe the format of it in a blog post though.

      http://www.desert-home.com/2014/10/using-rc-file-in-my-system.html

      The python file to read it is in github, it's called houseutils.py

      The database schema isn't secret, and it isn't finished. I store everything as a separate record right now, so there's a table for outside temp, rooftop temp, etc. I figured out how to print the structure of a table, but not the entire database. Basically, I'd post the schema, but I can't figure out how.

      Delete
    2. If you use phpmyadmin, which I reccommend, it is easy to EXPORT your database to a SQL file. After the file is created you can delete all of the "INSERT" statements and what is left will be the structure for all of the tables.

      That file can then be read in by MySql and it will recreate the database as it stands.

      The INSERT statements insert the data into each table so would not be needed for a new database.

      BTW, phpmyadmin is free and is available for download at: http://www.phpmyadmin.net/home_page/index.php. This works with MySQL databases and makes many operations very easy.

      Hope this helps.

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. Actually, my version of phpmyadmin had a set of check marks that allowed me to leave out the data during the export. I just used the check marks and dumped it into a text file.

      There was a lot of indentation and stuff, so I checked into the weatherstation on GitHub so you wouldn't have to deal with that.

      Thanks for the pointer.

      Delete
  3. I just downloaded and compiled your version of rtl_433. Discovered that it does not support the -R parameter.
    It looks like I am picking up some other 433mhz device so I could use that -R option to specify the Acurite 5N1 device.

    Thanks.

    ReplyDelete
    Replies
    1. Hmm, mine does, and the one in GitHub has the code in it. Go into the rtl_433 directory, then into the src directory and look at rtl_433.c. Up at the top it should have the 'usage' help text, and down towards the bottom, it should have the argument parsing for the -R parameter.

      Delete
  4. Just deleted all of my old files. Downloaded the latest again. When I run cmake ../ I get the following error on both my PC and Raspi:

    "By not providing "FindLibRTLSDR.cmake" in CMAKE_MODULE_PATH this project
    has asked CMake to find a package configuration file provided by
    "LibRTLSDR", but CMake did not find one.

    Could not find a package configuration file provided by "LibRTLSDR" with
    any of the following names:

    LibRTLSDRConfig.cmake
    librtlsdr-config.cmake

    Add the installation prefix of "LibRTLSDR" to CMAKE_PREFIX_PATH or set
    "LibRTLSDR_DIR" to a directory containing one of the above files. If
    "LibRTLSDR" provides a separate development package or SDK, be sure it has
    been installed."

    I haven’t been able to find that file anywhere on my system or in a Google search.

    Thanks.

    ReplyDelete
    Replies
    1. It sounds like the build for rtlsdr didn't work. There are three step to this build librtlsdr, then build rtl_433, then compile the source file I use. The libraries for rtlsdr get put into /usr/local/lib during its build and then they are used by the rtl_433 build.

      It looks like the missing file is actually created by the build of something and that's not happening for some reason. If you already have the librtlsdr files in /usr/local, they may be owned by root and can't be copied over.

      Try the various builds again and watch closely for something failing.

      Delete
    2. I just deleted all of my old files, downloaded the latest files from github, compiled each of the first 2. When I try to do a cmake on your master file I still get the error where it can't find the 2 following files: LibRTLSDRConfig.cmake
      librtlsdr-config.cmake

      Any more ideas?

      Thanks.

      Delete
    3. Totally my fault, apologies. They had a .gitignore file in their stuff that caused 'FindLibRTLSDR.cmake' to be left out. I didn't see it because it was there and being ignored. I edited the .gitignore file and now the file is in github.

      The way I tested it is to download the zip file by clicking on the bottom link on the right hand side. Then I unzip the file on the pi. I go into the directory and then mkdir build; cd build; cmake ../; and make. I stop there and just copy the rtl_433 file where I need it.

      If you want to do the cmake install, you have to be root, but I don't even bother with that part.

      Delete
    4. I got it to compile OK now. The problem I now have is when I run rtl_433 it prints a bunch of stuff and then says "Tuned to 4339....." and it just hangs there with the cursor blinking. The first time I ran it tonight it said "Found Acurite 5n1...". It printed this line several times until I killed it. Before it would print out data that it found.

      Read your Weatherschema.txt file. 1 suggestion: Add a column that is an int, is autoincrementing and is a primary key to all of your tables. I have read that this speeds up searches of the DB.

      Delete
  5. You may not have a problem. Let it run a while. I have the code collecting data until it has both kinds of messages from the weather head before it outputs data, and it take some seconds to get both of them. Look back at the posts for this and you'll see what I did for my house. I also separate the logging to both stdout and stderr. The debug goes to stderr and the stuff you want to save goes to stdout.

    Feel free to change it to do what you want at your place.

    Having a primary key of some type does speed up searches because mysql builds a search tree as it puts data in the database. I don't search this database yet, but when I do the key will be related to time in some fashion. I may get rid of the various different records and just go to one record a day; this was just something to try out the database with.

    ReplyDelete
  6. When I run the command "rtl_433 -f 433.915e6 -R9 2>> saveweather.log | saveweather.py >> saveweather.log 2>&1" and look in the "saveweather.log" the first entry is "-bash: saveweather.py: command not found"

    I did a chmod to 755 on that file and still get the message in the .log file.

    I am making some progress but still problems keep coming...

    BTW, when I just run "rtl_433" it outputs data to the screen so that part is working.

    ReplyDelete
    Replies
    1. I put a "./" in front of the *.py file and the error went away. It is now running so will see what happens.

      Delete
    2. Welcome to the annoyance of the path variable. Each OS has one and they are different. If you're working linux, you can edit the .bashrc file and fix it. When I set jobs up in linux init, I use the entire path name because I got tired of trying to figure it out. Something like /usr/home/pi/... This kind of thing will drive you a little nuts, but once you've seen it, it's not hard to fix.

      Delete
  7. Thanks to the log file I was able to correct the errors that occurred because I was lacking some packages on my system.

    The program now runs but I don't get any data in the DB or in the log file. The log just shows "default" for all of the weather.values.

    I put a print statement just after the "char = sys.stdin.read(1)" statement but nothing got printed. Could it be my Raspi doesn't recognize that statement? There aren't any errors printed though.

    How is the saveweather.py script supposed to get just the numbers from the string that the RTL_433 program outputs? I haven't been able to figure that part out!

    ReplyDelete
    Replies
    1. saveweather.py get the stdout from getweather.py as a JSON string. It parses the string and pulls the values out, then does stuff with them. Traditionally unix based processes take stdin, do something with it and produce two outputs stderr and stdout. This is what getweather uses to pass the data and produce its output. The stuff you want to save goes to stdout and the log information goes to stderr.

      That's what the weird line to run the two of them does.The reason I use JSON format is because you get parsing and data conversion for free at the same time. This makes taking apart the data really simple and it also allows different processes to use the exact same data.

      The reason you don't see anything when you run it is because python buffers stdout when it is run non interactive. This is annoying and there's a few ways to fix it. I usually put a flush to stdout after each print to get it to go to the screen sys.stdout.flush() will force the characters out. Also adding a python option will do it, but I forgot the option.

      Delete
    2. I don't see the script "getweather.py" on my system in the files that I have downloaded.

      Delete
    3. OK, I screwed up again. That's the problem with doing a dozen projects in a few months. On each iteration of the weather station software I used different names for stuff. At some point in time the code that got the weather from some device was called getweather.py, probably back when I was still using the USB connection to the AcuRite console.

      Now, the thing that gets the weather from the device is called rtl_433 since I modified someone else's code to work for me. So, just replace getweather with rtl_433 in the above and the explanation stands.

      Delete
    4. I am using: "rtl_433 -R9 2>> saveweather.log | saveweather.py >> saveweather.log 2>&1" as per the explanation above, but I am not getting any data in the log file or the database. If I run rtl_433 by itself I get data as: "Detected Acurite 5n1 sensor, 1 bits
      wind speed: 3 kph, temp: 87.6° F, humidity: 76% RH
      Detected Acurite 5n1 sensor, 1 bits
      wind speed: 3 kph, wind direction: 135.0°, rain gauge: 0.00 in."

      So I can't figure out why the "saveweather.py" can't read it.

      Delete
    5. That's the wrong string. You should get a json string that has a bunch of colon and curly braces in it. The reason saveweather isn't parsing it is because it's in the wrong form.Did you get the files from here https://github.com/draythomp/Desert-Home-rtl_433 ?

      This is the actual line I run it with copied out of my init script:
      /home/pi/src/weatherstation/rtl_433 -D -R9 -f 433.915e6 2>> /var/log/house/
      saveweather.log | /home/pi/src/weatherstation/saveweather.py >> /var/log/house/
      saveweather.log 2>&1

      and this is the output of rtl_433:

      Detected Acurite 5n1 sensor, 1 bits
      Detected Acurite 5n1 sensor, 1 bits
      {"sensorId":{"SID":"92","t":"1434292619"},"channel":{"CH":"A","t":"1434292619"},"messageCaught":{"MC":"0","t":"1434292619"},"battLevel":{"BAT":"7","t":"1434292619"},"windSpeed":{"WS":"3.5","t":"1434292619"},"windDirection":{"WD":"NNE","t":"1434292619"},"temperature":{"T":"89.8","t":"1434292601"},"humidity":{"H":"27","t":"1434292601"},"rainCounter":{"RC":"445","t":"1434292619"}}
      {"sensorId":{"SID":"92","t":"1434292619"},"channel":{"CH":"A","t":"1434292619"},"messageCaught":{"MC":"0","t":"1434292619"},"battLevel":{"BAT":"7","t":"1434292619"},"windSpeed":{"WS":"3.5","t":"1434292619"},"windDirection":{"WD":"NNE","t":"1434292619"},"temperature":{"T":"89.8","t":"1434292601"},"humidity":{"H":"27","t":"1434292601"},"rainCounter":{"RC":"445","t":"1434292619"}}
      ^CSignal caught, exiting!

      That's the kind of output saveweather is looking for.

      Delete
    6. Gerald, if you drop me a note at the email address in the about me (on the right), we can actually do this together and get it running for you. You've been at this a while and I'll be glad to help you directly.

      Delete
  8. Hello! For a while now I've been capturing my 5N1 plus 8 or so Tower sensors by intercepting data from the Acurite Internet bridge. I bought an SDR to see if I can improve my range a bit, and I'd also like to get the data from the Acurite BBQ thermometer.

    I wasn't getting any output apart from a few rain gauge readings. In debug mode I got "Parity error byte 1, F7" - so I forced the party in acurite_crc to always return 1, and now I'm getting data from the 5N1..

    I don't seem to be getting anything from the tower sensors. Do you know if it's intended to decode the 592TXR? Thanks for your efforts on this!

    ReplyDelete
    Replies
    1. Last time I looked no one had cracked the datastream for that device. However, there are people working on it all the time. Take a look in GitHub at the original repository for the latest.

      Delete
    2. Doesn't look like it. Although with the newest code I'm even less successful getting 5N1 data. What could be different about mine? Debugging this might be a challenge since there are so many 433mhz devices around me. Interestingly the WSDL folks look like they've tackled the 592TXR: http://wmrx00.sourceforge.net/news.html

      Delete
    3. The latest rtl_433 has a new demodulator in it take a look at the AcuRite decoder and it's in there. Try different ones and your problems may go away. The way it is now works best for me and some other folk, but it may not for you. Check it out.

      Delete
    4. Rats, I thought I had posted a reply and it looks like it got lost. Anyway:

      1. I was definitely victim to the ABC switch. I changed it to A from C and the official rtl_433 started returning some data - but I'm still failing a lot of the if..buf tests. But I need to look into that more now that I've changed the ABC switch.
      2. I noticed that in your code,acurite_detect you've commented out the pRow[0] |= pRow[8]; line..?
      3. As I mentioned, I never seem to pass parity with your code. If I ignore that, I do get some real readings. However it looks like it's fewer than I originally thought. Since acuriteShowit is outside of those if..buf tests, I think it's just reprinting the old packet.

      Delete
    5. Go to the source and look at acurite.c down toward the bottom are structures that tell how to work with the radio. You want to play around with the modulation parameter. There are definitions for the different types in rtl_433.h that will tell you the values to try.

      Also, my sensor and radio combination will not work reliably at the default frequency setting. I get something similar to what you're getting. Bring up the radio on a pc or something and play with the frequency of the radio to zero in on exactly what gives you the best signal. I mentioned that in a previous post, but you may have missed it. What happens is there is some drift in the components and you may have to adjust for it.

      I even have to do it in cold weather because it drifts away from my setting when it gets below 30F up on the roof. I'm thinking about a summer setting and a winter setting because of that.

      Delete
    6. Thanks for the tips! I'll have to shut off all my other 433mhz sensors so I can isolate the signal. I'll report back!

      Delete
    7. Ok, I've shut down all my other sensors and fired up SDR#. The blip covers a range of frequencies - starting a bit faded around 433.916, then bright in the middle between 433.936 and 433.943, then fading off again.. so would I target the very middle?

      Delete
    8. Yep, that's what I did. I just picked the highest peak in the middle, read the frequency up top and that's what I used. Like I said though, low temps change it,so it will move around a bit on you.

      Delete
    9. I know this is old, but I've added 592TXR "Tower" temperature and humidity sensor support to rtl_433. I've also recently added the 986 refrigerator/freezer thermometers. See github.com/merbanan/rtl_433.

      Delete
  9. I have been reading all of your weatherstation trevails since December. Good stuff ! My ancient LaCrosse station finally took some lightning, so I'm going the Acurite/RTL route. Now I'm trying to get rtl_sdr installed on my rPi, and running into library dependency hell. It needs cmake, which doesn't already exist on my rPi. When I tried to build apt-get install cmake, it failed on libcurl3_gnutls:

    Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/c/curl/libcurl3-gnutls_7.26.0-1+wheezy3_armhf.deb 404 Not Found
    E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

    Suggestions ?

    ReplyDelete
  10. Ooops. Never mind on that, but .... Found the .deb and dpkg -i did the job on that libcurl. Went back to apt-get install cmake which claimed to have worked. But now I have the next issue. Tried to execute cmake to install rtl_sdr and got "Illegal instruction" .

    Should I just start with a new, clean OS on my rPi instead of Raspbian/Wheezy from September 2012 ? If so, which one ?

    Thanks,
    Steve

    ReplyDelete
    Replies
    1. Yes, get a new version of Raspbian. I had a similar problem and it took me two days to figure out that the darn thing was out of date. You can update it also if you want to, but it might be easier to just get a new version.

      Delete
    2. Thanks ! I'll do that tomorrow when I go to town. I live pretty close to you - a rural area near Silver City, NM. Stuck with Verizon 3G and its data cap.

      Delete
    3. With 3G, you definitely want to take the raspberry to town and both load the OS, then do an update-upgrade while you have internet that doesn't get charged to your account. The last time I did it it took three hours.

      Delete
    4. Success ! Also took me three hours, but the issue was getting the image onto the SD card. The dd appeared successful, but nothing was on the card. Finally did it on Windows and worked the first time. Smooth sailing after that. Have rtl_sdr and your rtl_433 built. The WX station and the SDR will be here next week.

      Delete
    5. Congratulations for sticking with it. Pain in the bottom isn't it?

      Delete
  11. After a little saga with Amazon, I am finally in action. Looks like channel A is 433.979 MHz. Does that sound right, or is there some calibration I need to do ? Using SDRSharp. The data bursts are pretty short, compared to my (mostly defunct) LaCrosse which is down a little in frequency and only transmits about every 2 minutes.

    ReplyDelete
  12. More success ! rtl_433 grabbing the data on 433.979 ! Thanks for all the great work !

    ReplyDelete
  13. Disappointment on the RF side. I have been a ham operator for 45 years and a EE, so I know a little about RF. For inexplicable reasons, the frequency is now 433.924. The problem is the range. Trying to go 100 feet, line-of-site. Very weak in sdrsharp, and no demodulation of data. That cheap antenna that came with the RTL-SDR looks close to 1/4 wavelength on 433 MHz. Surely I don't need to build a 433 MHz yagi to go 100 feet ? Have you peered inside the Acurite to see what its transmitter and antenna look like ?

    ReplyDelete
    Replies
    1. I did, but there's been a lot of water under the bridge since then and I don't remember squat about it. I was lax and didn't take pictures either. I suspect it is one of those printed circuit antennas that won't transmit worth a darn.I was anxious to get some real readings and put it up on the roof too quickly and it's too darn hot to get it down now.

      At some point, I want to drag it down and take a long look at it to see how it's actually put together and post about it, but it will have to wait until the heat drops some.

      Why do you need it 100 feet away? I suspect it has the same range as some of the toys we see at that frequency, 25 yards or two walls, whichever comes first.Mine even changes frequencies during the day as it heats up.

      Delete
  14. 25 yards or two walls, whichever comes first ! I'll have to remember that :) . I'll open up the Acurite today and see what I find and report back. It is nice having RTL-SDR and running sdrsharp - it gives me the chance to see if hardware changes have any effect on signal strength.

    ReplyDelete
  15. The antenna in the Acurite is a helically wound monopole. Or, for the less nerdy, the spring from a pen ! I'm not joking ! I replaced it with a 6" wire and drilled a small hole in the top of the Acurite for the wire to stick out of. Hard to tell how much of an improvement that is. Finally gave in and moved the rPi to my utility room, dropped the RTL-SDR antenna out the window, and now it is a 50 foot clear path to the Acurite mounted on one of my ham radio towers. Tried to find my way around the librtlsdr.c code to see if I could find out how much energy was being detected in each frequency bin, and see if I could better optimize it, but got lost in the code :) . A project for a later date.

    ReplyDelete
  16. Oh..found a small bug you might want to look at - in acurite.c, at the end of acuriteShowit(), I needed to add: reportsSeen = 0; to reinitialize that for the next sweep.

    ReplyDelete
  17. The problem with certain channel and ID combinations not working looks to be that the parity bit is not used on the channel byte. It can't be on the parity bit, because the high order bit is part of the channel number. There is a comment in the code to that effect, but the for loop that checks parity starts at least one byte too early.

    Are you subscribed the rtl_433 google group? https://groups.google.com/forum/#!forum/rtl_433

    ReplyDelete
    Replies
    1. Clarification, For the 5-n-1, it looks like parity isn't used on the two Channel/ID bytes. As I mentioned above it can't be used on the 1st byte due to the channel number, but it also looks like it isn't used on the 2nd ID byte.

      To fix: change the start value on the parity checking for loop inside of acurite_crc from i = 1 to i = 2.

      for (i = 2; i < cols - 1; i++) {

      Also your sensor id routine is incorrect. Looks like the 5-n-1 uses 12 bits for the ID, mine is reported as 0x0832. (I have the acu-link bridge, so I see what that thinks the IDs are.)



      Delete
  18. I finally got around to trying this again. Did a fresh install of all the libraries and got your latest code. What a difference! Before I wasn't getting any data at all. Now it picks up the 5N1, an Oregon Scientific sensor I have sitting somewhere, and now the tower sensors thanks to your update.

    I just bought the new 00275TX Room sensor, where you can chain water leak and soil temp sensors. Was hoping the bridge would pick them up, alas it does not. Wonder if someone will decode this one.

    If I wanted to take a stab, any thoughts on isolating the antenna so I don't get noise from other 433mhz devices around me? (other than powering off all other sensors)

    ReplyDelete
    Replies
    1. I can't be a lot of help on this because all this stuff works on the same frequency, however I have had some luck putting the antennae in a can. Get something link a Pringles can and cover the antenna, then set it close to the one you want to see. That might help isolate it from the others.

      You can also just pull the antenna loose and try moving the radio close to the device you want to monitor.

      You can also use the same pringles can to form a directional antenna and try aiming it.

      None of these may work, but I've had some success by limiting reception and moving the radio close to the transmitter.

      Delete
  19. People have tried predicting weather for a very long time now using a number of different methods, some of which proved effective while others did not work at all.Owen

    ReplyDelete