Friday, May 29, 2015

Arduino Controlled Sprinklers, and A Guest Speaker

Yes, I have a guest speaker for this post. Glenn lives in the part of Texas where they got about a ton (per square inch) of rain recently. Since he couldn't play outside, he decided to play inside with ... yes, a sprinkler system. I asked him to write about it because it is a novel and really, really cool setup. It uses those valves that latch into place and take zero current to run saving power and allowing for the possibility of battery operation. I'll shut up now and let Glenn take over:

Lately I have been working on my baler between rain storms. However, during the rain storms I have continued to work on my vineyard irrigation controller. Not that the grapes are going to need watering any time soon. But there will come a time and I must get this moved forward.

I have a large aluminum panel mounted to my steel pipe fence with a water hydrant next to it. I have my three zones of irrigation pipes running up to this panel (just underneath it). I have purchased a three zone manifold valve system from Orbit and equipped it with two 24VDC solenoids. The black object on top of the green valves is the solenoid. The 24VAC ones have two black wires. The 24VDC solenoids have one red and one black wire. The red goes to the set screw terminal and the black goes to the reset terminal of the zone. This is a really easy thing to do. The manifold will be mounted on the lower left of the panel just above the zone pipes. There is no need for me to put these in the ground as it is more of an industrial installation. The aluminum panel faces north so nothing will get direct sunlight, which here in East Texas is intense. Currently the aluminium panel has just a water filter mounted on it. I welded Unistrut ( this is the funny C channel electricians use) to the pipe fence then bolted the panel to the Unistrut. It makes taking the whole panel off really easy should it be necessary.



Back to the controller. Just above the valves I will be mounting a 4X4X2 inch plastic electrical box. I got this box from Home Depot. I comes with a waterproof cover and mounting tabs on each side. This is a shot of it open with the cover reattached backwards.



As you can see from the above photo I will be mounting my ardweeny and support shields to the cover of the box, not inside the box. There will be one or two access holes in the bottom with water tight entry fittings through which the wires will got to the valves as shown in the first picture. There will be three shields to the system. Ardweeny, XBee and Valve controller, all stacked as shown.

The really key thing here is that everything will mount to the cover, not inside the box.
I have several reasons for this. All fall under the category of convenience.
When I open the box and reverse the cover as shown, everything is readily available. I can access the usb port without problem. I can remove any or all of the boards conveniently. I can work with the enclosed wiring without having the shields also in the box getting in the way.
If I need to I can remove the entire controller just by disconnecting a few wires and take it away. The entire box stays mounted in place. Oh and by the way, since there are no screws on the back the box will site flat on the surface of the aluminium plate.

The box cover will also have a key pad mounted on the front for manually turning the irrigation zones off and on:



The Orbit manifold I purchased from Lowe's. They usually have them in stock. They come with everything you need including 24V AC solenoids. I think I paid $54.00 for it. I bought two 24V DC solenoids off Amazon that were "used" actually only the packaging was used for $7.00 each. The third I will buy from Lowe's locally as they are in stock and only cost about $10.00. They simply unscrew. My 24V DC solenoids came with a little gray adapter that has to be removed and then screw it into the manifold. I can take some pix and send them if you like to describe it.

Rain protection. None required. This stuff is meant to be in the ground. It gets wet. Everything will be mounted on a vertical plate of aluminium. If one wanted you could simply add a little roof shelter at the top of the aluminium, but I do not plan to. The 4X4X2 box is water tight. The connections coming in from the bottom mean rain can't enter unless it rains uphill.
I mentioned water proof connectors on the bottom will also prevent wasps, and other critters from getting at the circuitry.
I may put a little roof cover above the keypad like they do for gate openers that you drive up to. I will seal any entry points such as the keyboard cable entry with silicon sealant.

I plan on powering this with either a 9volt battery or a combination of 9volt and three AA's. This way I won't need to have a regulator. I plan on having it shut down through software much the same way as your remote temperature sensor works. The jury is still out just yet on this. I plan to liberally and unashamedly steel Dave's software......Hee Hee Hee. He's done a great job and created a voluminous library of functions that will save me from reinventing the wheel.

I chose to go with Ardweeny because there is a nice little shield for it that fits the Arduino form factor. I can do all my testing and code development with the regular Arduino and then simply replace it with the programmed Ardweeny. No other special mouning requirements. I have the shield on order and as soon as it arrives I can drill the holes.

I plan on a next issue of this showing how I will use insert nuts that have a built in standoff of about a 1/4 of an inch to mount the Ardweeny shield to the cover and gluing up of the finished numeric pad. There will also be more discussion on powering. I am thinking of a second box that could house a small motorcycle battery. I found one for $25.00. The big thing on this to remember is that this only will run through the summer months so I could conceivably charge the motorcycle battery once a year and run it all summer. More to come on this note.

Still a few things to think about with this yet. I'm not sure about a led in the outside of the cover plate that will illuminate when you push the button. I'm thinking that the clicking sound of the solenoid plus the rush sound of the water in the pipes might just be enough. Needs to pass the "wife" test though.

So there you have it. The main theme of the blog is really "Don't forget the cover" and the benefits it affords for mounting options.

Comments?

This is Dave again, I have a comment, AWESOME. I've been using those waterproof boxes from Home Depot for quite a while for various projects and it never, ever occurred to me to mount everything to the lid <link>. Simple idea that I should have thought of.

Yes, I'll be stealing that idea and others from him.

Tuesday, May 26, 2015

Hacking Into The Iris Door Sensor, Part 3, The code

The previous post in the project is here <link>

Last time I described the Key Fob that Lowe's sells and previous to that I showed the Door Switch, but I haven't shown you the way I got them to work. I'm still a little reluctant because my partner in this project hasn't succeeded in getting his devices to work properly, but it occurred to me that someone out there might pick up the code and give it a try. I know several folk have used my Smart Switch code to control and monitor things around the house, maybe they're just waiting for me to publish how to do it.

At any rate, this is how I got the switches and key fob to behave:

I have eight devices, five of the smart switches, two door switches, and a key fob, and all of them are working with the same monitoring code. The code is only to monitor the devices, there's no provisions for control, Actually, the only thing you can control is the smart switches, and I've already posted code for that, so this will show you how to monitor all the devices. This code works pretty easily, pull the battery out of the device, start the code, put the battery back in and push the button eight times quickly. The switch will contact the code and join on its own. Once joined, the door switch and key fob will send status every two minutes telling you they're alive and in range. The smart switches send status much more often with a cumulative status every minute. If you want to join other devices, just pull the battery, put it back, and push the button a bunch of times; joining is always enabled and the next device should work fine. Stopping the code (cntrl-C) and restarting it won't cause the devices to leave the network, they'll be fine when you restart the code. I did this a LOT when I was trying to get them to work.

I could go into intimate details of how the interaction works, but the code will show you this much better. What happens is that the switch sends an device announce message and the code will respond and then they exchange data. During this there are two specific message that are sent to Alertme devices to get them to accept the XBee coordinator; this is the secret part of the Iris system that makes these devices unique. The code I came up with for the smart switches wouldn't work because the door switch and key fob are slower devices and I was hitting them too quickly with the initialization messages. They didn't have time to recognize and react to them. A couple of sleep calls took care of that problem. I used the same setup on the XBee as when I hacked into the Smart Switch, you can look at it here <link>

Once running, you can press a switch on the key fob and it will transmit a messsage telling you that the button was pressed or released along with a counter that represents the time in milliseconds off the action. By subtracting the released time from the pressed time you can look for a long press as opposed to a short press. Nice feature.

The door switch has two switches, a tamper switch that will tell you if the cover is taken off and a reed switch that tells you the door has been opened. The tamper switch will tell you press and release like the key fob does, as will the reed switch. This means you can tell if the door is opened or closed as well as the cover off or on.

Like the smart switches, the key fob and door switch once joined, stay that way. You can pull the battery and put it back in without it having a problem. You can also drive away out of range and come back and everything works just fine. If you kill the coordinator by pulling the power or something, you might have to make them rejoin. This all depends on how long it was off and what happened during the off period. Sometimes the coordinator XBee will choose a different channel and then the switches will have trouble finding it. A simple power failure won't cause any problem, but reloading software to the controller could mean a visit to the devices to force them to rejoin. This kind of problem can be avoided by using the parameters on the XBee controller to limit it to a single predetermined channel. Doing this means the XBee comes up, establishes a network on the channel and the devices then just continue on as before.

So, you're going to get a lot of data as you add devices. Each device will interact at least every couple of minutes and every time you mess with them. For me, this meant missed messages. The traffic was pretty steady, but things didn't look right until I added a queue to the code to hold the messages and took them off one at a time to deal with. There's plenty of unused time to handle the messages, but they tend to come in in bunches and can cause problems. Adding a queue seemed to help that problem a lot; I don't see any missing or partial messages when I run it.

There's lots of debug and logging in the code as well as a test to help you isolate a single device. You'll see it in the code; I look for a list of devices to listen to and ignore the others. I had to do this to keep the other devices on the network from confusing me when I was trying figure out which bit meant what. Look for the test at the top of the code to decode the messages and adjust it as you need to. Also, this doesn't save anything from one run to the next. There's no database of devices that gets updated and the state of the devices isn't saved to compare with later. Once you're comfortable with the code, simply add whatever you want to handle things. I'm going to hook this code in with my database of house devices to save the states and decide exactly what I want to do with it later when I have a little more experience.

Here's the code:

#! /usr/bin/python

''' 
This is a hack into the operation of the Iris Smart Home Smart
Plug, Door Switch, and Key Fob. The evolution of this is discussed
on Desert-Home.com in detail.

Have fun
'''

from apscheduler.schedulers.background import BackgroundScheduler
from xbee import ZigBee 
import logging
import datetime
import Queue
import time
import serial
import sys, traceback
import shlex
from struct import *

def printData(data):
    print "********** Message Contents"
    for key, value in data.iteritems():
        if key == "id":
            print key, value
        else:
            print key, "".join("%02x " % ord(b) for b in data[key])
    print "**********"
    
def clusterData(lAddr,clusterId, data):
    print int(time.time()),
    print "".join("%02x" % ord(b) for b in lAddr) + \
        " clid "+"%04x" % clusterId + "-" + \
        "".join("%02x " % ord(b) for b in data)

# this is a call back function.  When a message
# comes in this function will get the data
# I had to use a queue to make sure there was enough time to 
# decode the incoming messages. Otherwise, in heavy traffic
# periods, I'd get a new message while I was still working on 
# the last one.
def messageReceived(data):
    #print "queueing message"
    messageQueue.put(data)

def handleMessage (data):
    try:
        ''' 
        I have a network of these devices and had to
        add this test to keep the log down to a reasonable
        size. If all the devices show up, it's hard to 
        tell what is going on. Just uncomment the check
        and put whatever you want to watch in.
        '''
            
        '''if data['source_addr_long'] not in \
            ['\x00\x0d\x6f\x00\x03\xc2\x71\xcc',
             '\x00\x0d\x6f\x00\x02\x83\xfa\x4e']:
            return'''
            
        #print ''
        #print 'gotta packet',
        #printData(data)
        if (data['id'] == 'rx_explicit'):
            #print "RX Explicit"
            #printData(data)
            clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
            #print 'Cluster ID:', hex(clusterId),
            #print "profile id:", repr(data['profile'])
            
            if (data['profile']=='\x00\x00'): # The General Profile
                print 'Cluster ID:', hex(clusterId),
                print "profile id:", repr(data['profile'])
                if (clusterId == 0x0000):
                    print ("Network (16-bit) Address Request")
                    #printData(data)
                elif (clusterId == 0x0004):
                    # Simple Descriptor Request, 
                    print("Simple Descriptor Request")
                    #printData(data)
                elif (clusterId == 0x0005):
                    # Active Endpoint Request, 
                    print("Active Endpoint Request")
                    #printData(data)
                elif (clusterId == 0x0006):
                    print "Match Descriptor Request"
                    #printData(data)
                    time.sleep(2)
                    print "Sending match descriptor response"
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x80\x06',
                        profile = '\x00\x00',
                        options = '\x01',
                        data = '\x04\x00\x00\x00\x01\x02'
                    )
                    # The contact switch is a bit slow, give it 
                    # some time to digest the messages.
                    time.sleep(2)
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x02',
                        dest_endpoint = '\x02',
                        cluster = '\x00\xf6',
                        profile = '\xc2\x16',
                        data = '\x11\x01\xfc'
                        )
                    time.sleep(2)
                elif (clusterId == 0x0008):
                    # I couldn't find a definition for this 
                    print("This was probably sent to the wrong profile")
                elif (clusterId == 0x0013):
                    # This is the device announce message.
                    print 'Device Announce Message'
                    # this will tell me the address of the new thing
                    # so I'm going to send an active endpoint request
                    print 'Sending active endpoint request'
                    epc = '\xaa'+data['source_addr'][1]+data['source_addr'][0]
                    print "".join("%02x " % ord(b) for b in epc)
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x00\x05',
                        profile = '\x00\x00',
                        options = '\x01',
                        data = epc
                    )

                    #printData(data)
                elif (clusterId == 0x8000):
                    print("Network (16-bit) Address Response")
                    #printData(data)
                elif (clusterId == 0x8038):
                    print("Management Network Update Request");
                elif (clusterId == 0x8005):
                    # this is the Active Endpoint Response This message tells you
                    # what the device can do
                    print 'Active Endpoint Response'
                    printData(data)
                elif (clusterId == 0x8004):
                    print "simple descriptor response"
                else:
                    print ("Unimplemented Cluster ID", hex(clusterId))
                    print
            elif (data['profile']=='\xc2\x16'): # Alertme Specific
                if data['source_addr_long'] not in devices:
                    devices.setdefault(data['source_addr_long'], []).append(data['source_addr'])
                # suppress printing for known clusters
                # so I can look at it more closely
                if clusterId not in [0X0500, 0x00ef, 0x00f0, 0x00f2, 0x00f3,0x00f6]:
                    printData(data)
                    print "Unhandled Message"
                if (clusterId == 0xef):
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    clusterCmd = ord(data['rf_data'][2])
                    status = data['rf_data'] # cut down on typing
                    if (clusterCmd == 0x81):
                        usage = unpack('<H', status[3:5])[0]
                        print " Current Usage:", usage
                    elif (clusterCmd == 0x82):
                        usage = unpack('<L', status[3:7])[0] / 3600
                        upTime = unpack('<L', status[7:11])[0]
                        print (" Switch Minute Stats: Usage, %d Watt Hours; Uptime, %d Seconds" %(usage/3600, upTime))
                elif (clusterId == 0x00f6):
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    print ''
                    print "Identify Message"
                    print "Sending init message"
                    zb.send('tx_explicit',
                       dest_addr_long = data['source_addr_long'],
                       dest_addr = data['source_addr'],
                       src_endpoint = '\x00',
                       dest_endpoint = '\x02',
                       cluster = '\x00\xf0',
                       profile = '\xc2\x16',
                       data = '\x19\x41\xfa\x00\x01'
                    )
                elif (clusterId == 0x00f3):
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    print ' Key Fob Button',
                    status = data['rf_data'] 
                    print ord(status[3]),
                    if status[2] == '\x01':
                        print 'Closed',
                    elif status[2] == '\x00':
                        print 'Open',
                    else:
                        print 'Unknown',
                    print 'Counter', unpack('<H',status[5:7])[0],
                    print ''
                    pass
                elif (clusterId == 0x00f2):
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    print 'Tamper Switch Changed State'
                    pass
                elif (clusterId == 0x00f0):
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    # If the cluster cmd byte is 'xfb', it's a status
                    if data['rf_data'][2] == '\xfb':
                        status = data['rf_data'] # just to make typing easier
                        if status[3] == '\x1f':
                            print " Door Sensor",
                            print str(float(unpack("<h", status[8:10])[0])\
                                / 100.0 * 1.8 + 32) + "F",
                        elif status[3] == '\x1c':
                            #  Never found anything useful in this
                            print "Power Switch",
                        elif status[3] == '\x1d':
                            print " Key Fob",
                            print str(float(unpack("<h", status[8:10])[0])\
                                / 100.0 * 1.8 + 32) + "F",
                            unpack('<I',status[4:8])[0]
                            print 'Counter', unpack('<I',status[4:8])[0],
                        elif status[3] == '\x1e':
                            # I haven't figured out what this is yet
                            # it comes from a door switch and the temperature
                            # field is always ff ff, it may be an error 
                            # indication.
                            pass
                        else:
                            print " Don't know this device yet",
                        print ''
                    pass
                elif (clusterId == 0x0500): # This is the security cluster
                    clusterData(data['source_addr_long'],clusterId,data['rf_data'])
                    # When the switch first connects, it come up in a state that needs
                    # initialization, this command seems to take care of that.
                    # So, look at the value of the data and send the command.
                    if data['rf_data'][3:7] == '\x15\x00\x39\x10':
                        print "sending initialization"
                        zb.send('tx_explicit',
                            dest_addr_long = data['source_addr_long'],
                            dest_addr = data['source_addr'],
                            src_endpoint = '\x00',
                            dest_endpoint = '\x00',
                            cluster = '\x05\x00',
                            profile = '\xc2\x16',
                            data = '\x11\x80\x00\x00\x05'
                        )
                    # The switch state is in byte [3] and is a bitfield
                    # bit 0 is the magnetic reed switch state
                    # bit 3 is the tamper switch state
                    switchState = ord(data['rf_data'][3])
                    if switchState & 0x04:
                        print 'Tamper Switch Closed',
                    else:
                        print 'Tamper Switch Open',
                    if switchState & 0x01:
                        print 'Reed Switch Opened',
                    else:
                        print 'Reed Switch Closed',
                    print ''
                    pass
            else:
                print ("Unimplemented Profile ID")
        elif(data['id'] == 'route_record_indicator'):
            print("Route Record Indicator")
        else:
            print("some other type of packet")
            print(data)
    except:
        print "I didn't expect this error:", sys.exc_info()[0]
        traceback.print_exc()

def showDevices():
    print "Known Devices ************"
    for key in devices:
        print "".join("%02x " % ord(b) for b in key)+':',
        print "".join("%02x " % ord(b) for b in devices[key][0])
    print "**************************"

def sendSwitch(whereLong, whereShort, srcEndpoint, destEndpoint, 
                clusterId, profileId, clusterCmd, databytes):
    payload = '\x11\x00' + clusterCmd + databytes
    zb.send('tx_explicit',
        dest_addr_long = whereLong,
        dest_addr = whereShort,
        src_endpoint = srcEndpoint,
        dest_endpoint = destEndpoint,
        cluster = clusterId,
        profile = profileId,
        data = payload
        )

def tryCommand():            
    # Try out commands here
    print '********* Sending Test Command'
    switchLongAddr ='\x00\x0d\x6f\x00\x03\xc2\x71\xcc'
    if switchLongAddr in devices:
        switchShortAddr = devices[switchLongAddr][0]
        print '     Switch Status'
        sendSwitch(switchLongAddr, switchShortAddr, '\x00', '\x02', 
            '\x00\xee', '\xc2\x16', '\x01', '\x01')
    else:
        print 'Waiting for short address'
    
    
    
#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
ZIGBEEBAUD_RATE = 9600
# Open serial port for use by the XBee
ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)


# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

#-------------------------------------------------
logging.basicConfig()

    
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)
# create a queue to put the messages into so they can
# be handled in turn without one interrupting the next.
messageQueue = Queue.Queue(0)

# A dictionary to put devices into as they show up
devices = {}
scheditem = BackgroundScheduler()
scheditem.add_job(showDevices, 'interval', seconds=60)
#scheditem.add_job(tryCommand, 'interval', seconds=15)
scheditem.start()

print ("started")
notYet = True;
firstTime = True;
while True:
    try:
        if (firstTime):
            # this is in case I need some initialization in the
            # future
            firstTime = False
        if messageQueue.qsize() > 0:
            #print "getting message"
            message = messageQueue.get()
            handleMessage(message)
            messageQueue.task_done();
        
        time.sleep(0.1)
        #print ("tick") # This is here to let you know it's alive
        
        sys.stdout.flush() # if you're running non interactive, do this

    except IndexError:
        print "empty line"
    except NameError as e:
        print "NameError:",
        print e.message.split("'")[1]
    except KeyboardInterrupt:
        print ("Keyboard interrupt")
        break
    except:
        print ("I didn't expect this error:", sys.exc_info()[0])
        traceback.print_exc()
        break
print ("After the while loop")
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
scheditem.shutdown(wait=False) # shut down the apscheduler
zb.halt()
ser.close()

Yes, it's a bit complex. More complex than other hacks I've posted. I had to do this to allow for multiple devices that had multiple endpoints and clusters involved. It still isn't too hard to understand, and you should be fine when you start messing with it.

You'll notice that I keep a list of devices that have contacted the code. This is to allow you to tell what happened with various devices as the interaction continues. I log the long address, time and relevant details as they happen to help you understand how to deal with the incoming data. When you have several devices, the log can become a bit overpowering since nothing stops sending, it just goes on forever. Here's a bit of the logging so you can get an idea what it looks like:

started
1432598997 000d6f0004510782 clid 00f0-09 9e fb 1f bf 9e 7a 0a b7 0b c4 01 af f4 03 02 
 Door Sensor 85.982F 
1432598998 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432598999 000d6f000283fa4e clid 00f0-09 a2 fb 1d 3c a2 e5 03 bc 0b 00 00 a7 fb 03 00 
 Key Fob 86.072F Counter 65380924 
1432599000 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599000 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 dd 95 83 66 32 00 00 b5 b0 01 00 
Power Switch 
1432599000 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599003 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599005 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599007 000d6f0002547a5d clid 00f0-09 00 fb 1c 2b 09 b3 1a 66 32 00 00 b7 fe 01 00 
Power Switch 
1432599008 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599009 000d6f00025886b3 clid 00ef-09 00 82 67 a4 a5 72 80 e5 20 01 00 
 Switch Minute Stats: Usage, 148 Watt Hours; Uptime, 18933120 Seconds
1432599010 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599010 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599013 000d6f000258a4cc clid 00ef-09 00 82 f7 ea 21 00 b8 bc 03 00 00 
 Switch Minute Stats: Usage, 0 Watt Hours; Uptime, 244920 Seconds
1432599013 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599015 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599018 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599020 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599020 000d6f000237b25a clid 00ef-09 00 82 a9 3d 8a 1f 40 e9 20 01 00 
 Switch Minute Stats: Usage, 40 Watt Hours; Uptime, 18934080 Seconds
1432599020 000d6f0003cf0e5b clid 00f0-09 00 fb 1c 34 a9 6e 64 fb 31 00 00 ba b7 01 00 
Power Switch 
1432599020 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599023 000d6f000237b25a clid 00f0-09 00 fb 1c 25 0d a5 83 4a 32 00 00 b4 b4 01 00 
Power Switch 
1432599023 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599025 000d6f0002547a5d clid 00ef-09 91 82 40 35 ff 00 d4 ac 06 00 00 
 Switch Minute Stats: Usage, 1 Watt Hours; Uptime, 437460 Seconds
1432599025 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599025 000d6f000258a4cc clid 00f0-09 00 fb 1c 18 11 f3 0e 66 32 00 00 bd b3 01 00 
Power Switch 
1432599028 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599030 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599030 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 55 96 83 66 32 00 00 b6 b6 01 00 
Power Switch 
1432599030 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599033 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599035 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599036 000d6f0002547a5d clid 00f0-09 00 fb 1c 2b 81 b3 1a 66 32 00 00 b7 fd 01 00 
Power Switch 
1432599038 000d6f0003cf0e5b clid 00ef-09 00 82 38 42 54 00 bc 1b 19 00 00 
 Switch Minute Stats: Usage, 0 Watt Hours; Uptime, 1645500 Seconds
1432599038 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599040 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599040 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599043 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599045 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599048 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599050 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599050 000d6f0003cf0e5b clid 00f0-09 00 fb 1c 34 21 6f 64 fb 31 00 00 ba bb 01 00 
Power Switch 
1432599050 000d6f000237b25a clid 00ef-09 00 81 01 00 
 Current Usage: 1
1432599053 000d6f000237b25a clid 00f0-09 00 fb 1c 25 85 a5 83 4a 32 00 00 b4 a4 01 00 
Power Switch 
1432599054 000d6f000258a4cc clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599055 000d6f0002547a5d clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599055 000d6f000258a4cc clid 00f0-09 00 fb 1c 18 89 f3 0e 66 32 00 00 bd b1 01 00 
Power Switch 
Known Devices ************
00 0d 6f 00 02 83 fa 4e : 92 26 
00 0d 6f 00 04 51 07 82 : a9 3b 
00 0d 6f 00 02 58 a4 cc : e1 3c 
00 0d 6f 00 02 58 86 b3 : 27 f6 
00 0d 6f 00 02 37 b2 5a : 2b d1 
00 0d 6f 00 02 54 7a 5d : 41 c7 
00 0d 6f 00 03 cf 0e 5b : 79 c5 
**************************
1432599058 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 
 Current Usage: 0
1432599060 000d6f00025886b3 clid 00ef-09 00 81 87 00 
 Current Usage: 135
1432599060 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 cd 96 83 66 32 00 00 b6 c1 01 00 
Power Switch 

The first field is the Unix timestamp. Basically the number of seconds since 1970. The second is the long address of the device; this never changes. Then the cluster id followed by the data specific to the cluster. I decoded the times for the key fob and the temperature for the door switch because those could be fun to play with. Notice that the smart switches really send a lot of data, this will help you understand why I limit the number of devices with a test at the top of the code. If everything is there, you have a heck of a time picking out what you want to see. The list of know devices is the long address and the currently assigned short address. The short address will change from time to time as you restart the process or pull a battery somewhere.

Remember, this runs on a Raspberry Pi, I haven't tried it on anything else. I have my XBee plugged into a USB port on the Pi, if you're using the serial port, just change the port in the code, that should be all that is needed. If you don't know which usb port to use, there's a ton of articles on the web that will help you find the port, just look around a bit, or to see how I did it look here <link>. What I'm hoping for is a couple of folk that want to use these nicely priced and well made devices with their own code for some project to grab it and give me some feedback on the interaction. It would be nice to see someone else running this code.

Have fun.

The next part to this project is here <link>

Saturday, May 23, 2015

Hacking Into The Iris Door Sensor, Part 2

The previous post on this project is here <link>

Yes, I've made some progress on this project, but first let me talk about why I'm bothering with this. A couple of folk have asked me why I hack into these devices in general; the answer is not an esoteric, "Because they're there." I enjoy the challenge, but mostly it's because I want to use them without a bunch of strings attached.

The Iris devices are fairly priced, but not if you have to subscribe to a service that cost $10+ a month for the rest of time. From dismantling them I can tell the construction is actually pretty well done even though I keep seeing complaints on the web about how they should be smaller, prettier or come in colors. I couldn't build one as nice for the price. Actually, I couldn't build one as nice for twice the price. The single disadvantage is that their usefulness is tied to a web site that is run by a corporation that can change their mind at any time about anything and leave me hanging. Been there, done that, want to avoid it in the future.

Enough philosophy, I'm going to talk about the Iris Key Fob a bit. When I got some reaction from the door switch, I wondered if the key fob would work in a similar way, so since I was going to town anyway, I stopped off at Lowe's and picked up another door switch (I have two now) and a key fob as well.

Edit: I originally forgot to put up a picture it the Key Fob assembled, here it is:



Here it is with the cover off.


Same good construction as the door switch with a pair of switches that you press to activate funtions remotely. Let's pull it apart.


This is the case with the board removed to show how the plastic that operates the buttons is done. This makes for a nice feel when you're pushing the switch. The only problem I saw is that it's easy to press both switches if you squeeze it. A little care can overcome that problem though. Here's one side of the board,


and this is the other


You can click on the picture and zoom in if you want.

The switches have tactile feedback so you know you pressed it and the battery seats firmly so you know it's in there. Nice switch, albeit a bit large for some people's taste.

This switch was designed to control the Iris system in a home. If you push one button, it arms the alarm, and the other switch disarms it. It also provides feedback to the system that the fob is in range. People describe it as the Iris Hub sends messages to the fob every two minutes and if it doesn't get a response, the hub arms the alarm. My reading of the messages that fly back and forth shows that the fob reports every two minutes to the hub, so that would mean two minutes without a message and the hub will arm itself. There may be something I'm missing, but I'll turn it up eventually.

I see this device as two remote control switches that I can do any darn thing I want with. It could be on and off for a machine, light, ceiling fan, whatever I want. I can have one in each room attached to the bed for ceiling lights, or turn off the smoker in the yard if I want to.

I can hear my readers saying, "Fine, nice pictures and philosophy; tell us how the interaction works and give us some code." Well, I'm going to make you wait a couple of days before I bury you in code because this gets somewhat complicated and I'm not sure I've got enough bugs out to trust it yet. I have two door switches, a key fob and five smart switches running on the same XBee controller quite reliably, but a reader who 'volunteered' to try the code also has had nothing but problems.

Where my stuff joins and just works, his decided to get flaky and mess with his mind. We're trading logs and code back and forth to try and figure out what's different.

More later, but you got a look at the internals of a device, so stop complaining.

The next post on this project is here <link>

Monday, May 11, 2015

Yet Another Update to the AcuRite 5N1 Weather Sensor

The previous post on this project is here <link>

Just when you thought I'd finally shut up about the AcuRite weather sensor I decide to post again. I promise, at some point I'll actually be satisfied with this device, but not quite yet. Over the months I've been working with this I've received comments and emails from folk that tried it and discovered things. One reader had an especially interesting situation; he had a neighbor that had a sensor within range. This meant that the code would pick them both up and present data from both sensors. To add insult to injury, when he tried changing his station to channel C, he found out that rtl_433 had a bug that prevented his receiving data on that channel. He must have been really annoyed at that point.

Since wind and temperature matter to placement, this was causing him grief and he set off to fix it. He isolated the sensor ID, and while he was at it, decoded the battery level, channel and found a better way to read the RF that allowed channel C to work.

During his trials he also found that the checksum was allowing bad data into the decoding and would present unusual readings. I've seen this over time as well. Things like a wind gust of 147 mph, a temperature spike of 30 or so degrees for no reason, the usual outlier readings, but with weather, it's hard to tell what an outlier really is.

I totally stole his ideas and some of his code and incorporated into my weather station,; I haven't been running it too long, but it looks like the vast majority of the outlier readings are gone. There may be some turn up, but so far it's been good.

First, inspection of the data showed that the bytes holding the various readings were all even parity. I added code to check the parity and discard the message if it didn't pass. Note that I only check the readings bytes because the first, second and last bytes are sensor, status, and checksum; these don't incorporate parity. In the c code I add the decoded sensor id, battery level, and channel to the JSON string I output, then check for the proper values in the python code that receives it. I did it this way because some folk may want to catch multiple sensors and average or compare them; especially if they can see a neighbors device for free.

I also noticed at my house that I receive a couple of devices from other folk that are not AcuRite devices. These things can cause problems as well, especially since they could easily be using the same checksum algorithm.

All the changes meant that I had to get the very latest version of rtl_433 from the original github repository. This was required because the rtl_433 folk made a new decoding method available that works better with the AcuRite data stream. That decoder wasn't there in January, but it is now. So, I tried again to fork it and failed miserably, and wound up just updating my copy of rtl_433. But, another good thing happened as well. The folk that wrote rtl_sdr, the actual radio software have improved it a lot and it installed first time, and I got to remove that from my copy.

Now, you get rtl_sdr, install it, build it, and then get my rtl_433, install it and build it. That will give you the version I'm using with all the changes to support the newest findings. My python code for the weather station I run is in GitHub as well. The specific stuff I use is:

rtl_sdr <link>
rtl_433 <link>
weather station <link>

The reason I linked to a particular commit in GitHub is so you can have exactly the same code to work with. The two rtl projects are in flux and could change dramatically at any time. You can always get the latest version if you think it will serve you better.

The latest version of rtl_433 not only has the new decoder software, it also allows you to specify the device you expect to receive. That makes it nice for me since I only have the AcuRite device I'm interested in. I changed my start up command to support this feature:

rtl_433 -f 433.915e6 -R9 2>> saveweather.log | saveweather.py  >> saveweather.log 2>&1

The '-R9' specifies the 5N1 device. Notice how I save the stderr output to a file? That helps me understand how many errors I'm getting and could help me fix a problem. Right now I'm also using the -D for debug option as well to get more information. Try '--help' on rtl_433 to see the other options it has.

I owe this latest version to Pcjunky (yes, I know his name, but I ain't gonna post it without his permission). He took my code and beat on it until it worked to his satisfaction. Thank's a lot.

Now, I want to show you the parity routine I ran across. Here's the code:

static int acuriteParity(uint8_t v){
    // returns 1 if parity odd, 0 if parity even
    v ^= v >> 4;
    v &= 0xf;
    return (0x6996 >> v) & 1;
}

This calculates the parity for a byte in around five cpu operations. It's an extremely clever implementation that I leave the explanation as an exercise for the student.

Have fun.


Saturday, May 9, 2015

Hacking Into The Iris Door Sensor

Several people have asked me about the Iris Door Sensor, and I haven't had much interest. My battery operated switch can do this and is totally programmable, but I figured, "What the heck." I went ahead and got one when I went to Lowe's to get another smart switch.

Now, this is one of those projects that will go on and on as I try to get it to work. Meanwhile, these are the first steps.

This is what one looks like:


I don't think Lowe's will mind that I stole the picture from their site. First thing I did was open it up and look inside:


The spot on the left is for the battery and there's a switch for joining it to a network. I tried to get it to join with my Iris network, but it didn't work, so being me, I took it apart. Here's the circuit board removed from the casing:


Not much on this side, so flipping it over:


This is a side view of the magnet it comes with:


Remember those little extrusions on the magnet case; I'll talk about them later. I got out the magnifying glass and took a look at the active components, the list isn't very long:

Battery - CR2 Lithium, 3V
Magnetic Reed Switch - Hamlin 59050-1-S-00-0 Normally Open
Magnet off center to be near the reed switch.
Antenna - Antenova a6150
Processor - Silicon Labs Ember 250
Temperature Sensor - Microchip MCP9801M

Nice set of high quality components. this thing is put together pretty well. The important part is the Ember 250 SOC (system on a chip) that runs the device. These are made specifically for ZigBee and are very nice devices; to bad the product comes from Alertme. They always ignore the ZigBee spec and go their own way in critical areas. This is going to make it tough to hack into.

I put together some code and turned off my normal network to keep down interference and gave it a try. I couldn't get it to join properly, but I was able to see it trying. Since this is an Alertme device, it has significant API differences from the ZigBee spec, so I started trying things. I got it to join eventually by a non-reproducible accident and saw it running.

The problem is that I can't get that to happen again. When it was working, I noticed that it was really sensitive to where the magnet was, so I looked at the reed switch specs and it's only got a 7 mm range. That means that the magnet needs to be right next to the sensor for it to close. The FAQ for the switch on the Lowe's site even talks about that. They blame it on polarity when it's really the fact that the magnet needs to be so close to the sensor; like polarity would matter on a reed switch. A neo magnet might make the range longer, but it wouldn't be as pretty.

Some of us may have problems with that on installation; seven millimeters is not very much, especially when it's about 4 mm from the switch to the case. Do you see why they have those little protusions on the case now? It gives some spacing and a little place for the magnet to rub on the sensor case when the door closes. Strange way to build it, but I'm not an engineer.

About how it works: When the battery is first inserted, it starts sending Device Announce messages and continues this until I answer it with an Active Endpoint  Request (which it hasn't answered yet). When it gets the Active Endpoint, it sends a Match Descriptor request and I answer it with a Match Descriptor Response tailored to it. Then, it sends one of the odd Alertme messages to profile 0xC216 that I've seen before from Alertme devices.

This is the point where, with the smart switches, I send a set of canned responses back and the device would join. The door switch doesn't join, It will stay around for a while trying, but will eventually give up and move on to another channel to look for another controller.

The time I got it to join, it started sending messages to cluster 0x500, the security device cluster. While it was doing that I could actually see the switch change state because the values it sent back were changing based on magnet location. However, it was erratic and extremely slow at sensing the magnet. There was some LED flashing on the device that seemed to correspond to the position of the magnet, but I couldn't be sure.

I'm going to post the code I'm working with as it is right now, but it's not pretty. I hacked up a piece of code I developed to break into another ZigBee device and am making experimental changes to it trying to understand this door switch.

If you grab it, there's an 'if' statement up at the top where I get the packets from the XBee, I check the incoming source address and skip anything that isn't coming from the switch. I have a number of the Iris devices and they were messing up the experiment, so I just ignore them in the code. You'll have to change that to your own devices address.

#! /usr/bin/python
# Have fun

from xbee import ZigBee 
import logging
import datetime
import time
import serial
import sys, traceback
import shlex
from struct import *
'''
Before we get started there's a piece of this that drove me nuts.  Each message to a 
Zigbee cluster has a length and a header.  The length isn't talked about at all in the 
Zigbee documentation (that I could find) and the header byte is drawn backwards to
everything I've ever dealt with.  So, I redrew the header byte so I could understand
and use it:

7   6   5   4   3   2   1   0
            X                   Disable Default Response 1 = return default message
                X               Direction 1 = server to client, 0 = client to server
                    X           Manufacturer Specific 
                            X   Frame Type 1 = cluster specific, 0 = entire profile
                                    
So, to send a cluster command, set bit zero, and to get an attribute from a cluster
set bit 4.  If you want to be sure you get a reply, set the default response.  I haven't
needed the manufacturer specific bit yet.

'''

def printData(data):
    print "********** Message Contents"
    for key, value in data.iteritems():
        if key == "id":
            print key, value
        else:
            print key, "".join("%02x " % ord(b) for b in data[key])
    print

def getAttributes(data, thisOne):
    ''' OK, now that I've listed the clusters, I'm going to see about 
    getting the attributes for one of them by sending a Discover
    attributes command.  This is not a ZDO command, it's a ZCL command.
    ZDO = ZigBee device object - the actual device
    ZCL = Zigbee device cluster - the collection of routines to control it.
        Frame control field, bit field,  first byte
            bits 0-1, frame type field
                00 entire profile
                01 specific to a particular cluster
                10-11 reserved (don't use)
                Note, if you're sending commands, this should be 01
                if you're reading attributes, it should be 00
            bit 2, manufacturer specific, if this bit is set, include
                   the manufacturer code (below)
            bit 3 direction, this determines if it is from a client to 
                   server.  
                 1 server to client
                 0 client to server
                 Note, server and client are specific to zigbee, not
                 the purpose of the machine, so think about this.  For
                 example to turn an on/off switch on, you have to be the
                 server so this bit will be 01
            bit 4 disable default response
            bits 5-7 reserved (set to zero)
        Manufacturer code,  either 2 bytes or not there 
        Transaction sequence number, byte
        Command identifier, byte
        Frame payload,  variable, the command bytes to do something
        
        frame control bits = 0b00 (this means a BINARY 00)
        manufacturer specific bit = 0, for normal, or one for manufacturer
        So, the frame control will be 0000000
        discover attributes command identifier = 0x0c
        
        then a zero to indicate the first attribute to be returned
        and a 0x0f to indicate the maximum number of attributes to 
        return.
    '''
    print "Sending Discover Attributes"
    zb.send('tx_explicit',
        dest_addr_long = data['source_addr_long'],
        dest_addr = data['source_addr'],
        src_endpoint = '\x00',
        dest_endpoint = '\x01',
        cluster = thisOne, # cluster I want to know about
        profile = '\x01\x04', # home automation profile
        # means: frame control 0, sequence number 0xaa, command 0c,
        # start at 0x0000 for a length of 0x0f
        data = '\x00' + '\xaa' + '\x0c'+ '\x00' + '\x00'+ '\x0f'
        )

# this is a call back function.  When a message
# comes in this function will get the data
def messageReceived(data):
    
    try:
        # This is the long address of my door switch device
        # since I have several other devices and they are transmitting
        # all the time, I'm excluding them and only allowing the
        # door switch in
        if data['source_addr_long'] != '\x00\x0d\x6f\x00\x03\xc2\x71\xcc':
            return
            
        print 'gotta packet',
        #print data
        if (data['id'] == 'rx_explicit'):
            print "RF Explicit"
            #printData(data)
            clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
            print 'Cluster ID:', hex(clusterId),
            print "profile id:", repr(data['profile'])
            if (data['profile']=='\x01\x04'): # Home Automation Profile
                # This has to be handled differently than the general profile
                # each response if from a cluster that is actually doing something
                # so there are attributes and commands to think about.
                #
                # Since this is a ZCL message; which actually means this message is 
                # is supposed to use the ZigBee cluster library to actually do something
                # like turn on a light or check to see if it's on, the command way down
                # in the rf_data is important.  So, the commands may be repeated in
                # each cluster and do slightly different things
                #
                # I'm going to grab the cluster command out of the rf_data first so 
                # I don't have to code it into each cluster
                #print "take this apart"
                #print repr(data['rf_data'])
                if (data['rf_data'][0] == '\x08'): # was it successful?
                    #should have a bit check to see if manufacturer data is here
                    cCommand = data['rf_data'][2]
                    print "Cluster command: ", hex(ord(cCommand))
                else:
                    print "Cluster command failed"
                    return
                # grab the payload data to make it easier to work with
                payload = data['rf_data'][3:] #from index 3 on is the payload for the command
                datatypes={'\x00':'no data',
                            '\x10':'boolean',
                            '\x18':'8 bit bitmap',
                            '\x20':'unsigned 8 bit integer',
                            '\x21':'unsigned 24 bit integer',
                            '\x30':'8 bit enumeration',
                            '\x42':'character string'}
                print "Raw payload:",repr(payload)
                # handle these first commands in a general way
                if (cCommand == '\x0d'): # Discover Attributes
                    # This tells you all the attributes for a particular cluster
                    # and their datatypes
                    print "Discover attributes response"
                    if (payload[0] == '\x01'):
                        print "All attributes returned"
                    else:
                        print "Didn't get all the attributes on one try"
                    i = 1
                    if (len(payload) == 1): # no actual attributes returned
                        print "No attributes"
                        return
                    while (i < len(payload)-1):
                        print "    Attribute = ", hex(ord(payload[i+1])) , hex(ord(payload[i])),
                        try:
                            print datatypes[payload[i+2]]
                            i += 3
                        except:
                            print "I don't have an entry for datatype:", hex(ord(payload[i+2]))
                            return
                            
                if (clusterId == 0x0000): # Under HA this is the 'Basic' Cluster
                    pass
                elif (clusterId == 0x0003): # 'identify' should make it flash a light or something 
                    pass
                elif (clusterId == 0x0004): # 'Groups'
                    pass
                elif (clusterId == 0x0005): # 'Scenes'  
                    pass
                elif (clusterId == 0x0006): # 'On/Off' this is for switching or checking on and off  
                    print "inside cluster 6"
                
                elif (clusterId == 0x0008): # 'Level'  
                    pass
                else:
                    print("Haven't implemented this yet")
            elif (data['profile']=='\x00\x00'): # The General Profile
                if (clusterId == 0x0000):
                    print ("Network (16-bit) Address Request")
                    #printData(data)
                elif (clusterId == 0x0004):
                    # Simple Descriptor Request, 
                    print("Simple Descriptor Request")
                    #printData(data)
                elif (clusterId == 0x0008):
                    # I couldn't find a definition for this 
                    print("This was probably sent to the wrong profile")
                elif (clusterId == 0x0013):
                    # This is the device announce message.
                    print 'Device Announce Message'
                    printData(data)
                    
                    print "sending Active Endpoint Request "
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x05\x00',
                        profile = '\xc2\x16',
                        # The first item is a number to identify the message
                        # The next two are the short address of the device
                        data = '\x12' + data['source_addr'][1]+ data['source_addr'][0]
                    )

                    
                elif (clusterId == 0x8000):
                    print("Network (16-bit) Address Response")
                    #printData(data)
                elif (clusterId == 0x8038):
                    print("Management Network Update Request");
                elif (clusterId == 0x8005):
                    # this is the Active Endpoint Response This message tells you
                    # what the device can do
                    print 'Active Endpoint Response'
                    printData(data)
                    if (ord(data['rf_data'][1]) == 0): # this means success
                        print "Active Endpoint reported back is: {0:02x}".format(ord(data['rf_data'][5]))
                        
                    print("Now trying simple descriptor request on an endpoint")
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00', # This has to go to endpoint 0 !
                        cluster = '\x05\x00', #simple descriptor request'
                        profile = '\xc2\x16',
                        data = '\x13' + data['source_addr'][1] + data['source_addr'][0] + '\x01'
                    )
                elif (clusterId == 0x8004):
                    print "simple descriptor response"
                    try:
                        clustersFound = []
                        r = data['rf_data']
                        if (ord(r[1]) == 0): # means success
                            #take apart the simple descriptor returned
                            endpoint, profileId, deviceId, version, inCount = \
                                unpack('<BHHBB',r[5:12])
                            print "    endpoint reported is: {0:02x}".format(endpoint)
                            print "    profile id:  {0:04x}".format(profileId)
                            print "    device id: {0:04x}".format(deviceId)
                            print "    device version: {0:02x}".format(version)
                            print "    input cluster count: {0:02x}".format(inCount)
                            position = 12
                            # input cluster list (16 bit words)
                            for x in range (0,inCount):
                                thisOne, = unpack("<H",r[position : position+2])
                                clustersFound.append(r[position+1] + r[position])
                                position += 2
                                print "        input cluster {0:04x}".format(thisOne)
                            outCount, = unpack("<B",r[position])
                            position += 1
                            print "    output cluster count: {0:02x}".format(outCount)
                            #output cluster list (16 bit words)
                            for x in range (0,outCount):
                                thisOne, = unpack("<H",r[position : position+2])
                                clustersFound.append(r[position+1] + r[position])
                                position += 2
                                print "        output cluster {0:04x}".format(thisOne)
                            clustersFound.append('\x0b\x04')
                            print "added special cluster"
                            print "Completed Cluster List"
                    except:
                        print "error parsing Simple Descriptor"
                        printData(data)
                    print repr(clustersFound)
                    for c in clustersFound:
                        getAttributes(data, c) # Now, go get the attribute list for the cluster
                elif (clusterId == 0x0006):
                    print "Match Descriptor Request"
                    # Match Descriptor Request
                    printData(data)
                    # Now the Match Descriptor Response
                    print "Sending match descriptor response"
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x80\x06',
                        profile = '\x00\x00',
                        #seq #, status, address,num endp, list
                        data = '\x00\x00\x00\x00\x01\x02')
                else:
                    print ("Unimplemented Cluster ID", hex(clusterId))
                    print
            elif (data['profile']=='\xc2\x16'): # Alertme Specific
                printData(data)
                '''print "Sending weird messages"
                if (clusterId == 0x00f6):
                    payload3 = '\x11\x01\x01'
                    zb.send('tx_explicit',
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = data['source_endpoint'],
                        cluster = data['cluster'],
                        profile = '\xc2\x16',
                        data = payload3
                        )
                    payload4 = '\x19\x01\xfa\x00\x01'
                    zb.send('tx_explicit',
                       dest_addr_long = data['source_addr_long'],
                       dest_addr = data['source_addr'],
                       src_endpoint = '\x00',
                       dest_endpoint = data['source_endpoint'],
                       cluster = data['cluster'],
                       profile = '\xc2\x16',
                       data = payload4
                    )'''
                    
                if (clusterId == 0x00ef):
                    pass
                elif (clusterId == 0x00f0):
                    pass
            else:
                print ("Unimplemented Profile ID")
        elif(data['id'] == 'route_record_indicator'):
            print("Route Record Indicator")
        else:
            print("some other type of packet")
            print(data)
    except:
        print "I didn't expect this error:", sys.exc_info()[0]
        traceback.print_exc()
        
def sendSwitch(whereLong, whereShort, srcEndpoint, destEndpoint, 
                clusterId, profileId, clusterCmd, databytes):
    
    payload = '\x11\x00' + clusterCmd + databytes
    # print 'payload',
    # for c in payload:
        # print hex(ord(c)),
    # print
    # print 'long address:',
    # for c in whereLong:
        # print hex(ord(c)),
    # print
        
    zb.send('tx_explicit',
        dest_addr_long = whereLong,
        dest_addr = whereShort,
        src_endpoint = srcEndpoint,
        dest_endpoint = destEndpoint,
        cluster = clusterId,
        profile = profileId,
        data = payload
        )

#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
ZIGBEEBAUD_RATE = 9600
# Open serial port for use by the XBee
ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)


# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
theSwitch = '\x00\x0d\x6f\x00\x03\x58\x05\xc2'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

#-------------------------------------------------
logging.basicConfig()

    
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)

print ("started")
notYet = True;
firstTime = True;
while True:
    try:
        time.sleep(5)
        if (firstTime):
#           sendSwitch(whereLong=theSwitch, whereShort=UNKNOWN, srcEndpoint='\x00', 
#                   destEndpoint='\x00', clusterId='\x00\x00', profileId='\x00\x00', 
#                   clusterCmd='\x00', databytes='\x00')
#            print "sending Active Endpoint Request "
#            zb.send('tx_explicit',
#                dest_addr_long = theSwitch,
#                dest_addr = '\x94\x65',
#                src_endpoint = '\x00',
#                dest_endpoint = '\x00',
#                cluster = '\x00\x05',
#                profile = '\x00\x00',
#                # The first item is a number to identify the message
#                # The next two are the short address of the device
#                data = '\x12' + '\x65' + '\x94'
#            )
#            time.sleep(10)
#            print "sending 'configure reporting'"
#            zb.send('tx_explicit',
#                dest_addr_long = theSwitch,
#                dest_addr = '\x94\x65',
#                src_endpoint = '\x00',
#                dest_endpoint = '\x01',
#                cluster = '\x00\x06', # cluster I want to deal with
#                profile = '\x01\x04', # home automation profile
#                data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
#            )

            firstTime = False
        print ("tick")
        
        sys.stdout.flush() # if you're running non interactive, do this

    except KeyboardInterrupt:
        print ("Keyboard interrupt")
        break
    except:
        print ("I didn't expect this error:", sys.exc_info()[0])
        traceback.print_exc()
        break

print ("After the while loop")
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()

I told you it wasn't pretty, so stop complaining.

I'm continuing to play with this switch as time, and new ideas permit, so I think the door switch can be conquered. However, if is as flaky as the initial results would indicate, I may not be using it. Having to have the magnet right up against the sensor and having so little elbow room would stop the switch from being useful on things like garage doors, doors that move due to weather or settling, wooden windows that may not shut all the way, that kind of thing. If someone has one that works well in an Iris setup can tell us about this switch, it would be great.

If you want, grab the code, experiment, play, and let me know the results you get.

The next post on this project is here <link>.

Sunday, May 3, 2015

My New Network Attached Storage

Well, I did it. I finally broke down and bought a NAS (network attached Storage). I researched these things for days looking for something that would fit my needs and couldn't make up my mind. I was tempted to just take a Raspberry Pi and add a disk drive, but I wanted more capability in the device than that would have given me. I asked other folk, and it came down to two manufacturers of this kind of device: Synology and QNAP. Frankly, they both sound great and have more good reviews than bad.

I settled on QNAP, not because it won out in the specification, but because I caught a nice sale. TigerDirect had the QNAP TS-231 on sale for $30 bucks off, so I ordered it. I got two 3TB WD red drives from Amazon and set it up. It's up in my attic data center working its little heart out.

When I was looking at what could be run on it, MySQL was listed. That meant that I could put my house databases up there and get them off the Pi's I have running. It also meant that I could expand the database and actually log data locally for my house.

Wow, that would mean all my griping about cloud services would end.

First step, set up a MySQL database on the server; that took about 10 minutes. Then understand how to put data on it; that took a day. Then convert my code to use the new databases on the NAS; that took a stinking week. The very basic use of MySQL is different enough from the way to use SQLite3 that I had to learn a whole lot to get it working.

But, when I did, it really sped things up. I'm not clear on why exactly; I'm making data base transactions across an Ethernet link instead of locally, but it runs faster. Since the NAS is set up Raid 1, I have 3TB to play with and I expect the databases to use a hunk of that so it may slow down over time as the data bases grow.

My impetus for finally biting the bullet on one of these devices was storage failures on the Raspberry Pi. That's the most significant problem I've had with the Pi, the SD cards wear out, and I had worse luck using a USB stick. Well, I won't be wearing the cards out as fast, and I don't need the extra space of a USB stick anymore. However, the SD cards will still wear out because all the system logs and such are beating on them.

To this end, I'm going to put the operating system on the NAS as well, but I haven't gotten that far yet. The idea is to put everything I can on the NAS and only boot the Pi from the SD card. That should make it last a while. Sort of the best of both worlds; the separate processor and hardware of the Pi combined with the storage and reliability of a NAS. I'll inherit the possibility of things quitting if the Ethernet fails, but my house already relies on Ethernet anyway. I have the various Arduinos set up to work in a default mode, so that problem hasn't gotten any worse ... I think.

One of the side-effects (affects?? I can never remember which one to use) is that I cleaned up a bunch of code, removed things I really wasn't using any more and restructured a lot of my processes. So, most of the processes that wrote data to the various cloud services I looked into are gone now. The code is still out there on GitHub (the internet is forever), but it's not in the latest update. If you need it, just look at a previous commit.

Another thing that changed is since I'm not updating the various clouds, the live graphs I have scattered in earlier posts may not work any more. I replaced some of them with pictures to keep it somewhat coherent, but gave up after a few. Also, the architecture of the overall house control and monitoring is a bit different. I guess I'll delete the page on that and try to create a new picture and description soon.

I'm keeping my legacy feeds on Xively running for now since the majority of my data is there. I wouldn't mind losing that data, I'll make more, but I want to annoy them as long a possible with a personal feed. They've been hinting that they may remove the personal feeds for a while now, and I just plan on staying there until they kick me off.

But, this thing is so cool. It just works and accepts transactions without complaining at all. MySOL is way capable and can do things I hadn't even thought of.

So, Dave's place has Arduinos running things like the garage, pool and air conditioning that talk to a Raspberry Pi that saves it's status to a NAS. There's also a weather head that sends RF data to another Raspberry Pi that saves weather data cumulatively to the NAS as well.

The code for the new way of doing things is out on GitHub, if you want to grab something.

Unfortunately, this exercise has given me a bunch of ideas on things I can do to improve the operation. Now that I can essentially have unlimited Raspberry Pi devices doing things and coordinating activities of other devices, I can put the house configuration file on the NAS and read it from any Ethernet connected device, I can separate the web services from everything else and increase house Ethernet security. I can save data on each room using the new temperature sensor in various ways. I can ... I can go completely nuts,

If I haven't already.