I Got One of Those New Fangled Amazon Dot Things

Obviously, this is part 1.

Yes, I took the plunge and bought an Amazon Dot before it was available and waited patiently for the release date for it to arrive. When it came in, I did the usual, "Alexa, tell me the weather." "Alexa, what time is it?" "Alexa, how tall is Scarlett Johansson?" You know, the usual.

Then, I got out the laptop and started looking at how to connect it to my house.

Let me tell you, this thing is really, really cool, but it's also the most frustrating device I have worked on in quite a while. I literally spent hours and hours on the web trying to figure out Amazon's amazingly complex and poorly documented interface to IOT (internet of things). I tried a large number of examples and tutorials out there with a singular result: they didn't work. I knew it was possible, there are youtube videos that prove it, but not a single one of them worked for me. I wrote, copied, stole thousands of lines of code and didn't get a single signal of any kind at my house. So, in a mass of frustration and disappointment, I managed to stumble (this is the third full 16 hour day of trying) across a little note on rules. It started to work part way.

Emboldened by a tiny success, I dug in and conquered the device all the way to the house, out to my patio lights and all the way back to the Alexa sitting on my kitchen counter. I had it working. No, I haven't got all the devices tied in, but I can get the weather readings taken at my house and actually turn on and off my patio light.

The problem for me, and I suspect many others, is that the interface is extremely complex and spans several of Amazon's cloud service products. For example, you speak to the Alexa and it runs many layers of recognition and permission checking that updates billing code and such before it hands something off to their Lambda service. The Lambda service runs some code that hands things off to their AWSIoT service who does some stuff and passes your signals off to Amazon's own MQTT service which will hand things off to your code at home. Your code reads this and does something returning it back to Amazon's MQTT, which passes it back to Lambda who passes it back to the Alexa service and then back to the device on the counter to be played back. In the middle of this are many, many layers of authentication, roles and permissions that have to be exactly right before things are allowed to pass. Heck, you can't even log anything without using their cloud logging service ... and you have to read it there as well.

Just forget about it being easy; it isn't. Maybe someday it will be, but it isn't right now. And, NO, I'm not going to add to the huge number of tutorials out there with my own version. Frankly, I really don't want to step through it all over again taking a hundred screen shots that will become obsolete in three or four days when they change the user interface to the various services all over again. One of the biggest problems I had was trying to work from some of the examples that were of an older interface and try finding similar capabilities on what I was looking at. You see, they have crews of people working on each piece expanding its capabilities, and even though they are very careful, things get missed. That of course means they have to fix the thing they broke and that can change the interface. It's one little guy out in the desert vs. hundreds spread around the world.

Let's talk about a specific example. Below is the session where I created the various devices using Amazon's CLI (command line interface) that I had to download and install on my Raspberry Pi. After installing it I used instructions I found out there on that world wide web thingie, and this is the console log I kept of the process. I annotated it to tell you what was actually happening after I completed the process, and it should give you a real example of what to do to set yourself up in the very beginning on the Pi:

#
# Configuring AWS ... but
# I kept the secrets that were produced by the user creation so I could use
# them here, but I forgot the step of adding permission for the user I created
# This is what happens when you forgot to add permissions to the user
#
pi@housemonitor:~/src/alexa$ aws configure
AWS Access Key ID [None]: ***Secret Stuff***
AWS Secret Access Key [None]: ***More Secret Stuff***
Default region name [None]: us-east-1
Default output format [None]:
#
# This was trying to create a 'thing' but didn't have permission 
# to do it.
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"

An error occurred (AccessDeniedException) when calling the CreateThing operation: User: arn:aws:iam::704749107060:user/alexaControl is not authorized to perform: iot:CreateThing on resource: arn:aws:iot:us-east-1:704749107060:thing/house
#
# After I went back and added:
#   IAMFullAccess, AWSIoTFullAccess and AWSLambdaFullAcess 
# to the user I created
#
# Then I created a 'thing' called "house"
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"
{
    "thingArn": "arn:aws:iot:us-east-1:704749107060:thing/house",
    "thingName": "house"
}
#
# And proved it was there by listing it
#
pi@housemonitor:~/src/alexa$ aws iot list-things
{
    "things": [
        {
            "attributes": {},
            "version": 1,
            "thingName": "house"
        }
    ]
}
#
# Custom Alexa "things" are all about security, so I generated keys
# which sent the keys to the console and scrolled the screen a bunch
#
pi@housemonitor:~/src/alexa$ aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile cert.pem --public-key-outfile publicKey.pem --private-key-outfile privkey.pem
{
    "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
    "certificatePem": "-----BEGIN CERTIFICATE-----\nbunch of stuff\n-----END CERTIFICATE-----\n",
    "keyPair": {
        "PublicKey": "-----BEGIN PUBLIC KEY-----\nbunch of stuff\n-----END PUBLIC KEY-----\n",
        "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\nbunch of stuff\n-----END RSA PRIVATE KEY-----\n"
    },
    "certificateId": "even more stuff"
}
#
# But, they were in the directory I created for this
# I proved it by listing them
#
pi@housemonitor:~/src/alexa$ aws iot list-certificates
{
    "certificates": [
        {
            "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
            "status": "ACTIVE",
            "creationDate": 1477339839.64,
            "certificateId": "e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
        }
    ]
}
#
# Here's the list of them
#
pi@housemonitor:~/src/alexa$ ls
cert.pem  privkey.pem  publicKey.pem
# These are the certificate, private key and public key. Keep
# the private key protected and use the public key for communication
#
# This is downloading the public key for Amazon AWS IOT
# I need this to encrypt the traffic to Amazon
#
pi@housemonitor:~/src/alexa$ wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O aws-iot-rootCA.crt
--2016-10-24 13:14:47--  https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem
Resolving www.symantec.com (www.symantec.com)... 104.100.196.23
Connecting to www.symantec.com (www.symantec.com)|104.100.196.23|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1758 (1.7K) [text/plain]
Saving to: ‘aws-iot-rootCA.crt’

aws-iot-rootCA.crt  100%[=====================>]   1.72K  --.-KB/s   in 0.001s

2016-10-24 13:14:49 (1.25 MB/s) - ‘aws-iot-rootCA.crt’ saved [1758/1758]
#
# Now I take a look at the directory to see what keys I 
# have accumulated so far
#
pi@housemonitor:~/src/alexa$ ls
aws-iot-rootCA.crt  cert.pem  privkey.pem  publicKey.pem
#
# This is where it starts to get even more complicated
# I have to have a policy in place to allow this Amazon user
# to publish data. The policy is a JSON formatted file I called
# iotpolicy.json (I totally stole that name from Nick Triantafillou)
# 
pi@housemonitor:~/src/alexa$ aws iot create-policy --policy-name "PubSubToAnyTopic" --policy-document file://iotpolicy.json
{
    "policyName": "PubSubToAnyTopic",
    "policyArn": "arn:aws:iot:us-east-1:704749107060:policy/PubSubToAnyTopic",
    "policyDocument": "{\n    \"Version\": \"2012-10-17\", \n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\":[\"iot:*\"],\n        \"Resource\": [\"*\"]\n    }]\n}",
    "policyVersionId": "1"
}
#
# Now that a policy has been put in place up on Amazon AWS
# I have to attache it to the arn (Amazon resource name) which is 
# actually saying to attach it to the place I'll eventually send it 
# to
pi@housemonitor:~/src/alexa$  aws iot attach-principal-policy --principal "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4" --policy-name "PubSubToAnyTopic"
pi@housemonitor:~/src/alexa$
# I get the endpoint that I'll send data to. This is actually a
# URL that accepts mqtt data and gets it into Amazon
# I will need the endpoint to actually send the data
# This endpoint will show up in the code as the destination for data
pi@housemonitor:~/src/alexa$ aws iot describe-endpoint
{
    "endpointAddress": "ayccly8mhj4t3.iot.us-east-1.amazonaws.com"
}
#
# So I have a place to send data to and a policy to handle the data,
# I need to attach that to the 'thing' I created called 'house'
#
pi@housemonitor:~/src/alexa$ aws iot attach-thing-principal --thing-name "house" --principal  "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
pi@housemonitor:~/src/alexa$
#
# Finally, the linkage to aws is done I just have to test it.
# wish me luck

This little set of steps didn't seem to bad, but I didn't understand much of it at first. That seems to be the general rule, you poke at it until you get something to work sort of, and then you put out a tutorial or example and explain things about half way because that's all you really understand. As an example, there is a line down near the bottom of the above about attach-thing-principal. This is one line in a json file that the Amazon service holds that needs to be there. They call it a principal and you have to create it with a command they provide in the CLI. It's one line, they make it sound like something important. Well, actually it is, but only because they made it important. The Amazon interface is full of things like this.

They came up with a rather clever idea. They hold the state of your device in a regular old json file on their server. No, it isn't a database, it's just a file. This is clever because they can modify the file to represent things as they change. When I was working on getting my weather data up to Amazon I was able to add one item at a time and watch the file change as I added them. I started with the temperature (naturally) and it showed up in the file. Then I added the barometric pressure and saw it show up. I stepped through the items I wanted to check on until I got a nice little weather report I could ask for while walking around the kitchen.

Once I got that working pretty nicely, I wanted to actually change something, and this is where the way that they handle the file became really useful. You get the device to be reported and it shows up in the JSON file, then you tell alexa to change it and the file changes to show new things. It has a 'desired' entry where the new state of the device shows up, and then a 'delta' entry where the difference from the overall status and the 'desired' show up. Only the delta is sent to the Pi for it to act on. You don't have to worry about what went on before at the Pi, you just do whatever the 'delta' tells you to.

This may not sound like much, but it allows the device to go off line for periods and come back to catch up with what you last told it to do. It also provide readings for the last time the device was on line. They call this entire mess a 'shadow' and I learned to like it.

So, don't despair, over the next (however many) postings I'll tell you about parts of the system that you can step through to get your own house hooked into the Alexa. But, NO, I won't take a look at your individual implementation and show you what's wrong, nor will I answer basic questions like how do you sign on to Alexa. That stuff changes almost as fast as I could write about it. Get good at bookmarking the various pages you use on Amazon, they're hard to find a second time. Remember, I'm learning about this as I go, just like you are.

The difference is that I already put over a week into the darn thing and have actually got a light I can control with it.

Oh, before you folk ask why I went my own way instead of using Smart Things or one of the other home automation systems out there. I hate cloud services. I don't want my data stored somewhere else under someone else's control and subject to their whim. If Amazon changes their policy, I just don't use them for remote control anymore; I get out the phone and touch the screen instead. The Alexa is cool, but it won't control me or my data.

Continued here <link>