Raspberry Pi, More About Installing a SSD

I just finished helping a reader get an SSD (solid state drive) running on his Pi, and realized that the documentation that the Pi folk provided <link>, while correct, left some things kinda hanging for people doing this. Mainly, what to expect when you type the various commands, basically what to expect while actually doing it. Don't misunderstand, they didn't mess anything up, or leave something out, they did a great job, but some of us dweebs can get lost in the responses and such that come back.

I saved key parts of hooking up a SSD to a USB port and getting the system over to it, then booting from it, to serve, both as an extended example, and as notes to myself for next time. Yes, I read my own blog to see what the heck I did last time.

So, you get yourself an SSD and some kind of enclosure to hold it. I went to Amazon and grabbed one of the cheapest combinations I could find and used that. I wound up with two of them due to a shipping problem. Actually, this is a cute story, so I'll briefly tell you about it.

I ordered a drive and USB enclosure and then waited for it to come in. It didn't. The Amazon order said it was delivered, but it wasn't on my porch or in the box down the road. Naturally, I called Amazon and complained. They immediately refunded the money and put in an expedited order for new devices. I got the new devices in two days. That was absolutely great; I love Amazon customer service a lot.

A week later, the neighbor came over with the original package. They just moved in and forgot about it until it turned up again. I thanked him then paced around wondering whether I should tell Amazon, or just keep the drive. You guessed it, conscience got to me and I called Amazon. I had a choice, send it back (they pay shipping) or buy it. I bought it; since I actually need three of them, might as well keep this one. Sure, it's more expensive than others I can get, but I already had it in my greedy hands. That's the drive I used in the explanation below. I still want to get a really cheap one and try the entire process again with cheaper hardware, but that can wait for later.

Anyway, you have your new SSD in hand and want to put the Pi software on it and use the thing, so plug it in and partition it. Since this is the first USB drive, it will be recognized and assigned /dev/sda; if you have another drive, unplug it or figure out what the drive shows up as. Then:

pi@housemonitor:~$ sudo parted /dev/sda

GNU Parted 3.2

Using /dev/sda

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) mktable msdos

(parted) mkpart primary fat32 0% 100M

(parted) mkpart primary ext4 100M 100%

(parted) print

Model: ASMT 2115 (scsi)

Disk /dev/sda: 63.4GB

Sector size (logical/physical): 512B/512B

Partition Table: msdos

Disk Flags:

Number  Start   End     Size    Type     File system  Flags

 1      1049kB  99.6MB  98.6MB  primary  fat32        lba

 2      99.6MB  63.4GB  63.3GB  primary  ext4         lba

(parted)

Information: You may need to update /etc/fstab.

pi@housemonitor:~$

The things I typed in are bold so you can see them easily and you have to type control D to get out of parted. Folk that have done this before will wonder why I created a boot partition when I'm not going to use it. Well, I figure the work on alternate boot capabilities will continue and I want a boot partition on there in case I want to use it later. It doesn't take much space, and I have way more than I need, so why not? Occasionally I actually think ahead.

Now we have to put file systems on the new partitions. We do that with mkfs and here's what it looks like:

pi@housemonitor:~$ sudo mkfs.vfat -n BOOT -F 32 /dev/sda1

mkfs.fat 3.0.27 (2014-11-12)

pi@housemonitor:~$ sudo mkfs.ext4 /dev/sda2

mke2fs 1.42.12 (29-Aug-2014)

Creating filesystem with 15442176 4k blocks and 3866624 inodes

Filesystem UUID: f1df7b73-5872-43b8-b0da-29d6d139557d

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000, 7962624, 11239424



Allocating group tables: done

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done



pi@housemonitor:~$

This makes the filesystems on both the /BOOT and / partitions. Now you're ready to put something on there. In most cases, you want to take a running system, clone it on the new drive, and then use it without having to change a bunch of stuff. So, let's get them mounted so we can copy what we already have to the new drive.

pi@housemonitor:~$ sudo mkdir /mnt/target
pi@housemonitor:~$ sudo mount /dev/sda2 /mnt/target/
pi@housemonitor:~$ sudo mkdir /mnt/target/boot
pi@housemonitor:~$ sudo mount /dev/sda1 /mnt/target/boot/

This mounts both of the new partitions. Next, you may have to install rsync; it depends on which version of Jessie you started with. Jesssie lite doesn't come with rsync, while the full version does. I had it on one Pi and not on the other. If you need to go get it do an update first to be sure you get the latest.

sudo apt-get update

There will be a ton of output from this depending on how recently you updated. I always get lots of stuff since I rarely update the machines. Then actually install rsync:

sudo apt-get install rsync

This will also have lots of output. You've done this before, you know what you're going to get. Now, copy all the stuff from the SD card to the new drive:

sudo rsync -ax --progress / /boot /mnt/target

Not only will this produce voluminous output, it takes a good long time. If you have a minimal system like Jessie lite, you can wait around for it. If you have the full system with a lot of installed packages, take a break. I fed the dog, put on some laundry, called the telephone company and it still wasn't done.

Since all my Pi's are run headless (without a monitor or keyboard), I had to generate new keys for ssh. The easy way to do this is to mount the new partitions and pretend they are the real thing. If that's unclear, take a look at the documentation for chroot:

pi@housemonitor:~$ cd /mnt/target
pi@housemonitor:/mnt/target$ sudo mount --bind /dev dev
pi@housemonitor:/mnt/target$ sudo mount --bind /sys sys
pi@housemonitor:/mnt/target$ sudo mount --bind /proc proc
pi@housemonitor:/mnt/target$ sudo chroot /mnt/target
root@housemonitor:/#

Now, actually regenerate the keys. This warns that it will take a while to do this, but compared to the rsync above, it's nothing.

root@housemonitor:/# rm /etc/ssh/ssh_host*
root@housemonitor:/# dpkg-reconfigure openssh-server
Creating SSH2 RSA key; this may take some time ...
2048 1b:51:b1:64:a3:ad:2a:3c:3f:54:b0:7e:e5:e8:b2:76 /etc/ssh/ssh_host_rsa_key.pub (RSA)
Creating SSH2 DSA key; this may take some time ...
1024 fc:75:4e:17:77:f1:7e:4c:50:47:95:d7:94:3a:24:31 /etc/ssh/ssh_host_dsa_key.pub (DSA)
Creating SSH2 ECDSA key; this may take some time ...
256 3f:56:f8:02:92:fe:c6:a5:3d:3b:b7:58:f1:f1:2a:0b /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 cc:ba:5e:54:df:52:19:1c:ec:be:96:39:ca:55:3b:8a /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
[ ok ] Restarting OpenBSD Secure Shell server: sshd.
root@housemonitor:/# exit
exit
pi@housemonitor:/mnt/target$

Whew ! Unmount the new stuff which will make sure it gets written out, and then we'll make changes to the boot files to complete the process.

pi@housemonitor:/mnt/target$ sudo umount dev
pi@housemonitor:/mnt/target$ sudo umount sys
pi@housemonitor:/mnt/target$ sudo umount proc
pi@housemonitor:/mnt/target$

This next part may confuse you some. I've noticed that all the examples out there don't show the mistakes we all make in following some tutorial or written process. I decided, for one window, to show what actually happened during some of the changes I made; this is where I edited the cmdline.txt and fstab files. I did a cat of the file contents before and after since the editor clears the screen of the changes:

pi@housemonitor:/mnt/target$ cd /boot
pi@housemonitor:/boot$ ls
bcm2708-rpi-b.dtb       COPYING.linux  LICENCE.broadcom
bcm2708-rpi-b-plus.dtb  fixup_cd.dat   LICENSE.oracle
bcm2708-rpi-cm.dtb      fixup.dat      overlays
bcm2709-rpi-2-b.dtb     fixup_db.dat   start_cd.elf
bcm2710-rpi-3-b.dtb     fixup_x.dat    start_db.elf
bootcode.bin            issue.txt      start.elf
cmdline.txt             kernel7.img    start_x.elf
config.txt              kernel.img     System Volume Information
pi@housemonitor:/boot$ cp cmdline.txt cmdline.sd
cp: cannot create regular file ‘cmdline.sd’: Permission denied
pi@housemonitor:/boot$ sudo !!
sudo cp cmdline.txt cmdline.sd
pi@housemonitor:/boot$ sudo vi cmdline.txt
pi@housemonitor:/boot$ cp cmdline.txt cmdline.usb
cp: cannot create regular file ‘cmdline.usb’: Permission denied
pi@housemonitor:/boot$ sudo !!
sudo cp cmdline.txt cmdline.usb
pi@housemonitor:/boot$
pi@housemonitor:/boot$ cd /mnt/target/etc/
pi@housemonitor:/mnt/target/etc$ cat fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
pi@housemonitor:/mnt/target/etc$ sudo vi fstab
pi@housemonitor:/mnt/target/etc$ cat fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    ro,auto,user,exec,async          0       2
/dev/sda2       /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
pi@housemonitor:/mnt/target/etc$

I'm constantly forgetting to use 'sudo' to edit system files. It really gets annoying making the same mistake over and over again. 

Now, the trial by fire. Notice above that I saved the cmdline.txt file for the SD card. That's just in case this doesn't work. If it fails all I have to do is copy the file back in place on a PC (remember, the BOOT partition is msdos) and I'm back to the SD card and can fix whatever I messed up. So, let's unmount the new disk and reboot the Pi:

pi@housemonitor:/mnt/target/etc$ cd

pi@housemonitor:~$ sudo umount /mnt/target/boot

pi@housemonitor:~$ sudo umount /mnt/target

pi@housemonitor:~$ sudo reboot

I usually find myself gritting my teeth at this point, and honestly, the Pi got into one of those situations where it wouldn't connect to the network. I had to walk over and cycle power to get it up. This happens to me once in a while, and the Pi will almost always come up after some period of retrying. I just got tired of watching it blink while it tried to connect. But, it came up and I did a 'df' to show the new disk was mounted and the SD card was read-only.

login as: pi
pi@192.168.0.205's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Sep 28 09:09:58 2016
pi@housemonitor:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       60666940 3968532  53593592   7% /
devtmpfs          218416       0    218416   0% /dev
tmpfs             222688       0    222688   0% /dev/shm
tmpfs             222688    5052    217636   3% /run
tmpfs               5120       8      5112   1% /run/lock
tmpfs             222688       0    222688   0% /sys/fs/cgroup
/dev/mmcblk0p1     61384   20400     40984  34% /boot
tmpfs              44540       0     44540   0% /run/user/1000
pi@housemonitor:~$ ps ax|grep python
  455 ?        Sl     0:09 /usr/bin/python /home/pi/src/house/updateoldxively.py
  458 ?        S      0:01 /usr/bin/python /home/pi/src/house/savehouse.py
  461 ?        Sl     0:05 /usr/bin/python /home/pi/src/house/healthcheck.py
  462 ?        Sl     0:03 /usr/bin/python /home/pi/src/house/events.py
  464 ?        Sl     0:05 /usr/bin/python /home/pi/src/house/iriscontrol.py
  466 ?        S      0:01 /usr/bin/python /home/pi/src/house/mqttlogger.py
  468 ?        Sl     0:08 /usr/bin/python /home/pi/src/house/monitorhouse.py
  470 ?        Sl     0:04 /usr/bin/python /home/pi/src/house/wemocontrol.py 4
  807 pts/0    S+     0:00 grep python
pi@housemonitor:~$

I also did a 'ps' to check that the processes I usually run started automatically. Everything worked.

So folks, here's a real example of how to do this with a lot of the text that comes along with it. It took me less than a morning, even with the long rsync process. I didn't bother backing up the SD card since I was only changing the one file, cmdline.txt, on it and even that was backed up. The big advantage to me is that the SD card is fully bootable and is very unlikely to fail since it is never written to. I can use it to fix something if I need to at a minutes' notice.

Have fun.

I'm Still Hung Up On Power Failures, but Suppose

OK, OK, maybe I'm a little OCD at times, but that can be a good thing. Folk have been making suggestions, and there have been some interesting pointers given to me, but none of them pass the test. But one suggestion caused a brain storm a little while ago.

A tiny bit of background. It's easy to control a relay or a light, what's hard is the smarts to know when to turn it on, or off. Sometimes this gets pretty complicated. Turn the sprinklers on at 10PM but only if the ground is dry and rain isn't predicted tonight. Turn the porch light on when someone approaches but not during the day. Turn the bathroom light on when someone enters the room, but not if it's after bedtime. We all have problems like that, and the problem is that every time we want to add a little more smarts to the device, we have to do some code and get it on the machine. That often means taking the device out of the box or whatever it's in and carrying it somewhere to reprogram it. Suppose I'm reprogramming something in the attic? It would totally suck having to crawl up there and modify it. Raspberry Pi's are great for this, just use a terminal program like Putty and connect to it over wireless. You can change things, try them, generally mess around until you get what you want and never have to lug the laptop out to the barn to plug it into the chicken feeder.

Suppose I split the work between two computers? One would be fully protected by a UPS that can do all the stuff I need, and another that can survive power failures, but only does the controlling. Let's take a specific but reasonably simple example: sprinklers. I don't need them, but other folk do, so let's take a case of a yard with 4 stations that need to be controlled.

I go out and get one of those relay boards and an Arduino to run it. I can put that in a weather proof box and stick it out close to the sprinkler valves. I will have a power supply wall wart running to it and an XBee to communicate. Arduino's handle power failures well since they have their software in ROM. I only need four command sets:

Relay one on-off
Relay two on-off
Relay three on-off
Relay four on-off

That code would be really easy to put together and implement. Then, I get a Raspberry Pi to send the commands and handle timing and such. I could easily put together menu screens to set the various on-off times, keep track of when they were run, etc. The Pi could be the ultimate sprinkler controller. Of course the Pi would be in a safe place with its power under the monitoring of a UPS that has the correct features.

Let's get a little more complex and consider a swimming pool controller. That would need more relays and possibly some really special features. The basic things I would need in my case would be:

Motor on-off
Fountain on-off
Waterfall on-off
Light on-off

Darn, doesn't that look exactly like the sprinkler control?  Now let's get a little bit more sophisticated and add control to the motor speed and the solar heater. Thinking about it a bit, I only need to add temperature readings for the water and the sun light on the roof to control the solar. Then for the motor, I only need to send motor commands and read their response. I've done all those things with my temperature sensor and pool control projects.

Take the sprinkler idea a bit further by adding some moisture sensors in strategic places with an XBee to transmit the data back to a Pi that figures out what needs to be done and sends back commands. Once again, the brains are in a protected spot with the actual worker computer somewhere else. The Pi could have elaborate software that looks pretty to control the pool and all the features could be made available to a web interface. That would be a fun project, but we might have to resort to battery powered devices to get to some of the hard to reach places.

One of my dreams is to monitor the temperature in each room, gather it in a central place, then control the heat pumps and air handlers intelligently to move heated or cooled air around the house. Little devices that use ROM and can send data would need to be placed in each room, then another simple computer would be placed in each air handler to actually control the fans, compressors and changeover switch. A Pi would be used to look at the data and do something appropriate. I would be able to use any program language or piece of software to develop the smart part from anywhere I feel comfortable working.

So, the various Arduinos would be worker computers that run software that doesn't need to be changed unless I want to add some hardware feature; they do just fine if the power is pulled. The Pi's would be more sophisticated and would handle all the real decisions. They would be backed up automatically and protected from nasty power crap. It would expand my XBee network, but I've got lots of bandwidth there. The other really great thing is that the Pi's can be programmed and reprogrammed any time I want without lugging a laptop out to the sprinkler junction or up into the attic. The PI's could send mail, sms, whatever is needed, heck they could send a signal to an alarm bell just like the big systems. This idea has been sitting right in front of me forever.

I just needed a nudge or two from my readers. Thanks.

I Really, Really Hate the SD Card in the Raspberry Pi

so much so, that I tried the beta version of the Boot From MSD (mass storage device) changes the Pi folk blogged about recently.

TL;DR version: It didn't work for me because I messed up, but I found another solution that is pretty darn compelling in the process of trying it out.

First, I found the Pi blog post here <link>, and read it over about three hundred times because it was the kind of thing I've been fighting for years now. Then, in typical form I got on Amazon and ordered a solid state disk and a little box to hold it and adapt it to USB.

Wait, before you go off and do the same thing I did, check out the link above and read the rest of this post. Also, the reason I said to use my link is because it points to the 'official' post about the boot from MSD, there are a lot of copies out there on the web and some of them are wrong. Go to the real source and start from there. I won't go into the details of what to do since I would only be copying someone else's hard work and I might make a mistake that would screw you up.

Here's the hardware I'm using:

These things cost me about $35, and I'm sure I could have gotten them cheaper if I was willing to wait for a boat from Taiwan, but I wanted to try this out right the heck now.

I went into Raspberry Pi's github repository <link> for the instructions and followed them exactly on my Pi 3B, and of course, it didn't work. Not because the instructions were bad, but because, in exuberance, I ordered a USB 3.0 device. They specifically say they don't support USB 3 and I just ignored that when I looked for the stuff I needed. However, it impressed on me something that I had overlooked way back when I tried a USB stick to overcome the problems with SD cards.

You ONLY NEED TO BOOT FROM THE SD CARD !! You don't need to run with it. That means, you put the boot stuff on the card, boot from it, and then ... get ready for this ... you mount it READ ONLY so a power failure won't corrupt the card because of files left open when the little guy dies. If you don't write to the card, it won't get messed up.

Sure, there will be a thing or two to consider doing this, for example when I do an update-upgrade (shudder), it will probably fail if there are changes to the boot code. Fine, I'll just make a note to myself to remember to remount the boot file system RW before I do that; or more likely, I'll fix it after the upgrade fails with some obscure error message. I'm actually running one of my little machines this way right now and it seems to be doing fine, and it boots awesomely fast compared to the SD card.

I've pulled the power plug a few times to see what happens, and YES darn it, I managed to corrupt the file system on the SSD. But, and this is a really big thing to me, fsck fixed it and it just kept on keeping on. I hardly noticed the problem.

Let's think about this a bit more. I don't have to even think about the long tedious process of backing up the SD card any more. I just make a couple of copies of Raspbian lite on cheap little SD cards just big enough to hold it and stick them to the wall taped to a 3x5 index card with a clear label I'll remember later. The SD card dies because of some random act, I grab another card, plug it in, and away the Pi goes because all the stuff that really matters is on the SSD. I can also use this technique on any of the Pi's I have, even the oldest. You just need to be able to read the SSD to finish the boot process; I don't need a whole system on the SD card. An unexpected benefit of this is that you have a fully boot-able system on the SD card that you can use to repair something if you need to.

Here's the cmdline.txt from the /boot on my Pi3B:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

And the entry I have in fstab to set the boot to read only:

/dev/mmcblk0p1  /boot           vfat    ro,auto,user,exec,async  0       2

I'm not sure that this fstab entry is totally correct, but it gets the job done; root can't write to the boot partition. I keep two cmdline.txt files for backup in the /root directory: cmdline.txt.ssd for ssd boots, cmdline.txt.sd for sd card boots. I just copy whichever I need to cmdline.txt so I don't have to look up the weird name mmcblk0p1 each time I test something in this area.

To back up the SSD, just do a network 'dd' to a big disk somewhere else, or a file by file copy, rsync, something along those lines. I can even automate it now because the SSD is super quick and I'm not wearing it out the same as with an SD card. My little machine may be a little behind, but the data is stored on my NAS, and who cares if a log file got out of date. I'll definitely be looking into that in the next couple of weeks (well maybe).

I could totally eliminate power fail corruption for all the machines I want to someday have if I can come up with a house-wide power-fail shutdown idea that doesn't include $60 UPS devices out in the yard. Remember, my power goes out every time the weather sneezes, or a random drop of water falls from the sky. I've gone through several SD cards along the way. I tried a USB card to solve the problem, but the USB card died in a particularly annoying fashion <link>. I won't go that route again.

I want to step through the requirements for a UPS of some kind so you folks understand (and maybe suggest something):

1. cheap

2. It has to sense power failure and tell the Pi to halt while holding power on
for a minute or more while the Pi actually halts.

3. It has to turn power off to the Pi.

3. It has to sense power coming back, wait a bit for those messy power failures, and supply power to the Pi if it has turned the power to the Pi off.

4. Small

5. cheap

Requirement 3 is important because there is no power management on the Pi. The only way out of a halt condition is to cycle the power. Requirement 4 is important because the power can bounce around a bit and I don't want a bunch of half boots where the Pi has just enough time to start booting before the power goes out again.

I've thought a bit about a little Arduino to do this job for me. If I combine it with one of those cell phone power extenders and a good wall wart, I might be able to come up with something suitable.

Now, I'm going to check out the Chinese vendors for really good buys.

OK, Now About UPS's and the Raspberry Pi

I have had a UPS for my NAS for a while now, but this season's storms caused me to seriously think about getting another one for my set of Raspberry Pi's. Every time the wind got over 20 mph, the power would go out, thunderstorms would kill the power, and the occasional dust devil in the right place would cause those short power failures that just mess things up. A couple of the power failures were those messy ones where the power dropped to almost nothing, then came back, went away again, and finally just died for a couple of hours.

I wasn't concerned about spikes; I have a bunch of protectors around for that, but it gets annoying fixing the various software problems power failures make on the Pi. So, I got a second UPS to install where I have my Pi collection. I bought another APC BE550G:

This is big enough to handle the short outages, and has enough guts to run the various things I may hook up. I chose this one because it's the same as the one on my NAS that runs it and several other little computers up in the attic. Why have two different devices?

Yes, it's an expensive solution, but after I buy a power supply board and the circuitry necessary for battery backup on the Pi, I would have spent just as much and still need more for the other little computers. I already know how to make one of these, so it would be just grunt work and not much fun.

The NAS up in the attic has really cool power fail software, so I went looking for something similar to run on the Pi and found a tool called apcusbd. This is a daemon that runs all the time monitoring the USP port on the UPS. When it finds a problem it can send email letting me know as well as messages to anyone logged into the machine. It will also send a power down to the UPS shutting off the power when a timer runs out.

This power down is important. When you halt a Pi, it just halts, there's no power down, and that means you have to unplug, plug to get it to come back online. So, for unattended operation, you have to somehow cycle the power to get the Pi back to working. I have to halt it because, if I don't, it will screw up the file system, or even worse, corrupt the SD card so I have to get it from backup. Been there, done that, too many times.

If I could get this all working correctly, it would go a long ways to solving some of my SD card problems over long periods of time. I dug in and installed, configured, and tested apcusbd. It worked really well on the first try after I configured it. There are literally thousands of sites out there that tell you how to install and configure this software, so I won't repeat it here, but there was one problem: it wouldn't turn the power off on the UPS.

Digging into the configuration did nothing for me; it looked like I got it right, so I started testing pieces of it. Apcupsd allows the use of scripts that I could modify a bit and test the power down functions. When I tried them, they worked and actually turned the power off. This left me confused; if they worked, why didn't the power turn off? I looked through the syslog and found nothing to help in troubleshooting this problem, so I dug into the shutdown procedure of the Pi running under Jessie. Man, what a complex mess that is.

See, systemctl sets run level zero which causes the scripts in /etc/rc0.d to be executed in order and the last one is 'halt'. This seems reasonable, but the shutoff code for the UPS is in the halt script. everything else is already done ... including syslogd which is the thing that logs all the data for the shutdown. No, I couldn't change it to write to a file since the file systems are all remouted read only by that time. There was no way to tell what the heck wasn't happening.

I never managed to solve that problem, so I decided to send the turn off signal to the UPS earlier in the halt steps since the UPS gives me a grace period of over a minute of power after the signal; plenty of time to finish the halt process. To this end, I created a little script to add to the shutdown process that would sense the power failure from apcupsd and send the signal using the scripts that already come with the software. I couldn't get the script to be run by systemctl.

Yes, I spent hours looking on the web for what was preventing it from being used, and found other folk that had the exact same problem. There were exactly zero actual solutions. Lots of talk about run levels, LSB's, running update-rc.d, keeping certain little things out of the script, but nothing definitive that would work. I kept reducing the stuff in the script until I had almost nothing in there and it still wouldn't get run during the shutdown and halt process. What I finally wound up with before I gave up on this idea was:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          davetesting
# Required-Start:
# Required-Stop:
# Default-Start:
# Default-Stop:      0
# Short-Description: Dave Testing
# Description:       Just testing this thing
### END INIT INFO

PATH=/sbin:/bin
echo "Dave Testing in aaa"

echo "Dave inside aaa before the file thing"
echo $1
echo "Dave inside aaa got a $1"

I put my name in there so I could find it in the syslog, but it didn't matter. The script simply wasn't getting run. What was most annoying is that all of the similar questions I saw on the web were simply unresolved. The other folk apparently just gave up and did something different. That's the same tactic I took.

I saw that the script to stop the apcupsd daemon was actually running, so I put a little test code in there to see what would happen. The test code was executed and did exactly what it was supposed to do. Here was my hook into getting it working since the 'right way' didn't work and my attempt at a separate script didn't either.

The way apcusb works is to sense a power failure signal from the UPS, then create a flag file to be read later. After the daemon is stopped, it should be called again after the file systems are mounted read only with a parameter telling it to not run as a daemon, but instead, send the power down signal to the UPS. This little action is controlled by the presence of a flag file. I took that code from the init.d script for halt and put it in the script that starts and stops apcupsd. Of course that means the file systems are not read only and leaves the possibility of a problem, but since the UPS gives me over a minute grace, there won't be any problem; the Pi would have long since halted.

Then when the power comes back up, the UPS waits a little bit, restores power to the Pi and then the normal boot process gets going. During the boot process the flag file is removed so everything is back to normal waiting for the next storm.

Here's the init script that I came up with:

#!/bin/sh

### BEGIN INIT INFO
# Provides:             apcupsd
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Starts apcupsd daemon
# Description:          apcupsd provides UPS power management for APC products.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/sbin/apcupsd
CONFIG=/etc/default/apcupsd
NAME=apcupsd
DESC="UPS power management"

test -x $DAEMON || exit 0
test -e $CONFIG || exit 0

set -e

. $CONFIG

echo "Dave is in ups code messing around"
echo "Dave found $1 "
if [ -f /etc/apcupsd/powerfail ]; then
        echo "Dave Found powerfail"
else
        echo "Dave No powerfail Found"
fi

if [ "x$ISCONFIGURED" != "xyes" ] ;
then
        echo "Please check your configuration ISCONFIGURED in /etc/default/apcupsd"
        exit 0
fi


case "$1" in
        start)
                echo -n "Starting $DESC: "

                rm -f /etc/apcupsd/powerfail

                if [ "`pidof apcupsd`" = "" ]
                then
                        start-stop-daemon --start --quiet --exec $DAEMON
                        echo "$NAME."
                else
                        echo ""
                        echo "A copy of the daemon is still running.  If you just stopped it,"
                        echo "please wait about 5 seconds for it to shut down."
                        exit 0
                fi
                ;;

        stop)
                echo -n "Stopping $DESC: "
                start-stop-daemon --stop --oknodo --pidfile /var/run/apcupsd.pid || echo "Not Running."
                rm -f /var/run/apcupsd.pid
                echo "$NAME."
                if [ -f /etc/apcupsd/powerfail ]; then
                        echo "Dave Doing  powerfail"
                        /etc/init.d/ups-monitor poweroff
                fi
                sync; sync;
                ;;

        restart|force-reload)
                $0 stop
                sleep 10
                $0 start
                ;;

        status)
                #/sbin/apcaccess status
                $APCACCESS status
                ;;

        *)
                N=/etc/init.d/$NAME
                echo "Usage: $N {start|stop|restart|force-reload}" >&2
                exit 1
                ;;
esac

exit 0

It's called apcupsd (duh) and, after running update-rc.d to create init links shows up in rc0.d as K01apcusbd. You'll have to scroll down a bit to find what I changed. I put my name in there so I could easily find the debug I added when prowling around in syslog. Needless to say, with booting it and killing it a hundred times, syslog got really large, so finding a line in there was a problem. The flag file is called 'powerfail' and if it's there, I cause the UPS to shut down. 

No, it isn't the 'correct' way to do this, but since I just couldn't find enough information to do it 'correctly', ... I hacked it.

I'm going to install apcupsd on my other Pi's because they can run as slaves to the one I already did. Apcupsd has a supper cool feature in that it can talk to other instances of itself on other machines and cause them to halt as well. Since my UPS up in the attic runs for 45 minutes after a power failure, I'll set this machine to run for 40 minutes then tell the other ones to shut down. At 42 minutes, I kill the power to the UPS, and everything should be just fine. 

I'm left with a nagging lack of understanding why the scripts don't work as they come from Raspberry Pi and apcupsd, and why I couldn't get my own version of a init.d script to work, but my particular problem is solved. If I ever figure it out, or someone points it out, I'll update this to reflect what really 'should' be done.

Now to implement the networked shutdown for the other machines and see what doesn't make sense there.

A New Web Interface for Desert-Home

Remotely monitoring and controlling my house has been part of the technical work on the house ever since I started. Like every other homeowner in the world I wonder if I closed the garage door ten miles away from the house. Monitoring the power usage and closing the garage doors was my main motivation for getting into this.

I came up with a web page that let me control things and see what was going on a few years ago, and have never been totally satisfied with it <link>. It worked and told me what I wanted, but it looks too small on a phone and is kinda clunky.

Here it is on my phone:

Later, I discovered the MIT Android App Inventor and created an app using it. This tool was (almost) exactly what I needed to get a native Android app on my phone<link>. The problem with it came up when the app got big. The way they work it is with a single work space that you place graphic elements into creating an application. When it gets big, you can't see enough of it to move things around and connect the dots. While I used that app, I looked around for other solutions.

This actually led me to create a native Android application that could talk to the house. The Android app was pretty cool:

Normally, the 'Wait' would be replaced with the current temp, and the various gauges would be showing data, but things changed (as usual) and I didn't update it. Obviously, I was really into green and brown at the time, plus I still liked the brass gauges. This app was fun because each of the device panels would slide in from the side and slide back off when I wanted to get rid of them. That left the display of only the items I was interested in at that moment. People were properly impressed; I was proud.

The problem is that Android is a pain to program to. The development environment is HUGE and takes a significant hunk of the storage on my laptop. Testing it means putting it on my phone, testing a bit, fixing it, putting it back on the phone, over and over again. The methods work, but they get really monotonous. There's an emulator in the SDK, but I couldn't get it to work reliably or fast enough to actually do things. Plus, at that time, the emulator couldn't talk over the network which means large portions of the code couldn't work there.

I'm not knocking the Android SDK, they did a fantastic job of putting it together and working out a way to develop apps, but it's a huge mess for an individual trying to do a simple job like this. I was taking on the jobs of a team of developers instead of just coming up with something to control my house.

I went looking again. Since I had come up with three methods already, I knew more about what I wanted to have:

A web interface of some kind that would work nicely on the phone as well as my laptop. I was perfectly willing to accept some compromises because of the difference in screen dimensions and mouse vs. finger input methods, but I wanted it to be mostly seamless between the two.

One set of code for both of them. It is too cumbersome for an individual bringing up a control system to keep track of code for different viewing devices. I could never remember to make corresponding changes to each piece, and one of them, or the other, always fell behind.

I want to be able to put a tablet on the wall, walk up to it, change something around the house and not think about what device I'm using to make the change. So, a phone, a tablet or two, the laptop, my friends computer, etc. Kind of daunting when you think about it; that's why it's been years of messing around.

But, I recently bought a Samsung Galaxy Note 7. Yes, the one that is exploding and catching fire and isn't allowed to be turned on when you're flying. That one. I love it. Sure it's big, but that means I can actually see things on it, and the fast charging is absolutely great. I can charge it from totally dead to full in less than two hours. No wonder they tend to blow up. Anyway, the new phone prompted me to take another look at something to control the house with.

Prowling around the web, I found several interfaces that looked promising, but they all had those esoteric marketing descriptions that have a lot of words, but very little to tell you how to actually use it. Some really good authoring systems that could fit the bill, but there were flexibility concerns with them. While I was looking, I kept thinking about how things that some corporation supplies change whenever they change a manager, so I decided to use some JavaScript library that I could, if needed, copy to my own machine in total and avoid the dependency on them keeping their stuff online and active. I do this with Arduino libraries, why not be prepared to do it with other stuff as well?

Enter JQuery Mobile. It's a couple of libraries that sit on top of the JQuery we all know and love (hate), and has been around for a few years. It's not maintained very much anymore since the lead developer has moved on, but it does what I want and handles all the crap of resizing various things when it displays on a different size screen. It even supports 'themes'; that's when you want to change the look of the presentation with different colors and such. They put in animation for popups and sliders, so I don't have to learn about that, I can just use them.

Sure, it may not keep up with the latest trends since it hasn't had a substantive change in quite a while, but that only means I don't have to keep changing things because it did. No, it doesn't support older browsers, but I don't use them; I keep my browser up to date. I can also copy the libraries to my Pi and keep what I have for a very long time.

It took me a couple of weeks to put the code together, and it looks like a whole lot of stuff already out there on the web, but it works with the way I do things. I still have my cool gauges to show various readings, and the charts that I use to tell me when something is starting to have problems. I managed to leverage a whole lot of the code I used for the original web interface to save me time learning how to do that all over again. So, I have my gauges, charts and control buttons. I even messed around a bit with a custom display that looks pretty cool for the thermostats. Let me take you on a quick tour; first a picture of what the main screen looks like on a laptop:

Boring right? I only show two gauges on the initial screen because that's really all I'm interested in most of the time. The temperature and the power usage right this minute. The other items are buttons to separate pages in the application that I can go to for more information. Looking at the thermostats:

Yes, they have color and are too far apart on the screen. The resolution of the laptop does this. I could easily expand them to be larger, but this works for now and adjusts fine on the phone or tablet. This is what it looks like on the phone in portrait:

This is the way it would look on a tablet mounted to the wall where I walk all the time. Tells me what I want to know in a glance. Similarly, in landscape:

I personally think this looks nice as a two thermostat control panel. Tells me everything I want to know at a glance and a little interaction can make changes. More about the way to change things in a moment, first, look at the presentation of the first page in landscape:

The two gauges are clear and easy to read. Notice I finally got tired of the brass gauges and went to pastels and chrome. I'll get tired of this too, but for now it suits me. In portrait:

The two gauges are still clear, but now they're vertical; I got this display for free with the JQuery Mobile library. I can scroll down the screen to see the other options by just brushing the screen:

In case you're wondering, the footer is a marque that slides from the left across the screen; that's why it looks different on each picture. Now, back to the thermostats and how to change them. When I tap on the little gear on the bottom right of the thermostat it slides in a popup that holds the controls:

Here I can change whatever I need to and then tap the 'Set' button to send the changes off to the thermostat. Slick huh? Yes, the house runs warm during the day; I live in the desert remember? Also, I don't like high electricity bills.

Yes, I have the charts on there. For example, the house power and outside temperature are displayed by the 'House Chart' button:

This is landscape mode, the portrait mode looks too crowded since all the data is compressed to fit on the screen. This is touch zoomable, and I can zoom in on any part of it. This comes free with HighCharts, the charting tool I've used for the last couple of years.

I do the same thing for the fridge, freezer and freezer in the garage:

That's the house freezer with its stats for the last 24 hours, and when I tap on it:

The chart for its activity during the last 24 hours slides in and displays. I even enabled the points on the chart for examination. I can keep really good tabs on the appliances I've implemented so far. I really want to include other things over time in a similar fashion.

The pool looks like this:

This tells me the pool motor is running on high, the waterfall is on, the light is on and the fountain is on as well. Yes, I turned them all on for this screenshot. The motor is black, green, red for off, low, high. I tried to animate it, but is looked silly. Each item is a button that can control that accessory, for example:

Tapping on the 'Off' button will actually turn the fountain off. There are other things I want to add like the temperature of the water and actual speed of the motor, but I haven't gotten to them yet. 

No, I didn't forget the weather station:

Each of the parameters I measure has its own gauge and shows the last reading I took. In portrait mode they are a series of gauges arranged vertically. When I have enough data to chart it accumulated, I'll add graphs for them. I'm not sure what's the right way to do that yet. Do I need 24 hours or the last week? I'll get to that someday.

The Statistics button on the bottom shows this:

I'm going to add stuff like rainfall this year, last rainfall, highest historic wind speed; you know, the stuff that impresses people that don't realize how easy a database query is to code. (Don't look at the low temp above, that's not a bug, it's a feature)

The last thing I'll bother you with is the control page for the lights. I currently only control four of them here. Here's the display as it normally appears:

All the lights are off. I do this control differently, each light is a button that can be toggled, so tapping the patio light will turn it on. So if I tap a couple of them the page will look like this:

Kind of adds a little color to the display. These are controlled by wemo light switches that I described breaking into through several posts.

Like I've said annoyingly, this is one source for all devices that I may want to use to talk to it. I kind of hated giving up on the other attempts, especially the Android native code, but it's already out of date and several versions behind in the Android system it runs on. It's unlikely the web will change that fast and the latest thrust is to use web applications anyway.

Those of you clever enough to figure out the URL are welcome to take a look, but remember, be kind; it's running on a little Raspberry Pi. You can't actually control anything unless you figure out the super-secret high security password, and if you do, I'll just change it.

Now it's time to see about exchanging that phone ...

Simple Temperature and Humidity Sensor

Anyone that plays with little computers has seen articles about the DHT11 or DHT22 sensors. They're clever little humidity and temperature sensors that don't require calibration and measure the two factors we all complain about when discussing the weather: temperature and humidity.

I've seen them, but actually never worked with one because, here in the desert, the humidity seldom gets out of the 20's. I've messed with a number of temperature sensors, but humidity didn't interest me much. That changed when my daughter and her husband bought a new house.

It's (the house) one of the new energy efficient homes that is really well sealed up and even has solar panels. The energy efficiency actually works, they use a lot less energy than I do per square foot and the place is comfortable in all kinds of weather. The big drawback is getting smells and moisture OUT of the house.

Suppose you're cooking dinner and have a couple of pots boiling on the stove, and the teenager takes one of those 20 minute showers they're famous for. The water build up in the house is huge. If the temperature outside is hovering around 20F, you can't really just open some doors and windows to air it out, so the water builds up until it becomes pretty uncomfortable. The steel front door starts to condense water and it runs down into a small puddle on the floor. The glass on the skylite (double paned, but covered with snow) starts to condense water and it drips on you when you stand under it. The solution is simple, air the place out and get rid of the moisture.

No, the fan over the stove doesn't help much. She turns it on, but naturally the steam misses most of the hood and goes for the ceiling. Those fans and hoods over the stove aren't very useful. They need to be bigger and have an actual FAN in them, not one of those six inch noise makers. If I have to listen to a fan, it really should actually be doing something.

I suggested they measure the humidity and see what is going on, so they can decide which of several methods they need to try to remove the water floating around, but my daughter had another idea, "Dad, why don't you make me one?"

OK, I can do that:

I got to try out the I2C input serial LCD displays and the sensor. It was a fun little project that took about three hours rounding up all the right libraries to make the devices talk. It took me a heck of a lot more time ordering and receiving the pieces. Of course, I already had the Arduino, cable, and sensor, but the display and sensor came from way far away.

It just works; no messing around with calibration or strange problems.

It's raining a bit today

Yes, I know you can buy these things for a few buck off Amazon or whatever, but where's the fun in that? Also, I managed to make a deal where I build it, and they figure out an enclosure that suits their taste. That was totally evil of me. The hardest part of this kind of project is enclosing it inside something and making it look nice, and I totally passed the buck on that part.

Total win for me. Plus, the kids can say, "Grandpa built that."