Using SSL security with ESP32 and ESP8266

The story so far…..

Robert has been trying to connect an ESP8266 embedded device to an Azure IoT Hub, something he has done many times with no problems. Except it no longer works. Having spent a day failing, perhaps it is time for Robert to have some success. Now read on…..

Yesterday was not a good day. Today was better. To summarise my problem: I want to use my embedded devices with Azure IoT hub so that I can exchange messages with services in the cloud. Azure IoT hub is a “proper service” in that it supports device registration and uses the Secure Sockets Layer (SSL) for connections. I’ve written my code so that it uses the secure WiFi client in the ESP8266 and it has worked before. But yesterday it broke and I couldn’t figure out why. Today I did. Go me.

It turns out that the solution is simple. Add the statement above to set your secure client to insecure mode. But why does this work and what does it do? Does it mean that my connection is no longer secure? Now read on again…..

There are two reasons for using SSL. The first is to make sure that the data that is being transferred securely. I don’t want someone to be able to peek into the packets and pull out passwords and data. The second reason for using SSL is to make sure that the devices is talking to the right host. Without this authentication it would be possible for someone to make a system that pretends to be Microsoft Azure, accept a device connection and then steal all my stuff.

The first part of the problem is solved by encryption. The systems each end of the connection use keys to encrypt and decrypt the data so that anyone watching it go past would just see gibberish.

The second part of the security, making sure my host is really Azure, is solved by certificates which are issued by certification authorities. Microsoft have a certificate for Azure and my my device can verify this. However, to make this work I have to make a copy of the public part of the certificate and store it in my device, which is not something I’ve bothered with in the past. It turns out that I’m not really concerned about someone spoofing Microsoft Azure just to see my data. There is always a balance to be struck between security and convenience, and this is where I draw the line. It may be the wrong place to draw it, but I’m OK with it so far.

This means that my devices don’t have local certificate copies. Up until recently (and with the ESP32 device) the lack of local validation doesn’t cause a problem. But something in the Arduino libraries has apparently changed (or I’m using a different library now). Anyhoo, to make it work now you have to tell the WiFiClientSecure instance created to manage the connection that you are using insecure mode. This doesn’t mean that your data is now being sent insecurely. What it means is that the device is not verifying the server any more. And then it works.

Azure IoT Hub for Free

Everything I achieved today I achieved before 9:30. It was all downhill after that. But I did achieve quite a bit. I’m making a bunch of connected devices and I want to use MQTT to link them together. I could set up my own MQTT broker on a local Raspberry Pi but that would only work around my house and I want these to work around the world. My weapon of choice for these things is Azure IoT hub which lets you connect remote devices to a central server (the clue is in the name).

It turns out that you can set up a Microsoft Azure account for free and then create an IoT hub which is also free and allows for up to 8,000 messages per day. I’m not sure how many messages I’ll need, but I’m hoping that will be enough for now.

Having created my free IoT hub the next thing I wanted to do was connect my devices to this. I’ve done it before and it worked just fine. I did it today and it didn’t. This was the cue to spend most of the day (or at least the portion up to the point where I gave up) trying t figure out why. Unsuccessfully. Wah.

Detecting different ESP devices in Arduino Code

This is one of these “Rob writes a blog post about something so that he can find it later when he forgets how to do it” moments. I’m writing some code using the Arduino platform that I’d like to run on both the ESP32 and the ESP8266. They both run C++ and have a lot in common. But some things are just different. For example they have different commands to ask a device for its unique ID is getFuseMac for the ESP32 and getChipId for the ESP8266.

Fortunately we can use the magic of conditional compilation to make our C++ code always do the right thing. The code below shows how it works.

#if defined(ARDUINO_ARCH_ESP32)
#define PROC_ID (unsigned int)ESP.getEfuseMac()
#endif

#if defined(ARDUINO_ARCH_ESP8266)
#define PROC_ID (unsigned int)ESP.getChipId()
#endif

The symbols ARDUINO_ARCH_ESP32 and ARDUINO_ARCHESP8266 are defined if the program is being built for the respective device. The C++ above defines a PROC_ID symbol that can be used in my code to get the right version.

Use OTADrive to remotely update your ESP devices

I’m making a bunch of devices that I’ve decided to call “clever little boxes”. I’ve taken the important first step and bought the domain name. Now I have to make everything else. I’ve decided that the best place to start is with the deployment. You might think that this is crazy, but actually it is the most important phase of your development. If I’m not able to easily update the devices in the field and manage their deployment I won’t have a product, just a demo. Also, modern product development is a continuous process. You don’t make something and ship it. You make version 1.0 and then create version 2.0 based on the feedback you get.

To help me with this I’ve started using otadrive. Once you have created an account you can store firmware images on the site and then your remote devices can download and install new firmware images from them. You’re given an access key and a web based interface for uploading firmware and managing the devices that need to be kept up to date.

You get a tiny bit of Arduino code you can drop into your device to check for new versions. It works a treat. The only snag in my mind is that the site doesn’t seem to have a business model. At no point in the process of configuring and deploying versions have I been asked for any payment. This worries me for two reasons:

• I might suddenly be hit with an enormous price list which makes the whole thing unviable.

• The site might vanish overnight taking with it my entire network of devices.

I’m not that concerned just at the moment though. And if things get tough I can look at this on GitHub which might be where I end up putting everything.

Wide eyed and pinless

The ESP32 is mostly my weapon of choice when making embedded devices. The only problem that I’ve had with the device is that tends to be supplied as a device with ready soldered pins. This is fine if you want to plug it straight into a socket but I don’t tend to do that. I’m much happier soldering the wires directly onto the unit. This makes for a more reliable device and also allows me to use smaller boxes. T

The good news is that I’ve now found a supplier on AliExpress who is selling versions that don’t have the pins already fitted. They cost slightly more than the “normal” devices (which I find a bit surprising) but they do give a lot more flexibility.

One thing to remember though; it is a pain to build the ESP32 device into your device and then find that you have a faulty one. So make sure you test the ESP32 before you solder it in.

Maketober Day 11: PIR controlled ESP32

I’ve been playing with PIR sensors. In this crazy world you can pick up five of them for around ten pounds. You can see one in the picture above. It’s the half bubble right in the middle of the frame. These trigger when they see something warm blooded walk past. They are used in burglar alarms and the like. I bought a bunch to play with. I want to make a PIR controlled nightlight that comes on when you walk past it. I’m going to use Neopixels as the light source.

The PIR sensor has an output pin that goes high when it sees someone go past. You can use the little orange trimmers on the device to adjust the sensitivity of the detector and how long it stays on once it has been triggered. You can also make the output signal a pulse or stay high as long as it detects something by moving the yellow jumper on the board.

I’d really like the whole thing to be battery powered. According to its data sheet the PIR sensor only sips tiny amounts of current, but this is not true for the ESP32 I’m using to control everything. The good news is that the ESP32 can be made to go into “deep sleep” mode in which most of the processor shuts down. We can use the trigger signal to wake up the processor when it sees someone. The ESP32 can then turn the light on and even log the detection over the network using MQTT or whatever. I’ve been playing with some code to make this work and I’ve come up with the following:

def startup():
    global np,pin,n
    if machine.reset_cause() == machine.DEEPSLEEP_RESET:
        # we can do stuff here when the PIR was triggered
        print('woke from a deep sleep')
    else:
        print('not been sleeping')
    # PIR on GPIO 4
    pin = machine.Pin(4, machine.Pin.IN)
    # configure to wake up ESP32 when the PIR is triggered
    esp32.wake_on_ext0(pin,esp32.WAKEUP_ANY_HIGH)
    # put the ESP32 to sleep
    print("Going to sleep")
    machine.deepsleep()

This MicroPython function can be added to a program that runs when the device is booted. In other words, you could put a call of startup() into the file boot.py

It checks to see if the ESP32 was woken from a deep sleep (i.e. the PIR triggered the restart. You can put whatever code you want to run in that part of the code. Then it configures pin GPIO4 as an input and tells the ESP32 to wake when this input goes high. Then it puts the ESP 32 into deep sleep.

I’ve tested it and it seems to work. The only problem is that not all ESP32 devices are equal as far as deep sleep power consumption is concerned. The DOIT board that I’m using leaves all the USB hardware switched on when it goes to sleep, which means that it still consumes quite a bit of power. If you want to do this kind of thing with an ESP32 I strongly recommend the DFRobot FireBeetle which only takes tiny amounts of power in sleep mode. Once I’ve made the code work I’ll move onto that platform.

Makertober Day 7: Led Array

I’m totally sold on Python development for embedded devices now. I had an idea for a silly device and ordered an 8x32 led array powered by Max2179 devices. The leds arrived last night and I plugged them into one of my cheap ESP32 devices and got it sort of working. I found an Arduino library and, after tweaking the C++ library code I manged to make it build and almost display something. Writing and deploying C++ from the Arduino IDE again reminded me why I don’t like this very much anymore. The compilers are very brittle and prone to failing if you happen to use a library that contains code that is not supported by the device you are targeting . And deployment is tiresome and slow.  I think that the Arduino toolchain is an awesome achievement and makes lots of lovely things possible but it can be a bit of a pain to use sometimes.

Rather than continue with that I thought that today I’d try using putting MicroPython on the ESP32 and using a Max2179 library that I found on GitHub. This. Just. Worked. I used Thony to write and deploy the code. I was able to use the REPL command prompt to test out the code and my tiny little programs took no time at all to deploy and run inside the device. Best of all, the Max2179 library I found just sits on top of the Framebufuffer (http://docs.micropython.org/en/latest/library/framebuf.html) class and has lots of easy to use graphics commands.

I’ve got text scrolling nicely and now I’m going to hook it up to a data source. If you want to have a go at this kind of thing I’ve written a little HowTo that you can find here.

Furby Article now in HackSpace Magazine

I’ve been turning out the odd (and I do mean odd) article for HackSpace magazine recently. You can find my description of how to use an ESP32 to control a Furby toy in the latest issue here. I’ve written a little set of helper files that let you connect to a Furby toy, but you can also use the code to talk to any Bluetooth BLE device from your ESP32. You can find the code for the article on GitHub here.

M5 Atom and Tailbat

The M5 Atom is a neat little ESP32 based device. They now do a version with 25 multi-coloured leds on it which is rather nice. You can also get a battery pack with the wonderful name of the “Tailbat” to plug into it and keep it going. You can program the device in Arduino C++, use MicroPython or even the UiFlow block based environment, all over the air from a browser.

Great fun and loads of potential.

Big programs in an ESP32

As part of the Furby connect development my plan was to be able to control the Furby from the internet using MQTT. This means that the ESP32 device I’m using will need both Bluetooth and WiFi. WiFi to talk to the MQTT server and Bluetooth to talk to the Furby. Both of these protocols are implemented by large lumps of code in the ESP32 libraries and when I turned them both on I promptly ran out of memory on my device. Wah.

A bit of searching revealed that there should be some menus in the Arduino IDE (see above) that you can use to specify how much of the program storage you want to use. By default a device is set up for Over The Air (OTA) updates, which means that only half of the available memory is available. Turn off OTA and you can install bigger programs. However, when I tried to do this on the devices that I was using there was no such menu. Double Wah.

It turns out that some of my devices (the DOIT and the FireBeetle) contain a version of the ESP32 chip (the ESP-VROOM-S2) that has built-in EEPROM program storage (which makes the device simpler and cheaper) but it also limits the maximum program size so the size cannot be extended. But on larger devices you can extend the application. This means that from a software point of view I can have a device that talks both WiFi and Bluetooth at the same time. However, since both technologies use the same frequency band (and on the ESP32 the same radio) it might not work. My fallback solution is to just use two ESP32 devices, one for WiFi and one for BLE. I can link them with a serial connection and then all my problems should go away.

It’s important to remember that throwing hardware at a tricky problem is not such a daft idea. In this case it eases the development and completely removes a whole set of possible failure areas. At the cost of around another fiver or so in hardware.

M5 Atom is absolutely tiny

A new computer arrived at our house today. The postman just popped it through the letterbox. The Atom is a tiny new microcontroller from M5Stack. It contains a button, a multicolour led and an accelerometer. It has enough pins to be useful along with a Grove socket.

It’s a little bit bigger than the top of my thumb, it contains an ESP32, it can run MicroPython and hang off Azure or Bluetooth. And you can buy them for six dollars apiece.

I know that the world today is a right old mess, but there are still some amazing things to be found.

Disabling the ESP32 Brownout detector

I hate it when things fail when they are not supposed to. I’ve now got some code that uses deep sleep on an ESP 32 to drop power consumption to a tiny amount between the readings made by the environmental sensor hardware that Brian has built.

Today I was doing some battery testing and I hit a snag. When the device is running from battery power the deep sleep mode breaks. The device does a power on reset rather than waking from its deep sleep as it is supposed to. Of course, this only happens when it is not connected to a computer, so I can’t use any of my debugging tools to find out what is going on. In the end I had to resort to flashing the LED on the device to indicate the reason for the reboot.

void flashLed(int flashes)
{
    pinMode(LED_BUILTIN, OUTPUT);
    for (int i = 0; i < flashes; i++)
    {
        digitalWrite(LED_BUILTIN, true);
        delay(500);
        digitalWrite(LED_BUILTIN, false);
        delay(500);
    }
}

void flashBootReasonOnBuiltInLed()
{
    esp_reset_reason_t reset_reason = esp_reset_reason();
    flashLed(reset_reason);

    delay(2000);

    esp_sleep_wakeup_cause_t wakeup_reason;
    wakeup_reason = esp_sleep_get_wakeup_cause();

    flashLed(wakeup_reason);
}

This is the code that I ended up writing. On usb power I get a reset reason of 8 flashes (deepsleep) and a wakeup reason of 4 flashes (timer). All good. On battery power I get a reset reason of 9 (brownout).

OK, so what is a brownout? A brownout is when the supply voltage drops a bit low. It’s something you need to worry about because low voltage can cause your processor to do some very strange things. It might get its sums wrong or forget stuff. The ESP32 has hardware that checks the supply voltage and resets the processor if it detects that the power supply is failing. Software in the device can then detect this and go about shutting down safely.

In the case of our sensor the brownout is caused by one specific event. It occurs when the program turns on the power supply that drives the particle sensor. The particle sensor has a fan in it, which takes quite a chunk of current just as it spins up. This sudden load causes the battery voltage to drop for a fraction of a second. The sequence that we have is as follows:

1. Sensor wakes up from a deep sleep.

2. Sensor runs the code that turns on the power to the particle sensor.

3. Supply voltage drops and triggers the brownout sensor which resets the ESP32.

4. The ESP32 wakes up as from a brownout reset, not a deep sleep.

5. Sensor runs the code that turns on the power again, but this time it doesn’t trigger the brownout sensor because the power supply has a little bit of residual power in it from the recent (failed) startup.

I’ve fixed the problem by turning off the brownout detector when I turn the power on.

WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0); //disable brownout detector   

// turn on the power
digitalWrite(powerControlSettings.powerControlOutputPin, HIGH);

// let things power up
delay(500);

WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 1); //enable brownout detector

This is the code that does the deed. The power is controlled by a pin that is lifted high to turn it on. So the program turns of the brownout detector, turns the power on, waits half a second for things to settle and then turns the brownout detector back on again.

I’m not sure whether to be proud of this code or not. But in the end I think I am. If you accuse me of masking a hardware problem with a bit of software then I’m going to plead guilty as charged. I could probably fix the hardware cause by adding a wacking great capacitor over the supply to stop the voltage dropping. But this might lead to other problems. And anyway, if you look at the programs in any of the devices that you use on a daily basis you will find bits of code like this. Little bits of behaviour that are only there because the hardware doesn’t work without them.

Oh, and as a bonus (and so I can find it again if I ever need it) here’s a chunk of C that prints out the reset reason for an ESP32.

void printBootReason()
{
    esp_reset_reason_t reset_reason = esp_reset_reason();

    switch (reset_reason)
    {
    case ESP_RST_UNKNOWN:    Serial.println("Reset reason can not be determined"); break;
    case ESP_RST_POWERON:    Serial.println("Reset due to power-on event"); break;
    case ESP_RST_EXT:        Serial.println("Reset by external pin (not applicable for ESP32)"); break;
    case ESP_RST_SW:         Serial.println("Software reset via esp_restart"); break;
    case ESP_RST_PANIC:      Serial.println("Software reset due to exception/panic"); break;
    case ESP_RST_INT_WDT:    Serial.println("Reset (software or hardware) due to interrupt watchdog"); break;
    case ESP_RST_TASK_WDT:   Serial.println("Reset due to task watchdog"); break;
    case ESP_RST_WDT:        Serial.println("Reset due to other watchdogs"); break;
    case ESP_RST_DEEPSLEEP:  Serial.println("Reset after exiting deep sleep mode"); break;
    case ESP_RST_BROWNOUT:   Serial.println("Brownout reset (software or hardware)"); break;
    case ESP_RST_SDIO:       Serial.println("Reset over SDIO"); break;
    }

    if (reset_reason == ESP_RST_DEEPSLEEP)
    {
        esp_sleep_wakeup_cause_t wakeup_reason = esp_sleep_get_wakeup_cause();

        switch(wakeup_reason)
        { 
            case ESP_SLEEP_WAKEUP_UNDEFINED:    Serial.println("In case of deep sleep: reset was not caused by exit from deep sleep"); break;
            case ESP_SLEEP_WAKEUP_ALL:          Serial.println("Not a wakeup cause: used to disable all wakeup sources with esp_sleep_disable_wakeup_source"); break;
            case ESP_SLEEP_WAKEUP_EXT0:         Serial.println("Wakeup caused by external signal using RTC_IO"); break;
            case ESP_SLEEP_WAKEUP_EXT1:         Serial.println("Wakeup caused by external signal using RTC_CNTL"); break;
            case ESP_SLEEP_WAKEUP_TIMER:        Serial.println("Wakeup caused by timer"); break;
            case ESP_SLEEP_WAKEUP_TOUCHPAD:     Serial.println("Wakeup caused by touchpad"); break;
            case ESP_SLEEP_WAKEUP_ULP:          Serial.println("Wakeup caused by ULP program"); break;
            case ESP_SLEEP_WAKEUP_GPIO:         Serial.println("Wakeup caused by GPIO (light sleep only)"); break;
            case ESP_SLEEP_WAKEUP_UART:         Serial.println("Wakeup caused by UART (light sleep only)"); break;
        }
    }
}

You’re welcome.

LMIC Frame Counter Problem

This is a fairly exotic post, in that I doubt that many readers will be concerned how they can maintain the frame count of LoRa packets sent using the LMIC library when a device goes into deep sleep. However, I’m putting it here so that in six months time, when I probably need to do this again, I can just search the internets and find this article.

A LoRa application maintains a frame count for each connected device. This is a security measure. Any frames that that the application receives with a frame count that is lower than the highest frame count seen so far will be rejected by the application. You can turn off this frame count checking for a device, but it is not advised because it makes things a bit less secure. I’ve turned it off for my devices for now, which meant that the frame count just increases when the device is running and goes back to 0 if the device is reset.

This is kind of OK by me. I’m not overly concerned with security issues at this level. However, when I started using deep sleep mode I found that every frame that was sent had a frame counter of 0, and I really didn’t like that. The problem is caused by the way that the device is completely restarted between each transmission. This restarts the LMIC library each time, setting the frame count to 0 for every frame.

It turns out that a program can reset the LMIC frame count value by accessing a member of the LMIC object. All my program needs to do is create a variable in the RTC memory to hold the current frame count:

RTC_DATA_ATTR u4_t lmicSeqNumber = 0;

The program can then store the frame count in this variable after each LoRa message has been transmitted:

lmicSeqNumber = LMIC.seqnoUp;

Now, during the reset process after a deep sleep I can put the frame count back:

LMIC.seqnoUp = lmicSeqNumber;

I’ve tested this and it seems to work fine. The frame counter is retained when the device goes to sleep. Of course if I turn the device off the frame counter resets back to 0. I could fix this by storing the frame count in EEPROM storage but I’m not too keen on doing this because I’d have to update the value after each LoRa message and might lead to lots of writes which might “wear out” the EEPROM storage.

ESP32 Retaining timing over deep sleep

In the last post we saw how easy it is to make an ESP32 processor sleep for a particular time. However, we also noticed that at the end of the sleep the processor is reset and all the variables in the program are reset. This is difficult if you want to keep track of time in your application. However, it is possible to get around this limitation by storing a time value in memory that is retained during sleep.

RTC_DATA_ATTR unsigned long millisOffset=0;

The statement above declares a variable called millisOffset. The attribute RTC_DATA_ATTR tells the compiler that the variable should be stored in the real time clock data area. This is a small area of storage in the ESP32 processor that is part of the Real Time Clock. This means that the value will be set to zero when the ESP32 first powers up but will retain its value after a deep sleep.

We can use this variable to create an offsetMillis function that gives a millisecond count which is adjusted by the offset:

unsigned long offsetMillis()
{
    return millis() + millisOffset;
}

Our program is going to call this function when it wants to know how much time has elapsed since the application was started. The final part of the solution is to update the offset each time our processor is put to sleep.

void sleepSensor(unsigned long sleepMillis)
{
    esp_sleep_enable_timer_wakeup(sleepMillis * uS_TO_mS_FACTOR);

    millisOffset = offsetMillis() + sleepMillis;

    esp_deep_sleep_start();
}

Now, when our application goes to sleep it records the value that the millisecond timer should have when the device wakes from sleep. This value is retained in non-volatile memory and used to offset time values when the ESP32 restarts.

#include <Arduino.h>

RTC_DATA_ATTR unsigned long millisOffset = 0;

unsigned long offsetMillis()
{
    return millis() + millisOffset;
}

#define uS_TO_mS_FACTOR 1000  /* Conversion factor for micro seconds to miliseconds */

void sleepSensor(unsigned long sleepMillis)
{
    esp_sleep_enable_timer_wakeup(sleepMillis * uS_TO_mS_FACTOR);

    millisOffset = offsetMillis() + sleepMillis;

    esp_deep_sleep_start();
}

void printTime()
{
    unsigned long seconds, sec, min, hrs;

    seconds = offsetMillis() / 1000;

    sec = seconds % 60;
    seconds /= 60;
    min = seconds % 60;    
    seconds /= 60;
    hrs = seconds % 24;

    Serial.printf("%02d:%02d:%02d\n", hrs, min, sec);
}


void setup() {
    Serial.begin(115200);
    printTime();
    sleepSensor(30000);
}

void loop() {
}

The complete program above shows how it all fits together. The program sleeps the device for 30 seconds but the time value is maintained after each reset. Note that the loop function is empty because the program never gets this far. The repeating behaviour of the program is caused by the reset after each sleep.

I was quite surprised just how poor the time keeping is when I ran this program. This is because the timing is not provided by a crystal but by an oscillator which is fitted onto the ESP32 chip. The timing can be out by a second or so over short intervals. Try timing the above program to see what I mean.

The millis function in the Arduino library returns an unsigned long integer value that holds the number of milliseconds since a device was turned on. The program above works by adding an offset to this millis value which reflects how long the device has been put to sleep. Of course, what with data storage in any computer being a finite size, there will come a point where the millis value will not fit in an unsigned long variable and so it will wrap round and go back to 0. This occurs after 4,294,967,295 milliseconds or around 50 days. If your program does any kind of calculation with the timing values you will need to make sure that the wrap round doesn’t cause problems.

TTGO Camera

A while back I ordered a TTGO camera from Ali Express. It came today and it is lovely. It is similar to the ESP32 webcam that I bought a while back, but it has a PIR sensor and a BME 280 environmental sensor along with a nifty OLED display.

It is supplied with firmware that sets it up as an Access Point. If you connect to it you can view the image and control it from the web page that it serves out. I think it is running the same software as I loaded into the earlier device, but it is nice to have it on the device at the end.

It works extremely well. I’ve printed a little case for it from a design I found on Thingiverse and now I’m looking forward to having a play. They say it can do facial recognition, which is very interesting, and it might be a good device for our Humber Care Tech Challenge entry.

Update: I’ve found a nice blog post here on how to use this device as a fridge alarm. This shows how to put the processor to sleep and trigger wakeup using the PIR sensor.