Chunking your ESP32 Web responses

Note that this is not actually how computer memory works

Great title for a blog post eh? One of the wonderful things about modern embedded devices is that they can host web sites. You can use your phone to browse content served by a device that cost less than a pint of beer. Although some people prefer to buy the beer.

Anyhoo, I’ve been making devices that host web pages as part of the configuration process for my Connected Little Boxes. You turn the device on, scan a QR code with your phone which connects to a WiFi hotspot served by the device and then scan another QR code to get to a configuration page so that you can enter your local WiFi credentials and generally set the device up. This is how lots of connected devices work, and I wanted to make my own system for doing this. And I ran into problems with memory. Not forgetting what I’m supposed to be doing (ho ho) that happens all the time. More like the way that once I’ve got everything running on my little ESP device I’ve got hardly any memory left for large things (comparatively) like web pages.

Now, one way to serve out a web page is to put the page content into a string and then send that string out. This works, but you need enough memory to hold the entire string. Which it turned out I didn’t have. So a better way is to send the page a chunk at a time. The ESP8266 web server has a way of doing this:

server->chunkedResponseModeStart(200, F("text/html"));
server->sendContent(settingBuffer);
server->chunkedResponseFinalize();

The first statement tells the server I’m sending a bunch of chunks. The second statement sends a chunk of text from the settingBuffer string. You can send as many chunks as you like. The third statement ends the chunking. Very useful, particularly when you discover that you can send out chunks of text that are stored in program memory rather than ram.

So I got all this working and then I wanted to move the code to the ESP32 device. This chip is a bit more expensive than the ESP8266 but it is a lot more powerful. And it doesn’t do chunking. Wah!. Fortunately I found this wonderful post which told me how to extend the ESP32 web server to make chunked responses work in the same way as the ESP8266. It will be part of the latest version of the Connected Little Boxes embedded code (tentatively titled HULLOS-X) soon.

Debugging with Magic

Trying a new blogging approach. Blogging when I have something to say, rather than fretting about making a post every day……

Only a code magician can fix a program by renaming a variable. That’s me.

I’ve just encountered an hilarious (and by hilarious I mean horrible) bug with an upgrade to the ESP8266 compiler. I’ve been building my Connected Little Boxes software for quite a while now and it just works. Today I noticed that I had upgrades to the Espressif SDK in Platform IO. So, like a fool (and I do mean that) I pressed OK to upgrade them.

And suddenly my newly built program broke. It would start running and then spontaneously explode with Exception(29). Which means that my program is twiddling a memory location that is not where it should be.

First step was to recompile the code for the ESP32 processor (my code works on both). That worked fine. OK, so it is not necessarily something stupid that I’ve done. Back to the ESP8266.

It’s hard to debug code running in a separate device, but after a few minutes (I am pretty good at this - although I was slowed down as bit by the way that the bug vanished when I enabled debug mode) I’d isolated it to this statement:

RegistrationProcessDescriptor.status = REGISTRATION_OFF;

It runs during startup and turns off the registration process. But I guess you’d already worked that out because my commenting game is so good. Anyhoo, when this statement runs the device explodes.

OK. Welcome to planet weird. First thought is that RegistrationProcessDescriptor is not pointing to the right place. But it is. Otherwise the call to the exploding function would not have worked. So, second thought is that REGISTRATION_OFF is a silly value. It isn’t. It’s 1004 since you asked. And status is an integer property in case you were wondering, so everything make sense and the code has been working perfectly for ages…..

Of course I’ve fixed it. I’ve renamed the variable RegistrationProcessDescriptor to RegistrationProcess. Something deep in the compiler must be getting upset about long variable names and dropping dodgy code. There were a few clues:

• It works with a different compiler

• It worked when I changed the code a bit (by adding debug)

• The variable name was a bit longer than ones I normally use

I’m not happy that the problem occurred, but I’m pleased I was able to fix it.

Simple encryption with the ESP32

Early versions of the software for my Connected Little Boxes stored all the settings in an area of EEPROM memory. This is because the settings code was originally written to run on an Arduino Uno which only provides EEPROM as persistent storage.

Now that I’m using a the ESP8266 and ESP32 I can use a proper file system to store the settings for a device. This is nice, but has left me with a problem. On the Arduino Uno I was quite happy to store passwords in EEPROM. My software won’t let you read back the contents of password settings, you can only put values in. If you want to read the settings you’d have to get hold of the device and then swap the internal program for one which shows you the contents of the EEPROM. But with the new code I’ll have a bunch of files on the device which someone might be able to just take a look at.

So I started looking at really simple encryption. Not really encryption as such, just something to make it impossible for someone reading one of the settings files to be able to read back the values of protected settings. It’s not really proper encryption as the key and the code which uses it are both stored in the device so anyone with the time and the inclination could break it. However, I think what I’ve done is OK for its purpose.

#if defined(ARDUINO_ARCH_ESP32)
#define PROC_ID (unsigned long)ESP.getEfuseMac()
#endif

#if defined(ARDUINO_ARCH_ESP8266)
#define PROC_ID (unsigned long)ESP.getChipId()
#endif

#define ENCRYPTION_SALT 1234

void encryptString(char * destination, int destLength, char * source)
{
    randomSeed(PROC_ID+ENCRYPTION_SALT);
    int pos = 0;
    char * dest = destination;
    destLength= destLength -1;
    while(*source)
    {
        int mask = random(1,30);
        *dest = *source ^ mask;
        dest++;
        source++;
        pos++;
        if(pos==destLength)
        {
            break;
        }
    }
    *dest=0;
}

This is my encryption code. You give it a string and it scrambles the text. I’ve done it this way so that the characters codes still remain in the printable ASCII range. I use the processor ID number of the device and a salt value for each device to seed the built-in random number generator. I then use the magic of exclusive-or to scramble the text. The decrypt process is exactly the same code.

It seems to work OK. I only encrypt the passwords themselves. This reduces the amount of data that a cracker has to work with. You could crack it using any one of a number of attacks, but what you can’t do is just read out the text from a settings file and then use it, which is the level of security I wanted.

The way I see it, once someone gets hold of your physical device all bets are off security wise. Particularly if the algorithm is in the public domain too. That’s why I advise you to make a guest WiFi network for your IoT devices so that you can reduce the effects of a security breach.

IoT Cricket looks very interesting

This looks like a very interesting device. It’s called the IoT Cricket from Things on Edge. It looks like it brings Lora levels of battery performance to a WiFi connected device. It apparently has the ability to wake up, connect to a WiFi network and push out a packet of MQTT data in around 8 seconds. Which is pretty darned fast. There is only very simple PIO available for data inputs but there is onboard temperature and a clock that turns everything off between readings. You can power the device from one or two AA batteries or a LiPo and they are talking about battery lifetimes in the months and years.

It’s only 16 quid to get one, which will be a good price if it does what they claim. I’ve ordered one, looking forward to having a play.

Free Azure Sphere Development Kit

One of my big concerns about the Internet of Things is that people might not do it properly. By that I meant that they will rush to market with the ability to put connected control into all kinds of situations without worrying about security, deployability and maintainability. That’s one reason why I very much like a new competition where you get to play with an Azure Sphere embedded device which has those abilities at its heart.

Another reason why I like the competition is that you can get a free device to play with if you enter. You can find out more here.

Using Serial1 with the Azure IoT Devkit

A while back I posted about changes you could make to the Azure IoT DevKit library so that your programs can use the second serial port of the device to talk to things like GPS receivers and Air Quality sensors (a particular interest of mine).

At the time I suggested that this could be fixed by adding the creation of a second serial port to the SDK. I’ve just heard back from the team and this is not going to be done. Instead we can create a Serial instance when we need one, which saves resources.

If you want to use Serial1 with your Azure IoT device you can add the following statement to your program.

UARTClass Serial1(UART_1);

You can then use Serial1 in exactly the same way as you use Serial. I’ve put a sample program on GitHub that shows how this works.

Tank Controller Fun and Games

Spent a happy afternoon playing games back at the university today. We were discussing new designs for the configurable controller project that we've been working on. It was great to play an awesome tank battle game using our existing controller. The new one is going to be even better....

mDNS Manager for Windows 10 IoT Universal Apps

You know how it is. You have built a robot army that you're going to use to take over the world, but first you have to get them all under your control. And if you are using tcp/ip (the world domination network of choice) then you have to give them all an ip address and then put those addresses into your world domination program.

mDNS makes this much easier. It's how Apple's Bonjour network discovery works. A device running mDNS is discoverable on a local network. You can find all the hosts and their ip addresses, along with the services they are providing and the ports. Windows 10 provides a Watcher service that you can use to discover all the machines on a local network, but it is a bit of a pain to use.

So I've written a tiny mDNSManager class that you can use to create and manage a watcher object which will start a search for devices, tell you when it finds one and also present a list of all the devices it has found so far. It's very easy to use. 

You can find the source code for both the manager and a simple demonstration application on GitHub here. 

If you want to just use the manager in your programs you can install it from NuGet:

Install-Package RobMiles.mDNSManager 

It works on any Windows 10 device, including the Raspberry Pi, and it makes it much easier to connect to a large number of devices. There are mDNS samples available in the example applications for the ESP8266 devices in the Arduino IDE.

I'm using this to allow me to create a Universal Application that will control a bunch of HullPixelbot devices without needing to know their ip addresses in advance. 

Bluetooth Manager for Windows 10 Universal Apps now on Nuget

I've been playing with nuget. 

It's awesome. 

It is fantastically easy to package up an assembly and make it available for just anyone to use in their applications. It is beautifully integrated into Visual Studio and I reckon it is actually easier to download a package from the other side of the world and make it part of your solution than it is to add one that you have written yourself on your own machine. 

To practice I've packaged up my Bluetooth Manager library. This is a little wrapper class that makes it really easy to use Bluetooth on Windows 10 applications. I've not tried it on Windows 10 on Raspberry Pi (but I've tried it on lots of other systems and it works fine). You can find out a bit more about it here.  I use it so that I can print message on my little home made Bluetooth printer, but you can use it anywhere you want to talk over a Bluetooth serial connection. 

Oh, and in case you are wondering why it is version 1.0.1 and not version 1.0.0 I found an interesting quirk in nuget. If your library class doesn't have a public constructor the package will fail to work because Visual Studio will complain that the class is "Inaccessible due to its protection level.". That's what happens when you try to use version 1.0.0

Taking a look at the Photon

The Photon is a neat little embedded device a bit like the Electric Imp I played with a while back, but  a bit cheaper to get started with. You write C code in your browser and the program is deployed over WiFi to the device. The hardware is very like the Arduino. The programs can communicate via the cloud using web requests and there are also bindings for If This Then That.

To be honest, I've not done much with it, but number one son has had a play and things it's quite neat. The only concern I have with these systems is that you are a bit dependent on the cloud backend. I've got a collection of cloud enabled devices that are now just paperweights, and I worry that things like the Photon might go the same way. Having said that though, for the price it is a lot of fun. 

Arduino Kits Going Live

So a couple of weeks ago I got the go-ahead to buy some Arduino kits for our Second Year embedded course. We have been using the Arduino robots and they are nice enough, but a bit too closed for my liking. So I went on-line and tracked down a lovely little Sintron kit on ebay. Twenty Five pounds for loads of bits and bobs.

Tomorrow I'm dishing out the hardware along with a bunch of little plastic bags and a page of sticky labels. Job one will be to use the chart above to put all the components in labeled bags so that we can find stuff later. Job two will be to get some lights flashing and buttons being pressed.

Exciting times.  

Windows 10 on Raspberry Pi

Windows 10 on Raspberry Pi works. In fact it works very well. You can deploy programs from Visual Studio 2015 into the Pi and they just run. You can even put breakpoints in the code while it is running, and remotely debug your code over an IP connection. Just like the "Good Old Days" (tm) when I was putting.NET Microframework code into embedded devices all those years ago. Except that Pi applications have proper .NET behind them and I can build a user interface using WPF.

The hardware connections are abstracted into classes, just like they should be:

const int LED_PIN = 5;
var gpio = GpioController.GetDefault();
GpioPin pin = gpio.OpenPin(LED_PIN);

This makes a pin that the program can use to interact with the outside world.

pin.Write(GpioPinValue.Low);

This sets the pin low.  A program can read and write the status of the pin and bind to events that fire when the pin changes state. This makes embedded development really easy.

Installation was smooth enough, although you'll need a PC running Windows 10 with Visual Studio 2015 RC to get started.

The walkthroughs are well written and accurate. You can get started here:

http://ms-iot.github.io/content/win10/SetupPC.htm

 One tip, if you have any problems getting Visual Studio to work after installation go to this page:

https://dev.windows.com/en-US/downloads/windows-10-developer-tools

Scroll to the bottom of the page and install "Standalone Windows SDK for Windows 10" and "Microsoft Emulator for Windows 10 Mobile". This fixed a few problems I had with stuff not installing properly.

I'm properly excited about this. I'm going to have a go at getting my Galileo to run Windows 10 as well. There are also bindings for frameworks that can talk over Bluetooth to Arduino devices.

Update: Turns out that Galileo doesn't run Windows 10 core, it uses the previous version of Windows. Apologies for any confusion.

Vetinari Clock Complete

Tonight I thought I'd build my Vetinari clock. It's a silly thing, but great fun.

Lord Vetinari is a character in the Discworld series of books by the now sadly deceased Sir Terry Pratchett. Vetinari, a powerful ruler of the largest city on the Discworld, has a clock in his waiting room which is designed to unnerve people by ticking irregularly. However, it still keeps accurate time. 

You can get a kit of parts to convert any quartz controlled clock into a Vetinari clock. It uses a PIC microcontroller to generate a sequence of 32 pulses of different lengths which give the appearance of randomness. But the clock still keeps time accurately.

I ordered mine a way back and, after my expensive trip to Ikea on Sunday, I finally got a clock mechanism I thought worthy of the conversion. Tonight I put it together and carefully connected the outputs from the circuit to the driver coil in the clock mechanism. It works a treat. If you are a fan of Discworld who likes soldering you might have some fun with this.

Hmm. Perhaps a video of the clock would make more sense....

Four Letter Word Generator

Well, here's something completely crackers that you can buy for less than ten pounds from Tindie. It's a Four Letter word associative database all on a single tiny chip. You can ask it for a random word, and then get another word kind of associated with it. There's a blog post here that tells you how it works. 

Of course I ordered one. It arrived today in a big padded envelope, sent all the way from Japan. Next I have to build a bit of hardware to read the words out and display them. I'm thinking it might make an awesome idea generator for Three Thing Game.  The "Three Thing'omatic", I can see it now.....

Raspberry Pi 2 and Windows 10

News of the Raspberry Pi 2 took me completely by surprise. And then, to add to this shock I find that it is not only an awesome device, but available to buy right away. Amazing. Who do these people think they are? Apple?

If the news of a new Pi with quad core processor and twice as much RAM wasn't enough though, I then find that the new Pi will be able to run Windows 10. For free. The only word I can think of to sum up this turn of events is blimey. 

The Windows 10 support will be limited to "universal" applications, which means an end to  any dreams of running Visual Studio on a device you paid 25 quid for. But that does mean that you can use Visual Studio to write and deploy programs onto the device, which will be great. You can find out more about Windows 10 on Pi here.

The development puts a mild question mark over devices such as Galileo and Edison, which are of the same ilk, but are more expensive and have no display capability. I guess everything will settle down and find its place eventually. The two Intel devices probably have a bit more straight line speed, although the 900 MHz four core processor in the new PI is not to be sneezed at. 

Anyhoo, my Raspberry Pi 2 is presently sitting in Cottingham Post Office waiting for me to drop round and pick it up. I actually bought 2. One to give the Video Arcade Table a shot in the arm, and the other for playing with.  We really do live in interesting times. 

Update: I've had a chance to do a very quick benchmark and compare the performance of the original and the new Rapberry Pi. You can find out more here. 

Embedded Developers Club

Quite a few folks seem very keen to set up an Embedded Development Club in the department. And that's just the staff members. We'll see how many students turn up at our first meeting on Wednesday 4th of Feb. at 2:15 pm in room 321 (the ITMB Lab) in the Robert Blackburn Building. 

You can find out more here

Ultra Cheap Arduino WiFi

Embedded devices get really interesting once you connect them together. There are a few ways to do this, you can use the same kind of modules that I used for my Wedding Lights, or you can use Bluetooth. Or you can use WiFi. WiFi is by far the best way to link an embedded device to a surrounding network, but up until now the cheapest WiFi  hardware for a device like the Arduino would cost around 25 pounds.

But now we have the ESP8266 chip. This is tiny (as you can see) , costs less than five pounds, and puts your Arduino on the local WiFi network. 

A few health warnings: The device needs a high speed serial connection to the host computer, and so you must use the proper hardware serial port on your Arduino, not a software one. Also they are not very good at address discovery and so you will probably have to hard-wire the IP address. This is fine if you are using them at home but might make them a bit tricky to deploy on a corporate network. They will let a program connect to a TCP socket on a remote machine and send packets backwards and forwards, but that is about it for the moment. Fortunately that is just about what you want from such a device. 

Peter has been playing with them and getting then to work I've bought some (and put them in envelopes in my little boxes) and I'm looking forward to having a play with them.

Home to a Bluetooth Printer

We spent the night in Whitby at the Dolphin Hotel. We had a lovely large room which overlooks the bridge right in the centre of the town. I took the photo from the room first thing in the morning, just before we headed down for a really nice (and huge) breakfast. If you are looking for somewhere to stay, I strongly recommend the place.

Then we headed home and I found some time to finish off the hardware for the Bluetooth printer I've been working on.  Note how it is a Bluetooth device, and I've put it in a bright yellow enclosure. And why not....

When I designed the box it seemed like a good idea to put the switch in the bottom. Of course it is actually a bit silly, as whenever you put the box down you turn it on or off. Fortunately I have some bit stick-on feet that help with this, but I'm going to have to refine the design anyway as I don't seem to have left much room to allow the actual construction of the device.....

This is the guts of the printer, just a bunch of batteries and a trusty Bluetooth adapter. I'll post full details of construction and the software later this week. 

Using Big Led Panels with Arduino

So last week I found these Big Led Panels at Cool Components. And today I got one working. The wiring was quite fun, pushing jumper wires into connectors.  Turns out that was the easy part though. Once I'd got the hardware connected I had to get the software working. There are a couple of libraries out there that you can just download and use. But they didn't work. 

Whenever I tried a build I got loads of messages about symbols not being present in my code. I'm not sure why, but the build process seems to search the entire library structure for include files and it found some Adafruit_GFX ones in the Robot_Control folder that caused real problems. The solution was just to remove the Robot_Control folder from the Arduino libraries folder. After that the programs built and I got some stunning displays. The picture above, even after some serious tinkerage, doesn't really do them justice. 

This is a bit better, but doesn't really capture the eye-watering brightness of the pixels. The display is quite solid, but my little Arduino Uno is going pretty much flat out just to generate the colours. There was noticeable flicker during the display updates and so I think I need something with a bit more processing grunt to get smooth animation on it.

One plan is to make a coffee table, but I've just found some 8x8 inch picture frames that I think would take a panel very nicely. So maybe I put the panels in frames and turn them into works of art. Either way, great fun is being had....

Wednesday Open Day with Light Panels

We had another Open Day today. A slightly more select gathering than the last jam-packed one. I showed off my Windows Phone remote controlled coloured light to good effect, and then I went on to mention to folks that, for some reason, I had bought one of these:

It is a whacking great big display panel, like the ones you see on the side of buildings. It is 7.5 inches across and contains 1,024 RGB leds in a 32x32 matrix. I've been playing with light panels for a while, and this one is awesome. 

The only snag is that, unlike the Adafruit Neopixel devices that I'm familiar with, on this one the led's don't remember the colour they've been asked to display. The computer controlling the panel must constantly light rows of leds in turn. repeatedly switching them to the required colour for a while before moving on to the next. If you do this fast enough you get the appearance of a steady image. This is hard work for the controlling computer, one panel would pretty much tie up an Ardunio completely.  But the good news is that there are a couple of really interesting controller chips that I've been looking for a reason to play with which should be able to drive this, and more, panels. 

First up is the Parallax Propellor, which should give me eight parallel processor cores, one of which should be able to drive a single panel. I've found a device on eBay and I'm looking forward to having a play when it arrives.

The other (and much more interesting) device is brand shiny new. It is from XMOS,  a company in Cambridge, and builds on expertise from ARM and Transputer to deliver a multi-cored device which has as its focus high performance, zero latency and total determinacy. In other words you can be sure that you will get processing power when you need it, and you can be absolutely sure how long something will take to complete. This, coupled with the raw power that is on offer would seem to make it perfect for driving these panels.

You can pick up development kits (with a processor, some inputs and outputs and access to the hardware) for around ten pounds from Farnell in the UK. The board looks quite spiffy and if you are into embedded you really should get one to play with I reckon. The XMOS web site makes it really easy to buy the board if you are in the USA, but the UK links are more tricky to track down. You can find it on the Farnell site here.

The funny thing is that I mentioned my ongoing obsession with bright coloured lights and this was seen to be a good thing by folks present at the open day. My ultimate plan, a coffee table with four of these panels in the surface, was actually lauded as a good idea. We shall see.....