Running Rollup on your PC

/

At the hardware meetup last night Lydia suggested that I get into Rollup. It’s a way of taking an npm project (which would normally run on a server) and packing it up so that it can be downloaded from a web site and run in a local browser. The rollup application runs on your PC and does the packaging. Which, if you have a PC, is a problem because Windows 10 is very unkeen to run programs that have been downloaded from the internets.

You can fix this, and get the program to work, by asking Windows to be a bit more relaxed for a while. The command to do that is:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Enter the command into the terminal in your Visual Studio Code session and you should be good to go.

Moving Music

/

Some things are hard to do just because you don’t know how to do it. Today I wanted to move some music off my PC onto my iPhone. I’ve got Apple Music but some of the stuff I like doesn’t exist there. Or it exists for a while and then vanishes. Anyhoo, it turns out to be very easy, at least for me. I just had to open iTunes on my PC, import the folder containing the music files onto the PC and then wait a while. Eventually they appear on the phone. Now I can play the copy of my “OK Chicago” single. Or I could just watch the YouTube video:

Dice or Die: the eternal question

/
dice1.jpg

One of the questions that popped up when editing “Begin to Code with JavaScript” (pre-order it here) is “die vs dice”. One of the example programs is making a program that delivers a random value in the range 1-6. In other words a die (die being the singular of dice). However, in the text I call it a “dice”. I know this is wrong, but I’m determined to avoid having a section with the title “How to make a computer die”. This makes sense to me, I’m always happy to trade a bit of credibility (people might think I don’t know it’s really called a die) for a lot of understanding.

Begin to Code with JavaScript Final Edit Pass

/
begin to code image.jpg

My efforts with my programming are taking a back seat for a little while as we are starting the final edit pass for “Begin to Code with JavaScript”, which is due to be in the shops later this year (but you can order one now if you like..)This is the point where I get to see all the splendid artwork that’s been added to the text, like the picture above. So far the text is looking good. Just another 11 chapters to go through.

Using the browser for serial data in JavaScript

/

Yesteday I ended the post with a cliff hanger ending. Who know that programming could be so exciting? Who knew? Anyhoo, on with the story. For those who are new to what I'm trying to do, the aim is to be able to plug a brand new embedded devices (ideally an ESP32 or ESP8266 based device) into a computer and then, by using a browser (perhaps Chrome or Edge) visit a web page that can load the device with a program. The web page actually exists so I did have some success. In this set of posts I'm going through how I did it.

https://www.connectedlittleboxes.com/gettingstarted.html

I'm replicating in the browser the behaviour of the esptool program supplied by Espressif. The esptool is "baked in" to the process of deploying a program when you use Arduino or Platform IO to put a program into a device. Esptool is written in Python so I'm essentially translating a big program from Python to JavaScript. I've got some very small parts of my program working. I now know how to reset a device and prepare it to receive a program. I also know how to encode the messages that I want to send into the device.

The first message that our program must send is called "sync" and it just allows the ESP device and the host computer to agree on the rate that they are going to send serial data and that they are both connected and awake.

To do the sync I have to make the browser send a message and then wait for a response from the device. I'm not actually guaranteed to get a response however, because the target device might need several messages before it can work out the speed it should reply at.

To bring things right up to date, yesterday I discovered that when a JavaScript program asks for data from a serial device via the browser the request only returns when some data arrives. So, if my program shouts "Hi" and the device doesn't respond my program will get stuck waiting for a response.

In real life, if you shout "Hi" to someone and they don't respond you'll wait a decent interval and then shout "Hi" again. We need this behaviour in our code. We need a way of "timing out" a read request so that we can send another message again. It turns out that this can be done. But it involves some knowledge of how JavaScript asychronous code works, and what a JavaScript Promise does. So stand buy to learn something.

Asynchronous code in JavaScript

Don't you hate it when a web site gets stuck? When you click a button, nothing happens, you click it again, and then you notice that everything on the site is now locked. The people at JavaScript know your pain. They hate it when the code behind your button click does something that takes a while (or fails) and you are left waiting. So they've invented a way of trying to stop web pages getting stuck. The principle they are using is not unique to JavaScript. Other languages (for example the wonderful C#) also provide what they call "asynchronous" operation.

Asynchronous is a way of saying "I'll get round to doing this, but in my own time". It is how I used to tidy my bedroom when I was a teenager. Mum would ask me to tidy up the room and I'd say "Later..". Sometimes mum would refuse to leave my room until it was tidy. Making mum wait like this is a good example of "synchronous" operation. Mum can't leave until I've finished tidying up.

As I got older and more mature (oh yes) mum started to trust me a bit more. Now she would ask me to tidy up and then she could go back to cooking lunch, cleaning the house, ironing my clothes or any one of a number of active tasks. When I'd finished tidying up I'd call out to mum, she would come back and praise my handiwork. The tidying processs happened asynchronously as far as mum was concerned.

Nice family story, but what does this have to do with JavaScript? Well, some operations that a program needs to do are a bit like me tidying my room. They will take a while to complete and there is a chance that they might never finish. So JavaScript uses a mechanism called a "promise" to manage operations like these. Rather than calling a function and then being stuck waiting for it to complete (which is synchronous operation) we can call a function that returns a promise to do something (which is hopefully more solid than my promises to clean my room). The promise provides a means by which the asynchronous operation can say the digitial equivalent of "I've tidied my bedroom" by accepting two functions. One is called when the promise is fullfilled. The other, more depressing function is called when the promise could not be fulfilled.

if (this.reader == null) {
  this.reader = this.port.readable.getReader();
}

if (this.readPromise == null) {
  this.readPromise = this.reader.read();
}

This is how I start reading from a port. The action looks very like the write action we saw yesterday. For writing you get a writer object and then ask it to write. For reading you get a reader object and then ask it to read. However the read you get from a reader is not data from the port. It is a Promise to fetch some data. The above code sets the value of the variable this.readPromise to this promise. When the serial data arrives the promise will be fulfilled, but in the meantime my code can keep running.

The sharper eyed ones amongst you will have spotted that I only make a new reader and readPromise if I don't already have one. This is important. The first time round, and after a successful read, my program will need to make new reader and readPromise items. But if this code is running on the back of a timeout (no data arrived down the serial port after a time interval) then the program must re-use the existing reader and promise. So now we have the beginnings of a way of dealing with timeouts, but next we actually have to make the timeout itself. We are going to use another Promise to do this.

Making Promises

Next our program performs the following statements to create a timeout Promise:

const timeoutPromise = new Promise((resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
});

So, how does a Promise object work? Well, I've no idea really. I'm fairly sure it doesn't involve pixies, but beyond that I'm drawing a blank here. But I do know how to use a Promise, which is fine by me. I'm going to give the Promise a lump of code to run as a promise. The lump of code will take the form of a function. The promise will run that code for me and manage the outcome. My code might take a while but at some point later the outcome will either be resolved (hurrah) or rejected (boo). The code above is making a timeout, so this particular Promise will resolve when the time out period has expired. The read call that we made above to fetch serial data will resolve when a packet of data has been received from the serial port.

The code being managed by the Promise needs a way of telling the Promise either it worked or it failed. It does this by calling one of the two functions that are passed into it. If you look at the code above you will see that my timeout promise code accepts "resolve" and "reject" as references to functions to call when my promise is resolved or rejected. It turns out that the timeout always works, it can never reject, so my function doesn't use the rejected parameter. But it does use the resolve parameter. It puts it straight into a call of the setTimeout function. So (deep breath) what does setTimeout do?

A timeout is a way that a JavaScript program can ask for an event to be fired at some point in the future. The first parameter to the setTimeout call is a function to be called when the timeout expires. I can pass it a reference to the function to that will be called when the promise is fulfilled (so that it will be called when the promise is fulfilled). The second parameter to setTimeout is the time in milliseconds that JavaScript will wait before calling the timeout function. I want the program to wait half a second before giving up on a serial read request, so I've set this to 500.

The setTimeout function returns an id (actually a number) that the program can use to turn a timeout off before it occurs. We will use this later. So, a recap of what is going to happen when I set up my timout is in order. Here's the code again:

const timeoutPromise = new Promise((resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
});

The program hits the statement and is asked to make a new Promise. The Promise accepts the function that is supplied to it. In this case the function looks like this:

(resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
};

The function controled by a Promise takes two parameters, does something and then calls the resolve function if it ends well and the reject function if it ends badly. In our case the code in the function takes the resolve parameter (the function we want to call when the promise has ended well) and drops it into a call to setTimeout. setTimeout will call the resolve function at some point (500 milliseconds) in the future, ending the Promise.

A Promise race

Right. Now I've got two promises. One of them will be fulfilled when it gets some data from the serial port. The other will be fulfilled in half a second. What I can do now is creates a promise race:

const { value, done } = await Promise.race([this.readPromise, timeoutPromise]);

The JavaScript Promise class provides a method called race. The race method accepts a list of promises. It completes when one of the promises in the list is fulfilled and it returns the result from the fastest promise. And no, I don't really know how it works either.

However, it is very useful in making our timeout behaviour work. If the read promise is fulfilled first (i.e. we get some data) the race will return an object containing a value and a done property (of which more in a minute). If timeoutPromise is fulfilled first the race returns nothing. So I can test to see which promise "won" the race as follows:

if (value === undefined) {
  // The timout has won the race - leave the read hanging and 
  // abandon this loop.
  console.log("Timeout");
  timeout = true;
  break;
}

If there is nothing in the value return (which contains the data returned by read) then the timeout promise has been fulfilled. If this is the case we print a message, set a flag and then break out of the loop which is controlling the read process. The read promise is still "out there doing its thing". My code can handle the timeout (perhaps by re-sending the message to the device) and then come back and run another "promise race" against a new timeout.

If the read promise is fulfilled first the program has got some data to deal with. But before it handles the data it has some tidying up to do. 

// Clear the timeout timer
clearTimeout(this.timeoutID);

The first thing that the program must do is turn off the timeout promise. Remember that timeoutID that we grabbed earlier? We feed that into the clearTimeout function to clear it because we no longer need the timeout.

The next piece of tidying up is where we make ready for the next read operation. It took me a while to get my head around this bit, and I'm still not completely sure, but the code I've written works, so I'm standing by my reading of what is happening....

As far as I can tell a reader is good for one read. Once you've read from it you must give it away and then get a new one for next time. If you re-read the contents you will get back exactly the same content. This can lead to a lot of confusion as your program keeps getting back the same data. In my experience the next thing that happens is that the fans on your computer start to run at full speed, the web page becomes unresponsive and you have to close the browser tab to regain control again. So, once the reader has worked we have to throw it away. The same goes for the promise:

// Clear down the reader that we have just successfully used
// so that the loop will make another one. 
// Note that if we timeout these statements are not performed
// and the existing reader and promise is used next time 
// getSLIPpacket is called

this.reader.releaseLock();
this.reader = null;
this.readPromise = null;

The way the code is sequenced means that if we get a timeout it will just reuse the reader and the promise that have not completed yet. So I can use this cleverness to try to read data from the serial port and get control when nothing arrives.

How I use this, and what form the data takes will have to wait for another day..

Serial conversations in JavaScript

/

Well, I've put it off as long as I could. I've delved into SLIP and DTR lines and allsorts, but today I have to get my head around the way that the JavaScript usb serial library sends and receives data. Way back on day 2 we discovered how a JavaScript application can get a reference to an object that represents a connection to a serial port, now I actually have to use it to move data.

try {
  this.port = await navigator.serial.requestPort();
  await this.port.open({ baudRate: 115200, bufferSize: 10000 });
}
catch (error) {
  return { worked: false, message: `Serial port open failed:${error.message}` };
}

For a refresher, this is what you do to open a port. At the end of this, if the user selects a port properly, you end up with a port object (in a member variable called this.port). We can ask this object to give us reader and writer objects (if it is able to) which we can then use to read and write serial data. The data is always sent and recieved in UInt8Array objects.

Sending serial data

Sending serial data is easy. 

async sendBytes(bytes) {
  let buffer = "";
  let limit = bytes.length<10?bytes.length:10;
  for(let i=0;i<limit;i++){
    buffer = buffer + bytes[i].toString(10)+":"+bytes[i].toString(16)+"  ";
  }
  console.log(`Sending:${buffer}...`);
  const writer = this.port.writable.getWriter();
  await writer.write(bytes);
  writer.releaseLock();
}

This is my sendBytes method. You give it a UInt8Array and it sends it out. It also prints out the first 10 bytes of the array on the console so you can take a quick peek at what is being sent. Note that it uses the writable property of the port which provides a method called getWriter() that delivers a write object that I called writer. I use this to write the bytes and then release the lock on this writer, effectivly discarding it and making it possible for other code to grab a write and write something. Note also that all of this is awaited so that the writing of the data can take place while other parts of JavaScript keep going.

Receiving serial data

Receiving serial data in JavaScript is hard. The main reason for this is the way that on the whole JavaScript takes a very dim view of programs that hang around and wait for stuff. Python doesn't care so much. In pytool, when the program wants to read something from the serial port it just calls the read method on the serial port and waits for a response. Either something will arrive (good) or nothing will arrive (bad). If nothing arrives the serial port will time out at some point in the future and tell the program that nothing was received. The program can note this and give the remote system another prod to see if it is awake or display a message or whatnot. In the case of JavaScript it looks quite easy to read data:

this.reader = this.port.readable.getReader();
const { value, done } = await this.reader.read();

The tricky bit is not the way that reader returns a result. It returns two items:

  • value - the buffer of data you were waiting for
  • done - a flag that is set to true if the reader is finished (perhaps it has been closed or the user has unplugged their device)

The syntax of the call above means that when the reader has finished it will return those two values which my program can then test. If done is true I shut everything down and go home. Otherwise I take a look at what is in value, which will be a UInt8Array full of bytes that have been received from the device. The tricky bit is that the read operation will not return untill it has received something. There is also added trickiness, in that if anything bad happens to the data stream the read operation will throw an exception that I need to handle. Bad things that might happen include remote devices sending badly formed serial data, just the sort of thing that happens when you reset something.

So, my code must handle the fact that read may never return (the remote device might not have detected the right baud rate after reset) and the fact that read may explode with an exception, at which point I need to tidy everything up. Tricky indeed.

I quite like a cliff hanger ending. And I've got other things to do today as well as code. So I'll leave it at that for now.

Given the SLIP

/

Serial Line Internet Protocol goes back a long, long way. Back to the early days of the internet. The original internet connections used packet based transport protocols. What does that mean? Well, it means that data was sent in lumps. A lump had a header, the data and probably a checksum at the end. EtherNet connections (which were shiny new and used to underpin the early internet) shouted packets into the ether (actually a wire) and then waited for responses that made sense.

This was all fine and dandy for connecting together machines in the computer room, but what if you were using a serial connection? On a serial connection the unit of transmission is a single 8 bit byte, not a lump of stuff. If all you can receive is individual bytes, how do you know when you've got a packet's worth?

Enter SLIP. This is a protocol that lets you send blocks of data down a channel which can only send individual bytes. It works by nominating a character called "Frame End" (0xc0) that marks the end of a packet. A receiver can know that it has received a complete packet when it receives a frame end. Of course, the next question is "How do we send the character 0xc0 in a stream of data that could contain any 8 bit value". We do this by nominating another character as "Frame Escape" (0xdb). This character marks the start of an "escape sequence". There are actually only two escape sequences. One sequence (0xdb 0xdc) means "this is actually the character 0xC0" and the other sequence (0xdb 0xdd) means "this is actually the character 0xdb". (I'm giving my numbers in hex to show that I'm a real computer person. For the record 0xc0 is the value 192)

In olden, olden, times you would buy a modem so you could connect your home computer to a distant server and then you would have to write a SLIP driver for your computer that could assemble the internet packets recieved froom the phone line and then pass them into the TCP/IP stack on your home computer. Happy days.

But what has this got to do with the price of fish, or even sending data to an ESP device? Well, it turns out that ESP devices use SLIP as the protocol for their link to the PC. That packet that I carefully assembled yesterday now has to be made into a SLIP packet so that I can send it to the device. This is actually quite a simple piece of code. It has to do three things:

  1. Look for the character 0xC0 in the input data and convert it into the sequence 0xDB 0xDC.
  2. Look for the character 0xDB in the input data and convert it into the sequence 0xDB 0xDD.
  3. Put the character 0xC0 on the end of the data packet.

There's a function in pytool (at line 394) that does this conversion:

def write(self, packet):
    buf = b'\xc0' \
            + (packet.replace(b'\xdb', b'\xdb\xdd').replace(b'\xc0', b'\xdb\xdc')) \
            + b'\xc0'
    self.trace("Write %d bytes: %s", len(buf), HexFormatter(buf))
    self._port.write(buf)

It uses some neat Python features that allow replacment of elements in arrays. You can see how it swaps the escape characters. 

packSlip(message) {
    // find out how many extra spaces we need for escape
    // count the 0xdb and 0xc0 values in the message
    let extra = 0;
    message.map((i) => { if ((i == 0xdb) || (i == 0xC0)) extra++ });

    let inputLength = message.length;

    // add in extra plus space for 0xc0 at start and end
    let totalLength = inputLength + extra + 2;

    let out = new Uint8Array(totalLength);

    let wPos = 0;

    out[wPos++] = 0xc0;

    for (let b of message) {
        switch (b) {
            case 0xdb:
                out[wPos++] = 0xdb;
                out[wPos++] = 0xdd;
                break;

            case 0xc0:
                out[wPos++] = 0xdb;
                out[wPos++] = 0xdc;
                break;

            default:
                out[wPos++] = b;
        }
    }

    out[wPos++] = 0xc0;

    return out;
}

This is my JavaScript slip packing code. You give it an array of byte values and it gives you back a slip encoded string with 0xc0 on the end (and at the beginning too - which clears out any nastiness with old characters lying around in the input buffer of the receiver).

When you start using UInt8Arrays you discover that all the nice dynamic array stuff in JavaScript (which will make an array automatically grow when you put things it it) has disappeared. You have to make the array the exact length you need it to be.

I use the map function to count the number of characters that need to be encoded and use this to work out how long the result array needs to be. Then I spin through the array and do the encoding.

There are almost certainly much more efficient and cunning ways of doing this, but the above code does work. So now I have a nice block of data that I want to send to the ESP device. And then I'll want to see what comes back...

Talking to the ESP

/

Before I try to send any messages, perhaps I'd better try to discover what the messages are. This seems to be a good place to find out what is going on:

https://github.com/espressif/esptool/wiki/Serial-Protocol

Each command is a SLIP packet (ah - that brings back memories) which starts with a 0 which is followed by an opcode, size field, checksum and then a bunch of data. If I dig into the esptool.py file and at line 425 I find the command method, which is used in the program to send a command to the ESP device. After a bit of setup stuff I find this statement:

pkt = struct.pack(b'<BBHI', 0x00, op, len(data), chk) + data

Aha. this looks like the statement that builds a command. What does struct.pack do?

https://docs.python.org/3/library/struct.html

It's all explained here. It is how Python manages the movement of data from variables in the program into bytes of data. The '<BBHI' data-preserve-html-node="true" string tells pack the format of the data it is to pack, in this case it says:

  • < - use "little endian" (in other words when assembling values over more than one byte put the lower bytes first)
  • B - pack a single 8 bit value - a byte
  • H - this is an unsigned "short" int that is stored in 16 bits, that is two bytes of data
  • I - this is an unsigned int that is stored in 32 bits, that is two bytes of data.

From this I can work out that the 0 at the start and op code are single byte values, the length of the data is sent over two bytes and the checksum is sent over four bytes. The statement above that assembles the packet then puts a thing called "data" on the end, which must be the block of data to be sent.

Assembling a command

JavaScript handles 8 bit data using a special type called a Uint8Array. This is my first attempt at packing a command into such an array. 

packCommand(opCode, dataBlock, check) {
    let dataLength = dataBlock.length;
    let totalLength = dataLength + 8;
    const message = new Uint8Array(totalLength);
    let pos = 0;
    message[pos++] = 0;
    message[pos++] = opCode;
    message[pos++] = dataLength & 0xff;
    message[pos++] = (dataLength >> 8) & 0xff;
    message[pos++] = check;
    pos += 3;
    for (let i = 0; i < dataLength; i++) {
        let value = dataBlock[i];
        message[pos++] = value & 0xFF;
    }
    return message;
}

The packCommand method takes in an opcode, a block of data and a checksum value. It returns a Uint8Array with the command at the top and the data. It splits out the parts of each value by masking off the required bits and shifting them into position. It's probably not an optimal way of doing this, but it should work.

Sending a sync

The first command that is sent is a sync message that allows the host computer and the ESP device to talk to each other. Serial data (which is what we are using to exchange messages) is sent at a particular speed, called the baud rate. We set the baud rate when we created our JavaScript serial connection. Now we have to send a message to the ESP device so that it can work out what baud rate we are using and then start receiving at that rate. The communications protocol has a sync message which contains data values which have been chosen to produce a nice sequence of pulses for the ESP device to look at and work out the data rate. I've had a look at the packet and carefully assembled the data that we need. This is my sync command in JavaScript:

async sync() {
    console.log("Syncing the connection");
    const params = {
        op: ESPToolJS.ESP_SYNC,
        dataBlock: [0x07, 0x07, 0x12, 0x20,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55],
        check: 0,
        waitResponse: true,
        timeout: ESPToolJS.DEFAULT_TIMEOUT
    };
    let { val, data } = await this.command(params);
    return data;
}

I've copied all the command numbers into the program as static values, the value of ESpToolJS.ESP_SYNC is 0x08, which is the sync command opcode. The datablock is not delivering any data as such, but it does contain a lot of 0x55 values. If you look at the binary for the value 0x55 you find that it is 0101010101, which means that the serial data signal will bounce up and down in a square wave which is perfect for the ESP to use to work out the serial data speed.

This sync configuration means that the first sync command from the computer probably won't get a response, as the ESP device will use this to work out what speed it should listen. The pytool code sends multiple sync commands, so I'll have to as well.

Serial Reset

/

Task of the day is get a JavaScript program running in a browser to send messages to a serial port. And maybe even reset an ESP device using the hardware handshake lines. Then I can start having a conversation with a device. This website tells me how to make a serial connection from a browser:

https://web.dev/serial/

I’m going to make a little SerialManager class to manage the connection to the serial port.

Connecting to a serial port

Let’s have a look at the function that makes the connection to the device:

async connectToSerialPort() {
  // Prompt user to select any serial port.

  if (!"serial" in navigator) {
    this.port = null;
    return { worked: false, message: "This browser doesn't support serial connection. Try Edge or Chrome." };
  }

  try {
    this.port = await navigator.serial.requestPort();
    await this.port.open({ baudRate: 115200, bufferSize: 10000 });
  }
  catch (error) {
    return { worked: false, message: `Serial port open failed:${error.message}` };
  }

  return { worked: true, message: "Connected OK" };
}

When the function runs it pops up a dialog asking the user to select a serial device. Then it creates a connection. Note that I'm using a funky error return mechanism where the function returns whether or not it worked along with a message. I can use this to get a serial connection and it seems to work.

Resetting the ESP device

Next we need to reset the ESP device into upload mode. This seems to involve some fancy footwork with the hardware handshake lines. You can find the code in the esptoo.py file at line 564:

self._setDTR(False)  # IO0=HIGH
self._setRTS(True)   # EN=LOW, chip in reset
time.sleep(0.1)
if esp32r0_delay:
    # Some chips are more likely to trigger the esp32r0
    # watchdog reset silicon bug if they're held with EN=LOW
    # for a longer period
    time.sleep(1.2)
self._setDTR(True)   # IO0=LOW
self._setRTS(False)  # EN=HIGH, chip out of reset
if esp32r0_delay:
    # Sleep longer after reset.
    # This workaround only works on revision 0 ESP32 chips,
    # it exploits a silicon bug spurious watchdog reset.
    time.sleep(0.4)  # allow watchdog reset to occur
time.sleep(0.05)
self._setDTR(False)  # IO0=HIGH, done

It uses the DTR and RTS lines (which are normally used for handshaking the serial connection) to reset the chip. Not all ESP devices have these connected, but the Wemos devices that I want to specifically target with my program do. You can kind of see what they do:

  1. drop DTR
  2. raise RTS
  3. wait a while
  4. raise DTR
  5. drop RTS
  6. drop DTR

This combination of changes means "please start the chip in bootloader mode". Here's my JavaScript interpretation:

async resetIntoBootloader() {
  console.log("Resetting into the Bootloader");
  await this.port.setSignals({ dataTerminalReady: false });
  await this.port.setSignals({ requestToSend: true });
  await this.delay(100);
  await this.port.setSignals({ dataTerminalReady: true });
  await this.port.setSignals({ requestToSend: false });
  await this.delay(50);
  await this.port.setSignals({ dataTerminalReady: false });
  console.log("Reset into bootloader");
}

This does exactly the same sequence. It uses a little function called delay that I've made that will pause the program for a while.

async delay(timeInMs) {
  return new Promise(async (kept, broken) => {
    setTimeout(async () => {
      return kept("tick");
    }, timeInMs);
  });
}

Note that the JavaScript does a lot of awaiting. I haven't really got time to go into this in detail, suffice it to say that the allows my code to stop for a while without the webpage that it is part of getting stuck. Anyhoo, it seems to work, in that the ESP device stops when I hit it with the reset sequence. Tomorrow I'm going to try sending some commands.

Playing with esptool

/

So, where to start? The pytool program is a large chunk of Python that I know nothing about. I know what it does: it takes files off my PC and squirts them into ESP32 and ESP8266 devices. However, I’ve no idea how it works.

Fetching esptool

The good news is that the program source is a free download from here:

https://github.com/espressif/esptool

I used the GitHub desktop tool to clone this and then opened it with Visual Studio code. The first part of my plan is to get it running and then use the Python debugging tools to find out what it does.

Sending a command

The first thing I have to do is get pytool to do something. I settled for just reading the MAC address from a device. I’m using version 3.2 (which is interesting because it came out after the date of this blog post - I’ll leave you to figure out how that works). With this version you can “hard-wire” commands into the program so that it performs them when it starts. You have to find the main function in the Python program and add some arguments. In the version I’m using I found the main function at line 3789 in the esptool.py program file:

def main(argv=[ 'read_mac'], esp=None):

I've added the argument “read_mac”. When I run the program I now get this:

esptool.py v3.2-dev
Found 2 serial ports
Serial port COM12
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: ec:fa:bc:28:56:27
Uploading stub...
Running stub...
Stub running...
MAC: ec:fa:bc:28:56:27
Hard resetting via RTS pin...

The worrying part of this conversation is the message “Uploading stub”, but we’ll get to that later. For now, lets figure out how we can get a JavaScript program to reset a remote device and talk to it over the serial port.

JavaScript Pytool

/
espytool.jpg

I’ve had a great idea. At least I think I have. I’m going to try to make a JavaScript version of the esptool program that is used to download firmware into the ESP32 and ESP8266. If you’ve ever built and deployed a C program onto these devices you’ve run this program. It is how your program actually gets onto the device. It works really well, but to make it work you have to download it and then figure out how to use it.

I want to make something that you can use from your browser. My dream is to get to a position where you can unwrap a brand new device, plug it into your PC and then browse a website that deploys binary software to the device, configures it and gets it going. I want to make this how you get a Connected Little Box running. It’s possible because of the Web Serial Interface which is now part of the Chrome and Edge browsers. This lets you connect your browser directly to your devices.

On one hand this should be quite easy, in that I can download the Python source of esptool from here and step through the program and see what it does. On the other hand I’ll have to convert the Python into JavaScript and also figure out how the ESP download protocol works. There’s a description here, which is useful.

Whatever happens I’ll learn a ton of stuff and have lots to put into the blog as I go. Which is probably the real of the effort. Wish me luck.

Beverley Westwood

/
westwood.jpg

This picture wasn’t taken with my Floppy Disk camera. I used the phone for this one. It illustrates quite well how we’ve come on in the world of photography. The picture is brighter, sharper and technically much better. But when I look at it closely I get the impression that there’s been some image trickery going on. the brightness of the sky seems go up and down a bit around the trees and the horizon. It brings home to me that the primary job of a phone camera is to make pictures that look good on the phone. But I think that sometimes the software tries a bit too hard.

Hardware Debugging with the ESP32

/
In circuit debugging.jpg

When your programs fail it’s really useful to be able to look inside them and find out what they are doing. If you’re writing code for your desktop or laptop you can use a debugger to step through the code and see what is going on. But that’s not as easy when you’re using an embedded device. The program is not running on the machine that built it, and this means that you end up putting print statements in your code to find out what is going on.

If you’re using an ESP32 device you can always buy one of these. It connects to a device and allows you to do lots of lovely debugging. I’ve been playing with it and writing an article for Hackspace magazine all about embedded debugging. It’s a really nice piece of kit.