Posts tagged as cleeve

Lots of things to talk about, plus a story about debugging, because I’ve not talked much about what working looks like.

Not a lot to report from Selworthy: post-deploy, we’re beginning to work out what’s next, corralling future plans, and I think now is a good time to re-examine process. It also looks like I have a few days of sysadminning ahead of me – never my favourite task, but doing just enough to hand it off to somebody else at least means it’s a finite task.

I continued typing away at Cleeve. Only a few big projects left to write up, but there are definitely some second drafts to come.

A couple of phonecalls and meetings mean that there are two new things on the horizon and inches from actualisation. Let’s call them Gummershow (gummers-how, not gummer-show) and Lowick for now. Should have more about these next week.

And over on Longcrag, I continued hacking away at some embedded code and learning a few things on the way. Let’s talk about how that’s going.

My development setup looks like this:

At the top is the Silicon Labs development board. At the bottom is my test harness: rather than having a messy breadboard, I designed and built a simple circuitboard containing my test hardware – a button, a switch, some jack sockets and LEDs, and a pinheader to connect them all up. That can just be put straight into my breadboard as an intermediate between it and the pinouts of the development board. Much easier to work with. (Why isn’t it a ‘shield’? a) It’s cheaper – PCBs are priced by size but also b) I don’t know which pins I want to work with yet. This lets me choose and change my mind).

The code is all running on the Silabs board, and I’m able to debug it over the USB line. Proper watching of expressions and variables here – a huge quality-of-life improvement over printing strings to a serial port.

I had the code working end-to-end last week, and had been compiling it in the ‘debug’ profile. I swapped over to the ‘release’ profile to see what would happen, and the answer is: a bunch of things broke in unexpected ways. After eyeballing my code, I altered my approach to debouncing, but no dice.

I decided to stop using my eyeballs and start using the right tool for the job – so I fired up the oscilloscope. One pin in particular, which ought to light an LED for 50ms, was blinking imperceptibly. I used the scope to time it. In ‘debug’, it was blinking for 50ms; in ‘release’, it was blinking on for 390µs or so. Not what I had in mind.

I continued poking to no avail; I ended up wiring a pin into the oscilloscope and using it to debug all manner of variables. The system clock was running correctly; all the timing code was behaving correctly; evidently the error was somewhere else.

I decided to look at something else – another output that ought to toggle on button presses and would, unreliable. I discovered that it was not staying lit but bouncing, imperceptibly – a tiny square wave of pulses before it settled into a state. And the scope told me those pulses were 390µs wide.

And that was the key that cracked it.

The bounce indicated that the variables was altering, but far too fast – it was being changed in the main loop, when it really ought to have been changed in the system tick. That was a mistake – when I corrected it, things started working correctly.

The real key to understanding the problem, though, was the duration – 390µs – being exactly the same as the burst on the other LED. It turned out that in the ‘debug’ profile, when the board runs slower owing to all the other things it’s doing, the main loop was taking more than 1ms, and so the problem never emerged. In the debug profile, the main loop was taking less than 1ms – probably around 390µs, in fact – and the problem reared its head.

It took what felt like an absurd amount of time to solve this problem – about a day and a half, I reckon, over this week and the previous one. But when I solved it, I felt satisfied: I’d gone from ‘trying everything’ to real diagnosis – and debugging software on an oscilloscope. (It also reminded me about the value for the right tool for the job. Sometimes, I wonder if the scope is a bit of an indulgence – and every time it turns out to be exactly what I need, I am super grateful for it).

Embedded code, eh. I’m hitting a point with this where I think it might be time to think about building a full prototype. Erk.

Three main pieces of work this week.

Over on Cleeve, I continued write-ups of projects. They’re nearly done – although the largest ones are still left – and I need to dig out some imagery and artwork at some point.

I spent a while on the Silicon Labs board, hacking up this Longcrag project. A good chat with a friend convinced me it might be worth prodding it for longer. If it’s a goer, it’d be interesting to see if I could manufacture a physical product properly – a short run, for retail sale. That’d be an interesting way to learn a lot of things, even with a relatively niche product.

I’ve been battling C – learning more about structs, getting better at writing headers, and tearing my hair out about the quality of documentation. The Silicon Labs stuff is pretty well documented, too! It’s just not quite what I’m used to, and it’s been a long while since I’ve been stuck for hours on what feel like the most basic of problems. At the end of the week, I had most of the functionality done, but everything decided to stop working once I moved the code into the release profile. So there’s a bit more to be done there.

Selworthy saw a huge release. I’d been working on a parallel track since late summer, overhauling some of the underlying concepts and conceits – real foundational stuff that would have a knock-on to all the higher-level systems. Some of the change was, for a while, a bit experimental, but once the experiment looked like it’d come to fruition, it was time to cloose the loop. This week, I merged 291 commits into master. The merge was relatively seamless, thanks to regular rebasing, some code review with colleagues, and freezing our master branch shortly before the deployment.

The dev team deployed it early on Thursday morning. It was pretty seamless for a deploy involving small team distributed around the world, a moderate amount of DNS wrangling, and a long-ish data migration. We’d rehearsed the process on a set of rehearsal servers, and that helped iron out the kinks of the data changes. By 9am Thursday, Selworthy was running in its new world. We had almost no glitches post-launch, and users were already commenting enthusiastically on new features that day. A big job, well done, and it frees us up to do less infrastructural work in future.

Some good impromptu chats in the week, too, including a nice coffee with Johanna who was working down the hallway. We compared notes notes on design process and practice – good stuff, and great to step away from the C for a while.

Weeknotes are easier when you only have to write them for a single week. Let’s get back on that.

Selworthy is gearing up for a fairly major deployment, incorporating the work I’ve put in over the summer that will form our new foundations. So I’ve been shepherding small pull requests around that, making sure we have a clean deployment branch, and planning out the deployment process and rehearsals. The little team is now motoring – someone else has taken the front-end by the scruff of the neck and is dragging it into the future, and our pull requests feel like they’re getting real reviews. It’s good.

I continued some prodding on Longcrag/Foxfield work. I’m really trying to finalise the next line of products. Three of them are so nearly there – one is done, one is on the edge of done but might get a last minute redesign… and one I’m umming and ahhing about. It’s the most complex kit I’ve got, and it’s causing me a few last minute issues. All the fun of hardware.

I also continued the ‘bring-up’ of the Silicon Labs devkit I’m working on. That’s going moderately well, now – I’m wrapping my head around the board a little more and have most features working, although, as ever, debouncing buttons continues to be a fun adventure. One last major feature to get into it, and then I’m going to have a think about what I’ve done there.

I did a small amount of work for the Good Night Lamp team – fettling the management tool a little with some feature requests to improve quality of life.

And I kicked off a new piece of work for myself called Cleeve. Cleeve is unexciting but important. You might note that my Projects page is about 2.5 years out of date. That is not deliberate. But it’s time to update it – so I’m working on writing everything up, researching the stuff I can’t remember, and getting it all nicely up-to-date. You’ll know when it’s all live, but in the meantime, that’s a thing to be working on.