23 October 2017
Lots of things to talk about, plus a story about debugging, because I’ve not talked much about what working looks like.
Not a lot to report from Selworthy: post-deploy, we’re beginning to work out what’s next, corralling future plans, and I think now is a good time to re-examine process. It also looks like I have a few days of sysadminning ahead of me – never my favourite task, but doing just enough to hand it off to somebody else at least means it’s a finite task.
I continued typing away at Cleeve. Only a few big projects left to write up, but there are definitely some second drafts to come.
A couple of phonecalls and meetings mean that there are two new things on the horizon and inches from actualisation. Let’s call them Gummershow (gummers-how, not gummer-show) and Lowick for now. Should have more about these next week.
And over on Longcrag, I continued hacking away at some embedded code and learning a few things on the way. Let’s talk about how that’s going.
My development setup looks like this:
At the top is the Silicon Labs development board. At the bottom is my test harness: rather than having a messy breadboard, I designed and built a simple circuitboard containing my test hardware – a button, a switch, some jack sockets and LEDs, and a pinheader to connect them all up. That can just be put straight into my breadboard as an intermediate between it and the pinouts of the development board. Much easier to work with. (Why isn’t it a ‘shield’? a) It’s cheaper – PCBs are priced by size but also b) I don’t know which pins I want to work with yet. This lets me choose and change my mind).
The code is all running on the Silabs board, and I’m able to debug it over the USB line. Proper watching of expressions and variables here – a huge quality-of-life improvement over printing strings to a serial port.
I had the code working end-to-end last week, and had been compiling it in the ‘debug’ profile. I swapped over to the ‘release’ profile to see what would happen, and the answer is: a bunch of things broke in unexpected ways. After eyeballing my code, I altered my approach to debouncing, but no dice.
I decided to stop using my eyeballs and start using the right tool for the job – so I fired up the oscilloscope. One pin in particular, which ought to light an LED for 50ms, was blinking imperceptibly. I used the scope to time it. In ‘debug’, it was blinking for 50ms; in ‘release’, it was blinking on for 390µs or so. Not what I had in mind.
I continued poking to no avail; I ended up wiring a pin into the oscilloscope and using it to debug all manner of variables. The system clock was running correctly; all the timing code was behaving correctly; evidently the error was somewhere else.
I decided to look at something else – another output that ought to toggle on button presses and would, unreliable. I discovered that it was not staying lit but bouncing, imperceptibly – a tiny square wave of pulses before it settled into a state. And the scope told me those pulses were 390µs wide.
And that was the key that cracked it.
The bounce indicated that the variables was altering, but far too fast – it was being changed in the main loop, when it really ought to have been changed in the system tick. That was a mistake – when I corrected it, things started working correctly.
The real key to understanding the problem, though, was the duration – 390µs – being exactly the same as the burst on the other LED. It turned out that in the ‘debug’ profile, when the board runs slower owing to all the other things it’s doing, the main loop was taking more than 1ms, and so the problem never emerged. In the debug profile, the main loop was taking less than 1ms – probably around 390µs, in fact – and the problem reared its head.
It took what felt like an absurd amount of time to solve this problem – about a day and a half, I reckon, over this week and the previous one. But when I solved it, I felt satisfied: I’d gone from ‘trying everything’ to real diagnosis – and debugging software on an oscilloscope. (It also reminded me about the value for the right tool for the job. Sometimes, I wonder if the scope is a bit of an indulgence – and every time it turns out to be exactly what I need, I am super grateful for it).
Embedded code, eh. I’m hitting a point with this where I think it might be time to think about building a full prototype. Erk.
16 October 2017
Three main pieces of work this week.
Over on Cleeve, I continued write-ups of projects. They’re nearly done – although the largest ones are still left – and I need to dig out some imagery and artwork at some point.
I spent a while on the Silicon Labs board, hacking up this Longcrag project. A good chat with a friend convinced me it might be worth prodding it for longer. If it’s a goer, it’d be interesting to see if I could manufacture a physical product properly – a short run, for retail sale. That’d be an interesting way to learn a lot of things, even with a relatively niche product.
I’ve been battling C – learning more about
structs, getting better at writing headers, and tearing my hair out about the quality of documentation. The Silicon Labs stuff is pretty well documented, too! It’s just not quite what I’m used to, and it’s been a long while since I’ve been stuck for hours on what feel like the most basic of problems. At the end of the week, I had most of the functionality done, but everything decided to stop working once I moved the code into the
releaseprofile. So there’s a bit more to be done there.
Selworthy saw a huge release. I’d been working on a parallel track since late summer, overhauling some of the underlying concepts and conceits – real foundational stuff that would have a knock-on to all the higher-level systems. Some of the change was, for a while, a bit experimental, but once the experiment looked like it’d come to fruition, it was time to cloose the loop. This week, I merged 291 commits into master. The merge was relatively seamless, thanks to regular rebasing, some code review with colleagues, and freezing our master branch shortly before the deployment.
The dev team deployed it early on Thursday morning. It was pretty seamless for a deploy involving small team distributed around the world, a moderate amount of DNS wrangling, and a long-ish data migration. We’d rehearsed the process on a set of rehearsal servers, and that helped iron out the kinks of the data changes. By 9am Thursday, Selworthy was running in its new world. We had almost no glitches post-launch, and users were already commenting enthusiastically on new features that day. A big job, well done, and it frees us up to do less infrastructural work in future.
Some good impromptu chats in the week, too, including a nice coffee with Johanna who was working down the hallway. We compared notes notes on design process and practice – good stuff, and great to step away from the C for a while.
09 October 2017
Weeknotes are easier when you only have to write them for a single week. Let’s get back on that.
Selworthy is gearing up for a fairly major deployment, incorporating the work I’ve put in over the summer that will form our new foundations. So I’ve been shepherding small pull requests around that, making sure we have a clean deployment branch, and planning out the deployment process and rehearsals. The little team is now motoring – someone else has taken the front-end by the scruff of the neck and is dragging it into the future, and our pull requests feel like they’re getting real reviews. It’s good.
I continued some prodding on Longcrag/Foxfield work. I’m really trying to finalise the next line of products. Three of them are so nearly there – one is done, one is on the edge of done but might get a last minute redesign… and one I’m umming and ahhing about. It’s the most complex kit I’ve got, and it’s causing me a few last minute issues. All the fun of hardware.
I also continued the ‘bring-up’ of the Silicon Labs devkit I’m working on. That’s going moderately well, now – I’m wrapping my head around the board a little more and have most features working, although, as ever, debouncing buttons continues to be a fun adventure. One last major feature to get into it, and then I’m going to have a think about what I’ve done there.
I did a small amount of work for the Good Night Lamp team – fettling the management tool a little with some feature requests to improve quality of life.
And I kicked off a new piece of work for myself called Cleeve. Cleeve is unexciting but important. You might note that my Projects page is about 2.5 years out of date. That is not deliberate. But it’s time to update it – so I’m working on writing everything up, researching the stuff I can’t remember, and getting it all nicely up-to-date. You’ll know when it’s all live, but in the meantime, that’s a thing to be working on.
02 October 2017
Week 246 and the majority of 247 were spent on holiday. Mid-week 247, I got back to work.
I shipped a new batch of Foxfield products to Thonk – 90 kits in total, to restock some that had sold out entirely and bolster the stock of other. That was a couple of days of bagging and wrangling shipping, at the end of Week 247 and early into week 248, but we got there!
I spent a handful of days in week 248 building out a small software utility for Sensible Object. In the end, slightly tighter timewise than I’d have liked, but we got the key features deployed and running in the time we had.
In week 249, I spent a day in Newcastle running an afternoon workshop on the digital landscape for filmmakers, on behalf of Northern Film + Media – half of it an extension of previous works about promoting and extending your projects online, half of it focused on technology as a material for artists to work with.
I spent some R&D time exploring working with a Silicon Labs Happy Gecko development board, porting a prototype based around Arduino to it. Why? Partly to get a feel for what more serious embedded development looks like; partly because the devtools are surprisingly alright (compared to some other offerings); partly because it offers perhaps a more practical route to market than ancient old Atmega chips. Slow progress, but not no progress – and exciting to see things come to life, as always.
And around all that I brought the big branch of Selworthy I’ve been working on into land. That’s involved some challenging code, and some particularly challenging rebases. But by the end of week 249, it was all lined up and ready to go. I’m looking forward to seeing that in the wild. Selworthy has also acquired a new front-end developer, who’s dragging some of my clunky front-end code kicking and screaming into 2017; it’s becoming more complex, also more organised as a result. More importantly, it’s already feeling better to an end user, and I think we’re going to reap some benefits from that.
04 September 2017
A good couple of weeks.
I’m working on a second production run of Foxfield products. I’ve got all the PCBs and panels already – all that remained to do was order up the components for the kits. These all arrived and in week 245, I started packaging up the second run. I’m halfway through after a day or so’s work.
I’ve spent quite a bit of time working on the course material for the circuit board design course I’m running in October. That’s been going well, too – working out what we’ll be doing, testing out my teaching material, making sure we have a good combination of practical exercises and some short talks.
I also confirmed details of a workshop later in the month in Newcastle for film-makers (and primarily artist-filmmakers):
Aimed at artists and filmmakers this session will explore two core topics: using technology as a means of developing artistic and creative practice; and using the internet to develop audiences for your work in the ways most appropriate for it.
Always pleasant to work with Northern Film – I’ve done a few courses with them before – so I’ll be working on bashing this into shape shortly.
I ordered up some Longcrag prototypes to get made whilst I’m away on holiday for the next two weeks – hopefully they’ll be the production prototypes for a few new products to get made before the end of the year.
I’ve started some explorations into Silicon Labs microcontroller hardware – trying to get away from writing Arduino-based C and into something that might be more suitable for production products. The world of firmware programming is a bit hairy, but this is easing me in moderately slowly, and I hope to spend a little more time with it in the coming months.
And finally, over on Selworthy, I had a meaty, big breakthrough: the large feature I’ve been hacking on all summer suddenly came into focus and, frankly, works. It changes some of the core concepts within the system and the net result it was aiming for had really paid off. Really pleased – it’s been a brain-twisting few months, and I now know more about frame-rates than I ever imagined possible.
And with that: I’m off on a proper vacation. Back Week 247; see you then.