Posts tagged as lewesdon

It had to happen at some point: the first big lapse in weeknotes. Everything’s been very busy – albeit largely focused on the same thing.

So what’s happened in those weeks?

Primarily: Swarmize/Lewsdon. The past four weeks has seen lots of progress here. After the big demo in week 95, we had another demo in week 96 where we shared the project with editors, journalists, and technical staff – many of whom had seen it at its earliest stages. This led to a variety of features requests and a lot of useful feedback, which we were able to feed into the ongoing work.

The main focus of my development in that time was permissions management – dreary as it may sound, the likelihood that one user and one alone will only work on a Swarm was low. So giving others permissions on them was going to be important – and that meant implementing a permission-granting system.

That also meant confirming that the entire application tested permissions appropriately throughout, and I spent several days knuckling down and writing some controller tests. They’re all a bit verbose, and I’m not sure it’s the best way of achieving this – but it means I have programmatic proof of who can do what, which is important when there’s a boundary between public and private content on the site.

We also spent some time refactoring the configuration of components to be shared between the Rails and various Scala apps. We’ve cut down on duplication a lot as a result.

And, based primarily on editorial feedback, I implemented various fixes to make it easier to edit swarms for minor things – typos and the like – after they’ve launched, as well as offering some more complex edit interactions for developers who want more control.

We’re at the point where we hope to test much of the toolchain in anger in the coming weeks, and I’m really excited about having live data in the system.

It wasn’t just Swarmize in this period, though. Around the August bank holiday I went up to the Peak District for Laptops and Looms, a very informal gathering in Cromford. It was a good few days for the spirit: clear air, good discussions in the morning, good trips in the afternoon, various thoughts provoked. I chatted a bit about making things to make noise with, and whilst it was really just voicing some thoughts on an interest, there were a few notes it hit that I’d like to return to in the coming months – perhaps in another period of quiet. Lovely company, and a good way to revitalise the head. (I also managed to get up Kinder Scout on the Sunday, which was excellent).

Finally, I spent two days in Week 99 working with Bohdan Piasecki and Ben Pacey, developing a digital brief for their theatre/digital work Palimpsest City. Two days of discussion, lots of sketching on playing cards, and whittling down some core details from a big picture. I’ll be writing that up for them in week 100, but it was a delightful two days: thoughtful, intense, and a change of pace and topic from the usual.

In week 100, I’m in Bristol for Making the City Playable, and cranking hard on Swarmize. And, I hope, returning to more regular weeknotes. I always knew there was going to be a blip – and now I know what it looked like.

Another week cranking hard on Lewesdon.

Graham, the other developer on the project, is on a well-earned holiday, so I’ve had two weeks to push the website that most people are going to utilise and experience quite far forward. It’s very much a working wireframe, extending Bootstrap just enough to do what we want, and to understand the pathways through the site. I’m also off the project during week 95, so I was aiming to leave it in a state where he’d be able to pick things up on his end of things – the Scala tools that are going to do some of the crunchier, high performance work.

This week, that meant adding things like authentication – so that Guardian users can log in with their own accounts – and making sure the alpha UI is coherent and functional. I also took some time to write quite a lot of tests, getting the ball rolling for me writing more in future, and also to cover a complex set of functionality: setting the “open” and “close” timestamps for “swarms”, our large-scale surveys. A useful exercise, and it immediately lifted my confidence in my code.

By the end of the week, much of the deck we’d been using to explain the site had been replaced by functioning code: just in time for a demo with the Editor. He was enthusiastic and asked useful, pertinent questions, which we’ve added to the list of editorial feedback. In a couple of weeks, we’ll be showing the work so far to everyone who’s offered us tips.

For the rest of the week, I was prepping for Week 95’s workshop: one large talk and one small to write, along with making sure we had a working timetable. By Friday, I had a decent schedule spreadsheeted up and shared with my collaborators, and a decent draft of my talk about digital media. Come Sunday, I’d be on a train to Newcastle.

Week 93 was jam-packed. The majority of it was spent on Swarmize/Lewesdon, hammering out the site’s basic UI. By the end of the week, a lot of the core functionality of the site existed – search was up and working, I’d solved a lot of workflow issues and create an end-to-end of the creation workflow, and integrated all the code from the earlier Sinatra-based spike. Even at this stage, I’ve been understanding the workflows and IA of the project better primarily through the act of making: chipping away at the code to reveal what the product wanted to be all along. Sometimes, that means going backwards to go forwards, but the breakthroughs came in the end, and I was pleased with that.

On Monday, I spent the day with David Varela, planning for the 3-day workshop we’re running in August for CreateInnovate. We came up with a flexible but detailed timetable, a variety of ideas for what we wanted to fit into the workshop, and a coherent structure that ought to help our participants develop their ideas. It was a fun day, and I’m looking forward to delivering the workshop with David.

And on Friday, I spent the day mentoring on the ODI‘s Open Data In Practice course. It was one of the best cohorts for the course yet – the participants all had fascinating backgrounds, and by the end, universal enthusiasm to take what they’d learned back to their day-jobs.

A really strong week on Lewesdon/Swarmize, moving closer to a final UI.

That meant a couple of days of staring at markup, Javascript, and the MDN pages for HTML5 drag/drop. I’m building a UI to help editors design a form, dragging fields into it. Drag and drop feels like a sensible paradigm for this, and because I’m building an internal alpha, I’m going to stick to the HTML5 APIs.

We’re building an early alpha, but there’s still a minimum standard of usability to meet, so I spent quite a while working on reducing flickering elements – a bug that emerges because of how drag events propagate through the browser, which makes entire sense if you’re a DOM model, but not much sense to how humans perceive visual elements. Still, some careful work removed almost all the issues here, with a few to solve next week.

I also investigated using Facebook’s React JS to handle the events from the UI and template the form. I was quietly impressed with React: I like that it only really cares about templates and front-end, rather than insisting you build a front-end MVC structure (as Angular or Backbone do).

I stared a lot, but in the end couldn’t see how to apply React to my particular interactions, so I sat down with jQuery, Underscore, and some Underscore templates, and wrote some plain old event-handling Javascript. This turned out to be enough – and quick enough – to get to a satisfying user experience.

I finished up the week building the back-end to store the specification for the form, and then the straightforward server-side templates to generate the form that can be embedded in a web page. I also added a splash of Verify to add some client-side validation. There’ll still be server-side validation, of course, but it feels sensible to maximise the chances of a user submitting useful data to us.

And, finally, I re-used the code from last week to deploy the whole shebang to Elastic Beanstalk and RDS in the final hours of Friday’s working day. I don’t normally deploy on Fridays, but there was no harm done if I didn’t get it done that day, and if I did, it’d be a satisfying conclusion to a busy week.

By the end of the week, UI I’d sketched up as aspirational a few weeks earlier was coming together in its earliest form, and some quick demos at the Guardian got really strong feedback.

So that was a good week on Abberley, and sets up next week nicely.

The majority of this week focused on Swarmize/Lewesdon: writing automatic deployment scripts and spelunking code.

I did spend a morning this week with PAN exploring some work they’ve got on the horizon – some discussion of how to approach development, as well as a good table discussion on the design of the project. Always pleasant to work with one’s studiomates, and I’m looking forward to seeing where the project we talked about goes.

Mainly, though it was Abberley/Swarmize this week. Lots of movement here. A lot of that was moving forward a stub of a web application that’s going to be the primary user front-end to it.

I’m not building that application yet, though. I’m building a tiny Sinatra app that talks directly to our ElasticSearch instance to get a feel for the materials: what queries I’m going to need, how to abstract away from a hard-coded single UI, how to get the whole stack up and running.

I took my hard-coded demo and abstracted just enough to make it easy to build a second demonstration on this app. I think that’s the final goal of this code: beyond that, I may as well work on the real thing. I extracted a lot of classes from the big tangle of code, to see where the boundaries between persisted objects for the front-end app, and interfaces to ElasticSearch (and other APIs) would lie.

I also worked on writing deployment code for the app. I’m deploying onto Amazon Elastic Beanstalk, because the whole stack is running on Amazon instances, and I want to play ball with the other developers – and stay inside their security group.

Rather than using the magic of auto-deployment out of git, I’ve been writing deployment code by hand. In part, to understand what’s actually happening – but also because I’m deploying out of a subdirectory, and Amazon’s “magic” code doesn’t play ball with subtrees.

My first pass at this code was a bash script that bundled up a zipfile of the app, pushed it to S3, created a version of the app with that package, and then applied that version to the live environment. I then rewrote it entirely, using the Ruby Fog library, and abstracting out lots of hardcoded variables, the goal being to make it more configurable and adaptable. It also would save time: Fog made it easy to interrogate the existing environments, so the code wouldn’t re-upload a version if it was already on S3, or make a deployment if one wasn’t necessary (which would throw an error, normally).

This took some time, and I began to doubt the utility of it. My doubts went away on Friday, when, with the aid of a few new configuration variables, I pushed the whole lot straight onto the Swarmize AWS account in no time at all. Time spent up-front to save time throughout. And: as a result of that work, I now understand much more about EB as a stack, and how I’ll go on to use it.

As part of that work, I submitted a patch back to Fog – the tiniest, simplest patch – but the pull request was accepted. It’s nice when open source works like everybody tells you it does.

By the end of the week, the end-to-end demo was out of the “rough and ready” version and feeling a lot more polished. I wrapped up the week by researching some Javascript techniques and libraries for Abberley’s front-end, which I’m going to break ground on next week: a Rails app, with its own persistence layer, that’s slowly going to replicate the crude functionality I already have. But to begin with, I’m going to be working out how to make the UI I’ve designed work within it, which I’m rather looking forward to. Week 92 will be a meaty week of HTML and Javascript.

Week 90 was busy, but with not a vast amount to report.

Lewesdon pushed forward: getting a fairly good 1.0 set of wireframes; building out a little more of the architecture; really understanding how the thing is going to come together. We’re heading towards a very early end-to-end demo, and I’m hoping that’ll come together in Week 91. This week, though, was spent continuing to chip away: drawing, and coming to understand more about Amazon Elastic Beanstalk.

I spent Thursday and Friday at the BBC Connected Studio for their Natural History Unit, as part of a team put together by Storythings. It was an intensive couple of days, but I think we were all pleased with the pitch we put together, and it was a delight to work with Matt, Darren, Silvia and Stephen. We’ll hear if our pitch was successful later in the month.

A quiet week carving away at Lewesdon: working out what it is and what it needs. After last week’s long explanation, there’s not so much to say this week: I’ve just been head down, trying to work out what the product is, moving it forward by writing and drawing.

I’ve described this process as being like pitching myself: telling myself a story, and then listening to see if it makes sense. When it doesn’t, I reframe it. There have been a few useful breakthroughs in my thinking, which usually generate as many questions as answers, but I think that’s good right now. The team are slowly feeling more confident in the ideas I’m putting forward to them, and we’ve begun to ask a few people who are likely to be impacted by the product for initial feedback.

The project I call Lewesdon is, incidentally, Swarmize – a data capture and aggregation platform at the largest scale. The Knight Foundation have now announced the grants, so it’s only reasonable to mention its real name here.

It’s moving forward, and I hope, swiftly enough – but my time leaps between observing, making, observing, making, and it’s tough work to balance it all and not lose momentum.

On Wednesday, a brief meeting set up Blackdown, a three-day workshop for CreateInnovate that I’ll be running alongside David Varela. I mentored briefly on this project last year, and am looking forward to three days with some filmmakers to explore what online projects could do for them. I’m also really looking forward to finally working with David.

But mainly, pushing Lewesdon forward, digging with my Wacom stylus, and pushing onwards.

Week 88, and kick-off on Lewesdon.

Lewesdon is entirely greenfield: we started breaking ground on it on Tuesday. There’s a definite broad overview of what we’re doing, and we made some decent headway on beginning to understand how the architecture of the project was going to have to be shaped.

That still doesn’t answer what the product is, though, and there are a lot of specifics to pin down there: how to make it communicable, and from that broad overview make it usable. The shape of the product is such that the end-user largely shouldn’t care about the underlying architecture – which is good, because the architecture doesn’t necessarily resemble the intent.

On the first day, Graham – the other developer and architect on this – and I drew up some broad architecture diagrams, and cut some very basic code: an outward-facing app and UI talking to a back-end service, written in our languages of choice (Ruby and Scala, respectively). The demo was simple, but it was a useful collaboration exercise, and also useful to prod some assumptions.

But I then began stepping back from code: how does all the configuration and management of this manifest to end users? And that’s been a hard question to answer. The only way to make forward movement is to make some assumptions, and start drawing at a reasonable level of detail. As I sketch, new questions arise that need answering, or discrepancies creep in that I have to assess. And, eventually, I share the sketch, we pick it apart, and I start again. It’s slow moving, but it’s the only way I know to think through this: to think with a pen or stylus, and then see how hot or cold I am.

Friday saw a couple of breakthroughs here, late in the day, after a day of sketching that led to some chats with Matt M, and I’m hoping they’ll help me start week 89 at a bit of a run. We’ll get there, but right now, I’m just going to keep chipping away.

Otherwise, I spent Week 88 exploring some feasibility for a quick sketch, and confirming a few other small pieces of work – which required me to plan out my calendar for the summer in some detail. It turns out that I now know that in quite some detail, and I am pretty much full up until November – primarily Lewesdon at the Guardian, and then smaller consultancy and workshops around this.

From a business-and-financial perspective, that’s a relief; from a work perspective, it’s going to be busy and full of exciting challenges; but it also means that I’m going to be quieter for a bit. Less to show, because I’m head down working on things. The good kind of quiet.

Now, the task is to do the work, giving it space to breathe and flow.

Week 87 brought some much-needed focus to my calendar for the next few months.

The big news was the confirmation of Lewesdon. This is another project with the Guardian. I’ll be able to talk more about that shortly, but it’s going to be an interesting few months wrangling data and building tools that talk to one another. I’m looking forward to working with the rest of the team, if only for the challenge!

I also confirmed a few other much smaller projects – workshops and the like. They fit neatly around Lewesdon, and will make for a nice change of pace. I also confirmed a speaking engagement towards the end of the year.

And I kicked off a new back-burner project, currently for myself, codenamed Abberley. It’s a very tiny poke at an idea for a product, and the main thing that’s interesting is not the functionality but a few design patterns. I’ve shown a tiny draft of it to people and had some encouragement, so I’m going to keep chipping away in a few meetings.

There was also a final burst of work on Housedon.

Busy, and a bit scattered, then, but the result of the past couple of weeks since returning from SF is that the next few months are much clearer. That feels like a good result, and I’m looking forward to the first few days of sketching a clearer picture of Lewesdon next week.