DevOpsDays Boston 2021: We're getting the hang of this!

Monday, November 1, 2021

I’ve been helping organize DevOpsDays Boston since 2017. In that time, our organizing community has changed considerably, yet been resilient to the many faces of change - from organizer turnover, to vendor relationships changing, to a worldwide pandemic that stopped many similar events in their tracks. Change, however, is not necessarily something that only goes in the forward direction. In this blog post, I’ll explain some of the ways, as well as some of the reasons, we progressed, backtracked, or kept the course for our 2021 event.

To set the stage (pun unabashedly intended), let me quickly describe some of the changes that we made in 2020:

The event became fully virtual. We used the Hopin platform hold the event, rather than hold an in-person event at the BCA Cyclorama and the Calderwood Pavilion.
Our traditionally two-day event became a single-day event.
Our traditionally two-track event became a single-track event.
Some of our (my?) favorite vendors, like Atomic Coffee Roasters, Uno Due Go, and perks like our free hoodies, all had to be put on the shelf - it just wasn’t possible to make them work in a fully virtual space.
We transitioned the event to a donation model, where attendees could receive a free ticket.
We dropped the price of sponsorships to reflect the lack of physical interactions.

We fulfilled last year’s prime goal - to put on a great event that people could learn from and be an interactive part of. That’s not to say that the event succeeded on all fronts, though:

Financially, we were in a great place to run another virtual event in 2021. However, if we ran the same virtual event in 2021, we’d have been in a precarious position if we tried to hold a physical or hybrid event in 2022.
Many of our sponsors, even with the reduced cost of sponsorship, did not find that they were able to justify having attended the event. That’s a stark contrast to years prior, where our sponsors absolutely love the amount of interaction they got.
We upended our choice of vendors on short notice, and found ourselves results-oriented rather than goal-oriented.
We did little to engage outside of the event. Our Twitter feed stayed mostly silent; we held no community outreach events, and we depended on sponsors reaching out to us to sponsor.

Clearly, we had some room for improvement.

What changed in 2020 that we changed again in 2021?

The video platform

In 2020, we used the Hopin platform to centrally manage all of our attendee interactions, from registration, to the main stage, to breakout sessions. While this has the advantage of keeping everyone in a single, easy-to-use spot, we had two issues with it:

Hopin eclipsed all of our other expenses for the year in terms of cost. That wouldn’t be an issue on its own - in the virtual space, the platform is the single most important part of the event. However, Hopin’s pricing model was largely incompatible with our event: it is costed out as a yearly subscription, which was incredibly difficult to justify for a single yearly event.
The one-size-fits-all approach to tooling worked well early in the pandemic, but as the space became competitive, the ability to mix and match tools proved incredibly useful. Best-in-class ticket management could be paired with best-in-class attendee communications and streaming services.

For the 2021 event, we successfully brought together several tools: Tito for ticket sales, Vito for streaming the main stage, and Discord for community and sponsor engagement. Even without factoring in any cost differences, the added ability to build community with Discord, the ability to aggregate the day’s events and recordings in Vito after the event has finished, all contribute to more meaningful engagement with our attendees.

Our CFP tool

In 2020, we switched from our tried-and-true CFP submission tool Papercall to a more polished newcomer, Sessionize. While the overall UX of Sessionize is considerably slicker than Papercall’s, we had trouble trusting the results of our blinded review process on Sessionize: it didn’t seem purely deterministic, and we had trouble getting a small experiment run on our end to match our expected results. Ultimately, we switched back to Papercall this year. If it ain’t broke… improve it, but keep your finger on the revert button.

As mentioned before, we underinvested in sponsor experience in 2020 - more so than even the price cut of a platinum sponsorship from $10,000 to $2,000 justified. For 2021, we completely eliminated the gold sponsorship tier, and instead focused on making the best possible experience for a small, finite set of platinum sponsors that we possibly could. Sponsors got five minutes on stage, and a breakout session of their choosing that ran parallel with the talk track- all designed to get sponsors more involved with our audience.

Logistically, we could handle eight platinum sponsors - any more, and we’d have clashed too much with stage time. However, since other sponsors still wanted to have a presence at the event, we instead came up with the idea of community sponsors: rather than sponsoring us directly, we offered a booth any company willing to donate $2000 to our community partners. Much like the ability to donate and even receive a hoodie, this was just another way we earmarked funds for donations to our community.

The conference format

While trying to strike a balance between speaker time on the stage and audience participation, we tried something distinctly nontraditional in 2020: 20 minutes of talk time on stage, followed by 20 minutes of Q&A with the audience, where we (the moderators) injected some pointed questions that allowed our speakers to touch on points that would normally have been in their talk. At the same time, we didn’t feel comfortable in our opportunity to engage an audience for two days, so we dropped the second day from our schedule.

Unfortunately, this caused many presentations to either run over, feel rushed, or worse: both. This year, having at least one year of experience running a virtual conference under our belts, we opted for a more traditional format: two half days, with a 30/10 minute split between talks and Q&A. We also toyed with the idea of putting the two half days in the middle of the day, with a late start and early end, but decided against it: we postulated that some attendees wanted to be able to dedicate half of the day to resuming their normal activities, and that putting the event in the middle of the day would just ruin both halves of the day.

What did we add for 2021?

We added hoodies! As part of our donation-based model for attending, we offered a limited-run hoodie to anyone who donated $100 or more to attend.

Excitingly, this started out as a conversation in the #general channel in the Boston DevOps Slack: some of our community members enthusiastically called out that they’d be interested in paying extra for a hoodie, shipped to them. And frankly, it doesn’t get any better than your customers telling you what they want.

In years past, we used CustomInk to mass-print 700+ hoodies. For reference, this is what you’re delivered when you order 700+ hoodies:

A wall of boxes

There are boxes to store prior to the event, because the venue doesn’t allow you to pre-ship materials. There are boxes to transport to the event - which, if you live in or near Boston’s South End, you may know is rather difficult to drive a delivery van through. And finally, there are boxes to transport from the event, if there are hoodies left over.

Translating hoodies to the virtual space was an interesting challenge. We started out with a solution that consisted mostly of known unknowns: things we knew we’d have to solve for, like where to store the boxes, or how to get them to the post office. But there were also known knowns, like exact shipping costs for anywhere in the United States, thanks to Priority Mail Flat Rate shipping (aside: I consider these the most criminally underappreciated feature of the US Postal Service).

We decided to answer for some of these known unknowns by changing our vendor, such that the vendor would be able to handle logistical challenges like transporting boxes and storing the hoodies. However, we accidentally traded our known unknowns for unknown unknowns: things which we are neither aware of, nor understand. The organizing community around DevOpsDays has, through significant trial and error, decomposed much of running the event into a science, but we no longer benefit from those learnings and that muscle memory when we change the formula.

Some example questions we didn’t think we’d have to answer:

What type of hoodies would we get? The vendor that our vendor contracted with didn’t stock our usual hoodies.
What do lead times look like when your vendors, and your vendors’ vendors, need to sync up?
What would the vendor pay for? We didn’t realize that, for instance, the padded envelopes that the hoodies would travel in, were not covered by the vendor.
What shipping tier would be used? We didn’t clarify ahead of time, and our hoodies ended up being sent out overnight, at no small cost to us. (Engineers and product managers - take heed!)

In hindsight, the hoodie offer was a success to attendees, and from the results-oriented perspective, we did well. However, from the goal-oriented perspective of utilizing funds as donations back to our community partners, there was significant room for improvement. The real question is whether we’d make the same choice again, and I’m fairly confident that even with the unknown unknowns having become known knowns, we’d still rather opt for answering the known unknowns the next time around.

What changed in 2020 that stayed the same in 2021?

The single track

The physical space requires a significant number of humans to lead people to the correct spots - and those correct spots could have been sponsor booths, the lunch table, one of several breakout rooms, and, mostly importantly, one of two talks. This is counter-intuitively a harder problem in the virtual space, but instead of lost humans, we instead expected to see lost attention, as discussions fragmented and momentum diminished.

The single talk track kept attendees focused on one subject, and one subject only. This enabled our attendees to have higher-quality discussions with more people. It also eased a significant cognitive burden off the organizers and moderators, who otherwise would’ve needed to do double duty ensuring that events were somewhat equally attended.

The donation model for tickets

In 2019, our cost to attend was $250 per person. No matter what we did, paying $250 to attend a virtual event probably would’ve been a non-starter for most of our attendees. That said, there are individuals who we underserved even in 2019 - for instance:

Those who couldn’t take time off from work, but couldn’t justify paying $250 just to sit in the audience, then dedicate at best half of their attention to talks.
Those who were used to an employer-sponsored ticket, but didn’t get one. (This was especially true for 2020, when many employers no longer saw an ROI on conferences.)
Those who could invest their full attention, but not $250 - students, juniors, and individuals looking to break into the engineering space.

Moving to a donation model, as my co-organizer Paul Bruce likes to say, feeds two birds with one scone. First, it ensures that anyone who wants to attend doesn’t have to worry about the financials of attending, or find a way to justify their attendance relative to their other tasks. Second, it allows us to explicitly earmark a portion of our revenue towards donations to our community. This is something we do regardless, but in years past, it was decoupled from individual contributions. Now, we have a way to tie it directly to individuals.

Our backroom A/V

I can’t understate how essential Aaron Aldrich is to our event’s success. In 2020, Aaron took the opportunity to physically build much of the streaming setup that we used to coordinate the event, and has been the voice of God in our ears during the event ever since. The services that Aaron brings to the table are immensely valuable to us. Since many of the organizers also operate in the DevOps space in their day jobs, we’re all acutely aware of bus factor in our designs - and we’re all also acutely aware that sometimes, we must take on technical debt when operating them. We made the decision to keep Aaron as our point person for all things A/V, owing largely to this being (hopefully) the last year we’ll hold a completely virtual event.

What might 2022 look like?

I’m only one organizer of our event, and I can’t single-handedly make edicts about it. However, I sincerely hope that the state of the world will be amenable to an in-person or hybrid event in 2022. We’ve learned a lot about keeping our audience engaged and participating, much of which can carry over beyond the virtual space as well. And we’ve set ourselves up financially to support a return to a physical venue, which was one of our most important goals for the year.

Stay tuned for an announcement next year - we usually aim towards a March-April date for an initial announcement, and an opening of our call for papers!