Blogs

2024-07-21 | Put applications and data through the same workflow: the CrowdStrike incident
I like to think I keep on top of Hacker News, and that I’m in the know that the world is on fire before most of my nontechnical peers. So imagine my surprise when my mother sends me this text message at 8:27am: I immediately checked my usual sources - Boston DevOps, the AWS health portal, the Azure health portal, the GCP health portal - none of the major suspects were reporting anything’s wrong.
2024-03-10 | Explaining my .bashrc
I have a number of tools in my .bashrc file that make my day-to-day activities fast. The ability to run commands like aws without needing to remember if I’ve SSOed or connected to the VPN, or to quickly change Kubernetes contexts and namespaces on a per-shell basis, are invaluable to me. Some have pointed out that switching to zsh, with addons like powerlevel10k, would probably benefit me with these use cases.
2022-08-27 | Installing AIX PS/2 v1.3 on a 486
In late August of 2022, after two years of remaining relatively healthy, COVID-19 finally caught up with me. Cut off from friendly gathering, working out, and even food that wasn’t protein bars, a completely normal thought occurred to me: “How about I install AIX on my old 486?” My 486 is, as I assume most extant 486es are today, a franken-486: a hodgepodge of pieces collected over the course of thirty years, ultimately a Ship of Theseus of opportunity (yard sales!
2021-11-01 | DevOpsDays Boston 2021: We're getting the hang of this!
I’ve been helping organize DevOpsDays Boston since 2017. In that time, our organizing community has changed considerably, yet been resilient to the many faces of change - from organizer turnover, to vendor relationships changing, to a worldwide pandemic that stopped many similar events in their tracks. Change, however, is not necessarily something that only goes in the forward direction. In this blog post, I’ll explain some of the ways, as well as some of the reasons, we progressed, backtracked, or kept the course for our 2021 event.
2021-10-16 | Compiling a custom Terraform provider
Several times now, I’ve been presented with provider bugs in Terraform - specifically the AWS provider - that are fixed in PRs that have sat unmerged for months, if not years. While Hashicorp has now codified that community pull requests aren’t being reviewed, I’ve had a dirty trick up my sleeve for about two years now: in some projects, I don’t use the upstream provider anymore. Instead, I run a fork, where I’ve cherry-picked in any fixes I’ve needed.
2020-12-30 | Thoughts on Disconnecting
Like many, I found myself ending 2020 without having taken any time off. After all, travel itself wasn’t exactly advisable, and I’d fruitlessly hoped that by delaying as long as possible, I might better my chances of getting a vacation. Little did I know, delaying until the end of the year actually put me directly in second surge territory – Christmas and New Year’s were straight out. I’ve never taken a staycation, and my last attempt at disconnecting was cut short a mere two days in by my place of work being bought out, so I figured now was high time to give it another try.
2016-07-24 | The Java classpath, shading, and Amazon EMR
The Java classpath allows dependencies to be brought in from various locations at runtime. However, if an environment already defines a classpath, it can override classpath resources specified downstream, as only one version of a particular named class can be loaded by the Java classloader at the same time. This can cause difficulties working in an environment with provided libraries, such as Amazon EMR’s ability to run an arbitrary JAR as a job.
2016-04-24 | Jenkins Job DSL and Jenkins as IaC
Note: this blog post is adapted from a talk give at a meeting of the Boston Jenkins Area Meetup on Thursday, April 21st, 2016. Jenkins and IaC A tool like Jenkins often is the cornerstone of a CI/CD pipeline: it brings together disparate tools like Git, Maven, Packer, and Terraform, providing the “glue” to automate processes and build deployment and testing pipelines. Eventually, the Jenkins server becomes a custom application in its own right, but unfortunately Jenkins does not inherently lend itself to being treated as an application.
2016-03-27 | Designing a Project Lifecycle with Contracts and Constraints
Note: this blog post is adapted from a lightning talk given at a meeting of Boston Devops on Thursday, March 24th, 2016, the slides from which are available. Why contracts and constraints? Put simply, it’s possible to deliver software in a rapid, predictable manner by establishing contracts with developers, and constraining what projects should look like to be considered supportable. This blog post talks primarily about a design pattern, and implementation of that design pattern using several common DevOps tools which each fit into a different stage of a continuously deployable workflow.
2016-03-13 | Setting up Tomcat with a Let's Encrypt certificate
Let’s Encrypt, the free and automated certificate registrar, by default provides certificates in PEM format. This guide details how to set up an output certificate from the Let’s Encrypt client in the Tomcat application server. It assumes that certificates have already been obtained using the client, independent of what method was used (e.g. --webonlyg or –standalone`) to obtain it. The workflow for doing so is: Requesting a certificate from Let’s Encrypt.
2016-01-24 | Setting up LVM to join EC2 instance store volumes as one disk
Some types of Amazon EC2 instances come with multiple instance store volumes - for instance, the c3.xlarge instance type comes with two 40GB SSD volumes. This guide shows how to utilize them as one large 80GB volume via LVM, should this be more desirable for any particular application. Note that instance store is ephemeral. When an instance is stopped or rebooted, the contents of both volumes will be lost, including the partition configuration set up in this document.
2015-12-29 | Unexpected HTTP 504 errors from AWS Elastic Load Balancers (ELBs)
Introduction Elastic Load Balancers (ELBs) make use of multiple concurrent connections to backend applications to improve throughput, and will also attempt to make use of HTTP keepalives to mitigate reconnection overhead. These performance gains introduce multiple points where timeouts must be configured. A misconfigured application / ELB pair can cause an HTTP 504 status code to be returned from an ELB to the client. Additionally, the request may not register in the application’s logs, or on a packet capture against the application.
2015-11-17 | HTTPS upgrades behind an AWS Elastic Load Balancer (ELB)
Introduction Users of Amazon Web Services can elect to terminate SSL using an Elastic Load Balancer (ELB), rather than at the application level. This centeralizes management of certificates, and removes the burden of implementing SSL across multiple technologies in a stack. In general, it is a good idea to force an HTTPS upgrade. This greatly enhances the integrity and privacy of users’ data in flight, and prevents against a downgrade attack.
2015-08-08 | Avoiding inline versioning of code when using an SCM
An SCM tool such as Git, Mercurial, Subversion, or even the venerable CVS makes it easy to keep a working history of an application. An added bonus of using such a tool is the ability to see what changed in a particular commit (e.g. git diff), who changed a particular block of code and when (e.g. git blame), and so on. This, of course, makes the assumption that developers are letting the SCM do the versioning.
2015-07-26 | Double-indexed application logs in Splunk and other log indexers
Introduction A well-behaved application should output reasonably verbose logs. Disk is cheap, and hierarchical datastores make it possible to store large volumes of log data at near-negligible cost. Factor in the benefit of the insight acquired (in which developers can leave a metaphorical trail of breadcrumbs for issues down the road), and the cost-benefit of even the first issue will begin to speak for itself. As highly-available applications mean that logging may be distributed among several distinct data sources (VMs, partitioned logs, or containers, for instance), use of Splunk or Logstash as a log indexer likewise becomes a no-brainer.
2015-07-19 | Deploying from multiple Maven repositories with Chef
Introduction A typical Git branching model will somehow link the contents of the ‘master’ branch with released-to-manufacturing code. For a project deploying builds to a binary repository such as Archiva, Artifactory, or similar, the branch-to-version map for a project using feature branches, Git Flow, or something else entirely might look similar to the following: Branch Name Software Version Binary Repository ID master 1.0.0 internal feature-branch-1 1.0.1-feature-branch-1-SNAPSHOT snapshots feature-branch-2 1.0.1-feature-branch-2-SNAPSHOT snapshots Because active development happens in feature branches, it’s a good idea to make such branches build SNAPSHOT builds, and to have a separate repository for snapshots to which they are deployed.