DEVOPS WEEKLY ISSUE #579 - 30th January 2022

Lots of good ops posts this week, from linux systems administration to outage retrospective and creating actionable alerts to managing large Kubernetes clusters and more besides.

StackHawk sponsors Devops Weekly

Teams are adding different types of automated application security testing in their CI/CD pipelines. See how you can add different types of automated security testing to your pipeline quickly and ship better quality code.
https://sthwk.com/appsec-to-your-pipeline

News

Is the art of maintaining linux servers a dying art? This post thinks maybe, and makes a case for reclaiming it.
https://www.pietrorea.com/2022/01/28/reclaiming-the-lost-art-of-linux-server-administration

A detailed retrospective of a 73-hour outage. Lots of technical details, and it’s interesting to see this sort of openness from a consumer brand.
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/

A look at redundancy as an approach to building resilient systems, using the James Webb space telescope as a case study.
https://flyingbarron.medium.com/the-james-webb-space-telescope-success-through-redundancy-dfe3f57fb772

An interesting post on using continuous testing to provide feedback to development teams to aim in managing availability.
https://engineering.salesforce.com/managing-availability-in-service-based-deployments-with-continuous-testing-61be968da4a

A post with lots of operational details about running a large Kubernetes installation, with over 4k nodes and 200k pods.
https://medium.com/paypal-tech/scaling-kubernetes-to-over-4k-nodes-and-200k-pods-29988fad6ed

A useful set of tips for creating actionable alerts in your monitoring system.
https://tech.devoted.com/making-alerts-actionable-3a65904c368e

Indexes speed up database queries right? Well, like most things, it’s more complicated than that. An interesting look at real world database performance.
https://engineering.nanit.com/an-index-gone-rogue-2d58dd824b22

An argument for using GraphQL for querying the state of cloud resources as the complexity and scale of the APIs provided by the public clouds continues to increase.
https://hackernoon.com/why-the-aws-azure-and-gcp-clis-need-to-die

Tools

A standalone reverse-proxy to enforce Webauthn authentication. It can be inserted in front of sensitive services or even chained with other proxies (e.g. OAuth, MFA) to enable a layered security model
https://github.com/Quiq/webauthn_proxy

A fuzzer for RESTful APIs, useful for finding security and reliability issues.
https://github.com/microsoft/restler-fuzzer