For more than 13 years I’ve been sending this newsletter every Sunday. It’s given me a unique viewpoint as devops grew from a small community to an international movement and beyond. The first issue was sent a little after the Hamburg devopsdays event back in 2010. Hamburg was the 4th such event, since then more than 500 devopsdays events have happened around the world.
Coming to a milestone like 700 issues allows for a bit of taking stock. I’ve often been asked how I’ve kept doing the Devops weekly newsletter consistently, every week, for so long. Some of that is just stubbornness in fairness, but it takes time every week. I’ve decided for now to use that time for other things.
For now, I’m not sure whether the newsletter will be back in the future or not. But you never know.
Thanks to everyone who’s ever sent me a blog post or a new tool they’ve worked on. Thanks to the companies that have sponsored the newsletter, or chosen to list jobs. Thanks to the event organisers who have built the community. And finally, thanks to all the readers of the newsletter, especially those who’ve sent me messages over the years to say Thanks.
Thanks for being part of Devops Weekly.
Gareth
]]>[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
How do you build an organisation with a culture of operational excellence? This post discusses establishing a Center of Production Excellence, and talks about patterns for influencing the rest of the organisation.
https://www.honeycomb.io/blog/establishing-center-of-production-excellence-pt1
An introduction to the DORA metrics for measuring SDLC performance, with a focus on the challenges of collecting the data.
https://www.datadoghq.com/blog/dora-metrics-software-delivery/
Building an internal platform comes with a number of challenges, including internal competition, limited adoption, migration management and more. This slide deck introduces the concept of Platform decay to discuss all of these.
https://speakerdeck.com/syntasso/the-number-1-platform-engineering-problem-youve-never-heard-of-platform-decay
A useful primer on SLOs, SLIs and error budgets, with lots of examples and analogies to help with understanding the concepts.
https://medium.com/@lokesh12/dont-get-lost-in-the-metrics-maze-a-practical-guide-to-slos-slis-error-budgets-and-toil-939ecd0181eb
I have an interesting job opportunity on my team at Snyk, for a Product Management Director focused on infrastructure and operations. Based in London, you’ll focus on the stability, security and internal developer experience of our platform that provides application security tooling to some of the world’s largest companies.
https://boards.greenhouse.io/snyk/jobs/7077559002
Amber is a new type-safe and runtime-safe programming language based on ECMA script syntax that is designed to compile to bash.
https://amber-lang.com/
https://github.com/Ph0enixKM/Amber
An open source implementation of TestGen-LLM that utilises generative AI to automate the generation of tests (currently mostly unit tests).
https://github.com/Codium-ai/cover-agent
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
An interesting look at a large production infrastructure, including anycast routing, Kubernetes and AWS Serverless features.
https://vercel.com/blog/behind-the-scenes-of-vercels-infrastructure
A presentation on building an SDK program, automatically generating API clients and documentation from OpenAPI specifications.
https://www.apimatic.io/events/build-a-winning-sdk-program-in-2024-event
Large monorepos can often pose a challenge for traditional CI systems, leading to slow build times. This post looks at using Yarn to only run tests against components that have changed, speeding up feedback.
https://mathieularose.com/ci-cd-pipeline-change-based-testing-yarn-based-monorepo
A run down of what makes a good REST API. Looking at OpenAPI and associated tooling, rate limiting tips, asynchronous APIs and more.
https://apitally.io/blog/what-makes-a-good-rest-api
A tutorial combining Pulumi with Docker Build Cloud, including caching images to AWS ECR and building multi-platform images.
https://www.docker.com/blog/pulumi-and-docker-build-cloud/
Bend is a massively parallel, high-level programming language. Think the expressiveness of Python, but with near-linear speedup on multi-core hardware like GPUs, but with zero explicit parallel annotations.
https://github.com/HigherOrderCO/Bend
Unleash is a feature flagging framework and application with a wide range of clients for different programming languages and frontend frameworks.
https://github.com/Unleash/unleash
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
A look at the current state of API specification languages, in particular looking at TypeSpec.
https://nordicapis.com/tis-but-a-scratch-does-typespec-reignite-the-specification-wars/
A look at using VolumeSnapshots in Kubernetes to speed up pod startup time for certain types of application.
https://medium.com/riskified-technology/optimize-kubernetes-pods-startup-time-using-volumesnapshots-c0a2b7d39a29
A look at the pros and cons of using JSON for structured logs.
https://medium.com/@oakley349/the-promise-and-peril-of-json-logging-ec2d1f47cee7
A nice walkthrough of building a CI/CD pipeline using GitHub Actions, with a focus on meeting SOC 2 compliance rules.
https://mathieularose.com/gitops-cicd-github-actions
Pug is an interactive terminal application for working with Terraform. You can view the output of plans and applies, manage state resources, work on tasks in parallel and more.
https://github.com/leg100/pug
Flow is a tool for managing a local development environment. It’s powered by Nix, but provides a nice high-level interface based around the local directory.
https://github.com/flox/flox
https://flox.dev/
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
The 2024 DORA Survey is open for responses, with questions focused on hot topics like AI adoption, DevEx, and platform engineering
https://cloud.google.com/blog/products/devops-sre/2024-dora-survey-now-open
Very relevant to the evolution of systems administration over the past 15 years, this post looks at the relationship between Jevons paradox and automation.
https://www.timpaul.co.uk/posts/automation-and-the-jevons-paradox/
A post on building robust serverless architectures, describing patterns for dealing with failures and preventing them spreading through an application.
https://community.aws/content/2fdtPfnbvTo5SxTtIC3Emhl0UVG/navigating-through-failures-build-resilient-serverless-systems
A wild story about an interesting implication of S3’s cost model for DOS-like attacks to drive up costs if bucket names are known, and the risk of default configuration to lead to security challenges.
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
MemoryDB is a new service from AWS that provides improved durability, fan-out, leader election, safety during reconfiguration and resharding on top of Redis. This post explains some of the technology and approach in accomplishing that.
https://brooker.co.za/blog/2024/04/25/memorydb.html
An interesting post on the challenges of monitoring ML models, introducing problems like training-serving skew, data and concept drift and data pipeline issues.
https://www.datadoghq.com/blog/ml-model-monitoring-in-production-best-practices/
A look at the rise of generative AI-driven code agents and the impact on the future of software engineering.
https://itrevolution.com/articles/the-dawn-of-code-agents-embracing-the-future-of-software-development/
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
A detailed post on one organisation’s Prometheus setup and an analysis of alerts over time to improve on-call experience.
https://blog.cloudflare.com/alerts-observability
A curated list of public Telegram channels and groups (chats) dedicated to DevOps, SRE, and Platform Engineering.
https://github.com/palark/awesome-devops-telegram
An interview focused on incident management, with a nice example and discussion of implementing processes and tools including chat integration, incident commander roles and more.
https://klaviyo.tech/klaviyo-incident-management-interview-with-laura-stone-28563ac9558b
If you’re using AWS, you’ve probably come across ARNs, Amazon Resource Names. This post explains what they are, and how information is encoded in the name.
https://everythingdevops.dev/what-is-amazon-resource-name-arn/
A set of posts on OpenTelemetry best practices, starting out with naming and automatic instrumentation.
https://www.honeycomb.io/blog/opentelemetry-best-practices-naming
https://www.honeycomb.io/blog/opentelemetry-best-practices-agents-sidecars-collectors
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
A look at one organisation’s approach to building a service that converts text input into SQL queries, in order to improve data analysis efficiency.
https://medium.com/pinterest-engineering/how-we-built-text-to-sql-at-pinterest-30bad30dabff
A post on migrating a large EC2 fleet to use the latest version of the AWS Instance Metadata Service, and enforcing its usage using a variety of tools.
https://slack.engineering/our-journey-migrating-to-aws-imdsv2/
A presentation presenting a model for reasoning about internal platforms, looking at choreography, orchestration and infrastructure composition.
https://speakerdeck.com/danielbryantuk/platformcon-24-platform-orchestrators-the-missing-middle-of-internal-developer-platforms
An interesting retrospective on a specific developer toolchain. While some of the observations are maybe Ruby specific, the general points about unforeseen impact of tooling and the design of software.
https://shopify.engineering/a-packwerk-retrospective
Many modern architectures combine microservices with asynchronous workflows and serverless functions. The following post explores one such evolution.
https://netflixtechblog.com/the-making-of-ves-the-cosmos-microservice-for-netflix-video-encoding-946b9b3cd300
Index Advisor is a PostgreSQL extension for recommending indexes to improve query performance.
https://github.com/supabase/index_advisor
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
A good opinion piece on security not being special, when compared to other disciplines - and the problems caused by security teams assuming it is.
https://kellyshortridge.com/blog/posts/cybersecurity-isnt-special/
If you’ve ever run into a problem that’s come down to time in computer systems then this is a good post for you.
https://brooker.co.za/blog/2023/11/27/about-time.html
A detailed, technical, post on embracing eBPF for monitoring at the network layer and providing better control of a large microservice and infrastructure platform.
https://doordash.engineering/2023/08/15/bpfagent-ebpf-for-monitoring-at-doordash/
A great post with tips for being on-call. Covering why on-call is hard, and what you and your team can do to make it suck less.
https://hart-michael.medium.com/how-to-be-on-call-034e3a202729
There is quite a bit of cross-over between how a central security team needs to interact with a larger development team, and what’s needed for cost-control in self-service platform teams. A good post on this topic.
https://stateofsecurity.com/how-information-security-and-risk-management-teams-can-support-finops/
Alert fatigue quickly becomes a problem as systems grow, and monitoring software does its thing. This next post talks about how to prevent it.
https://www.datadoghq.com/blog/best-practices-to-prevent-alert-fatigue/
An epic post that’s well worth the long read. A look at each of the 14 points from Deming’s System of Profound Knowledge with modern cyber security examples.
https://itrevolution.com/articles/out-of-the-cyber-crisis-deming-in-the-world-of-cybersecurity/
An interesting post on the perils of productivity metrics for software development, in particular considering the impact of generative AI developer tools.
https://isthisit.nz/posts/2024/engineering-productivity-metrics-genai/
A couple of posts on evolving incident management practices, looking at the need to introduce gradual changes, standardising severity levels, the importance of training and more.
https://medium.com/dyninno/dyninnos-incident-management-an-introduction-a4516b910269
https://medium.com/dyninno/streamlining-and-implementing-incident-management-at-dyninno-c8ea06327f3a
A look at Platform Engineering, and introducing a layered model of platform, with the oft-missing platform orchestration layer binding together the application and infrastructure.
https://www.syntasso.io/post/platform-engineering-orchestrating-applications-platforms-and-infrastructure
A look at how one team used gamedays as a tool to test and improve performance and resilience.
https://firehydrant.com/blog/improving-signals-speed-and-resilience-through-pressure-testing/
A post on some of the pitfalls of platform engineering teams, including the ability for central teams to generate work for everyone else, and lose sight of their internal customers’ needs.
https://www.srepath.com/danger-of-unreliable-platform-engineering/
Pgxman is a package manager for PostgreSQL extensions, along with a repository of packages. It integrates with native build systems for installation.
https://pgxman.com/
Chalk is a new tool that captures metadata at build time, and can add a small ‘chalk mark’ with that information to any artefacts (like compiled binaries or container images).
https://github.com/crashappsec/chalk
testkube is a Kubernetes-native testing framework for test execution and orchestration. Store tests from any testing tool as CRDs and run them on the cluster.
https://testkube.io/
https://github.com/kubeshop/testkube
Daytona is a new tool for managing a development environment. It supports both local and remote environments as well as integration with various Git services and IDEs.
https://github.com/daytonaio/daytona
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
KubeCon EU finished up a couple of weeks ago in Paris. Here’s a collection of posts with key takeaways from various points of view. Mentions of platform engineering, AI, Dapr, eBPF, Backstage and lots more.
https://danielbryantuk.medium.com/kubecon-eu-2024-paris-key-takeaways-ad4c1bb7fbfe
https://isovalent.com/blog/post/kubecon-europe-2024-wrap-up/
https://backstage.io/blog/2024/03/27/backstagecon-kubecon-24/
https://medium.com/@jiekun/kubecon-europe-2024-cloud-native-in-la-ville-lumi%C3%A8re-in-the-spotlight-355a259311ac
A great list of linux tools you want to have installed to help debug a crisis, along with an argument for installing them by default, not trying to do so during an incident.
https://www.brendangregg.com/blog/2024-03-24/linux-crisis-tools.html
Monitorama is coming up, June 10th to 12th in Portland, Oregon. The schedule is now available, with talks on logs, metrics, tracing and a talk called Welcome to the OTEL California. The code DEVOPSWKLY2024 will get you $100 off the price.
https://ti.to/monitorama/pdx2024/discount/DEVOPSWKLY2024
https://monitorama.com/2024/pdx.html#schedule
OpenDevin is an autonomous AI software engineer who is capable of executing complex engineering tasks and collaborating actively with users on software development projects.
https://github.com/OpenDevin/OpenDevin
Elements is a set of UI components (React Components, or Web Components) for building beautiful API documentation powered by OpenAPI and Markdown.
https://github.com/stoplightio/elements
[ICYMI] DAST is Dead! Long Live DAST! The Evolution of Dynamic API security Testing webinar is now available on YouTube. Watch on-demand here.
https://sthwk.com/long-live-dast-webinar
A good post highlighting the importance of OpenID Connect, looking at fine-grained RBAC for GitHub Action using Vault.
https://www.digitalocean.com/blog/fine-grained-rbac-for-github-action-workflows-hashicorp-vault
An exciting 20 year journey to fix frame pointers in Linux so that profilers can work more accurately.
https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
A look at the recently published CNCF platform maturity model and how you can use it to assess your current platform investments.
https://www.getport.io/blog/using-the-platform-engineering-maturity-model-to-evaluate-your-investment-in-internal-developer-platforms
A look at the native sidecar capabilities in Kubernetes 1.29+ for Jobs and CronJobs. A very nice quality of life improvement that simplifies a common use case.
https://medium.com/teamsnap-engineering/properly-running-kubernetes-jobs-with-sidecars-in-2024-k8s-1-28-ad9b51d17d50
CI and testing infrastructure are too often not treated like the production environments they are. This post explores how you can use monitoring tools to improve the developer experience for CI systems.
https://www.datadoghq.com/blog/best-practices-for-monitoring-software-testing/
A post on using Tetragon for monitoring file change events, and why eBPF makes things like this scalable.
https://isovalent.com/blog/post/file-monitoring-with-ebpf-and-tetragon-part-1/
A run down of some of the problems with documenting software projects, and why storing the documentation with the code can have benefits.
https://www.tabnine.com/blog/documentation-as-code-principles-workflow-and-challenges/
Bytesize Architecture Sessions are a prescriptive format for a 45m to 90m architecture session focused on helping teams build a shared language and common understanding of a system.
https://bytesizearchitecturesessions.com/