https://joshliburdi.com/Recent content in Josh Liburdi on joshliburdi.comHugoen-usTue, 02 Jul 2024 00:00:00 +0000https://joshliburdi.com/writing/2024_aws_exactly_once_semantics/Tue, 02 Jul 2024 00:00:00 +0000https://joshliburdi.com/writing/2024_aws_exactly_once_semantics/WTF are Semantics? If you spend even a little bit of time reading about or building distributed systems, eventually you come to understand that there are three types of message delivery guarantees: At-most once: Message delivery is not guaranteed. At-least once: Messages delivery is guaranteed with possible duplication. Exactly once: Message delivery is guaranteed with no duplication. Ensuring that these semantics are met is a shared responsibility between the message queue – in this post, I’ll cover some popular AWS services – and the producer or consumer of messages (i.https://joshliburdi.com/writing/2024_reduce_aws_kinesis_costs/Sat, 15 Jun 2024 00:00:00 +0000https://joshliburdi.com/writing/2024_reduce_aws_kinesis_costs/The Trick Aggregate multiple records into a single aggregated record. That’s it. Why It Works The AWS Kinesis Data Streams service bills based on two criteria: the number of shards running per hour (“shard hours”) and the amount of bytes per record put into each stream. Each shard has a maximum capacity (either 1,000 records written per second or 1 MB written per second) and while there are potential savings in managing shard hours, the greater cost savings is in managing record sizes.https://joshliburdi.com/writing/2023_bluenomicon/Sat, 01 Apr 2023 00:00:00 +0000https://joshliburdi.com/writing/2023_bluenomicon/I contributed the chapter “Building Loosely Coupled Threat Detection Systems” to Splunk SURGe’s book, Bluenomicon: The Network Defender’s Compendium. The book is a collection of essays from security professionals on a variety of topics, from threat hunting to incident response to cloud security, and it’s “free” (requires contact info). I also have several physical copies if anyone wants one.https://joshliburdi.com/writing/2022_announcing_substation/Tue, 25 Oct 2022 00:00:00 +0000https://joshliburdi.com/writing/2022_announcing_substation/Over on Brex’s tech blog I wrote a post announcing the release of Substation. The original text is copied below. Introduction We are thrilled to publicly announce the release of Substation, an open source toolkit for creating highly configurable, no maintenance, and cost-efficient data pipelines. Substation solves a problem that every security team has, but few may recognize - the need to normalize, correlate, and enrich their security event data at scale.https://joshliburdi.com/writing/2020_structured_threat_hunting/Sun, 29 Mar 2020 00:00:00 +0000https://joshliburdi.com/writing/2020_structured_threat_hunting/Preamble In October 2019 I joined the Splunk Global Security organization to build Splunk’s internal threat hunting program. Over a few months we went from an organization with no defined hunting program to one that can do full-scale, high-value hunting. This post describes how individual hunts can be structured to maintain focus and avoid “rabbit holes.” Before diving into this topic, I recommend that you take a moment to watch this presentation from Justin Kohler and Patrick Perry – it covers some of the concepts described here.https://joshliburdi.com/writing/2020_threat_hunting_metrics/Sat, 28 Mar 2020 00:00:00 +0000https://joshliburdi.com/writing/2020_threat_hunting_metrics/Preamble In October 2019 I joined the Splunk Global Security organization to build Splunk’s internal threat hunting program. Over a few months we went from an organization with no defined hunting program to one that can do full-scale, high-value hunting. This post describes the metrics used to evaluate a hunting program. Measuring Success Before we operationalized our hunting program, I knew that metrics tracking needed to be a part of everything we did.https://joshliburdi.com/writing/2019_not_today_microservices/Sat, 02 Mar 2019 00:00:00 +0000https://joshliburdi.com/writing/2019_not_today_microservices/This post is additional content for my presentation at 2019’s BSides SF conference that describes the decision to not build Strelka with a microservices architecture. Here’s my take on microservices: they’re for the 0.01% of application owners. Microservices, like any system architecture model, have advantages and disadvantages; for the majority of systems (including Strelka), I think the disadvantages outweigh the advantages. Don’t just take it from me, here’s a quote from Martin Fowler, an often-quoted proponent of microservices, taken from a blog entry named Microservice Premium:https://joshliburdi.com/writing/2019_stenographer_grpc/Sun, 20 Jan 2019 00:00:00 +0000https://joshliburdi.com/writing/2019_stenographer_grpc/This post describes the process of building a secure packet capture (PCAP) retrieval service for Google’s Stenographer using gRPC and Python. The service allows clients to securely request PCAP from a server running Stenographer and stream the PCAP back to the client. Components of Stenographer Stenographer contains several utilities (stenotype, which writes packets to disk; stenocurl, which interacts with the Stenographer system; etc.), but for this service we only need to build on top of stenoread.https://joshliburdi.com/writing/2018_huntpedia/Mon, 01 Jan 2018 00:00:00 +0000https://joshliburdi.com/writing/2018_huntpedia/I contributed the chapter “Hunting for Command and Control” to Sqrrl’s book, Huntpedia: Your Threat Hunting Knowledge Compendium. The chapter was originally a blog post on the Sqrrl website and was later reused in the book. I don’t remember when the book was released, but I think it was sometime in 2018 (after I left the company). I may also have a few physical copies if anyone wants one?! If you don’t know what Sqrrl was or what happened to it, then go here.