![]() We originally chose Postgres simply because it was a technology that the team already knew. Some workloads can’t take advantage of all those resources, or it otherwise won’t be cost-effective. That ought to be enough for anybody.Įven if your old, boring technology is having trouble scaling today, you might be able to scale up if put on a New, Big Box™ with very little effort. You can rent a 128-core, 3 TB of RAM instance from AWS. Most of the time, you’ll discover that the boring old technology is fast enough because Computers Are Fast™.Ĭomputers are also Big, too. It’s important to actually measure this stuff by running a stress test. I can’t tell you how many times I’ve heard this exact conversation in my career.Įngineer 1: “We need to adopt new technology X because boring technology Y is too slow.”Įngineer 2: “How slow is it, and how fast does it need to be?”Įngineer 1: “No idea, I never measured it.” □ Computers Are Fast™ (and so is your RDBMS) Really, I’d like to focus on the first point: performance. In environments where this is an issue, it does not need to be a deal breaker it can be wrapped in a service API just like any other piece of state. We don’t have any issues with #2, as we are proud of our majestic, well-factored monolith (but that’s another blog post… someday). Many haters want you to buy their message queue product.Sharing direct RDBMS access across a fleet of services is bad practice.RDBMS’s are too slow to act as a message queue.However, using a relational database - such as a Postgesql database - as a message queue is a well-known anti-pattern. □ Postgres as a message queueĪs we were already running Postgres, if we could prove to ourselves that Postgres wasn’t the wrong tool for the job, it might be worth using instead of a new piece of infra like a dedicated message queue. This was a technology our team already knew how to use, and from prior experience, we knew it could serve our initial use cases well. When we built the first features in the open-source project, we reached for Postgres. It’s better to use a tool that you’re already using, even if it’s not the perfect tool for the job.If existing infrastructure can’t solve the problem, or it will be too slow, expensive, or insecure/noncompliant, don’t use it. Never choose the wrong tool for the job.So here’s my philosophy when faced with a new problem: And it’s often sustained work over a long period of time. It’s just a lot of work to add a new piece of infra. Evangelizing it internally and driving the right level of adoption.When I say costs, I mean stuff like this: If you’re going to add new infrastructure, it better be really important and useful infrastructure. The tl dr is that adding infrastructure incurs significant costs. I am a big fan of Dan McKinley’s essay on Choosing Boring Technology. Or is it? □ Don’t choose the right tool for the job Furthermore, we have a lot of data coming from our customers, so it’s critical that we process this data in a scalable, cost-effective way. It’s an append-only, ordered queue of messages organized by topic that we want to consume in near-real-time. Imagine the logs rapidly scrolling by in the bottom paneĪt first glance, ingesting these logs looks like a classic Kafka-shaped problem. In this post, we’re focused on that second component.Īs the data pipeline executes, it spits out lots of event logs that we must store and index to render our UI. A logging system that aggregates structured and unstructured information about the pipelines and reports it to the user via a UI.A scheduler that kicks off runs of data pipelines.At its core, there are two main components: □ Scaling the database: archiving and rate limitingĭagster Cloud is a system for orchestrating data pipelines. ![]() □ Don’t choose the right tool for the job Today I’m going to talk about why we made the unconventional decision to build our logging system on top of Postgres instead of Kafka, what worked well, what didn’t work well, and how we did it. It’s been pretty successful so far, and this is the first in a series of blog posts talking about how we built it.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |