What Is Platform Engineering? From A Developer's Perspective
For teams ingrained in DevOps practices, this term ushers in a broader horizon, a fresh perspective on managing infrastructure and applications. But what does Platform Engineering offer?
In this episode, Cory talks to Dave Williams about platform engineering and why he sees this as the future of running applications in the cloud.
Guest: Dave Williams, CTO at Massdriver
Transcript
You're listening to the Platform Engineering Podcast, your expert guide to the fascinating world of platform engineering. Each episode brings you in-depth interviews with industry experts and professionals who break down the intricacies of platform architecture, cloud operations, and DevOps practices. From tool reviews to valuable lessons from real-world projects to insights about the best approaches and strategies, you can count on this show to provide you with expert knowledge that will truly elevate your own journey in the world of platform engineering.
Cory:Hi, I'm Corey O'Daniel, CEO and co-founder of MassDriver.
Dave:Hi, I'm Dave Williams, CTO of MassDriver.
Cory:Today, we're going to talk about platform engineering from the perspective of a developer.
The canonical definition of platform engineering is building, designing, and maintaining the infrastructure and tooling for software developers to build software applications. Your major goal is increasing developer productivity, standardizing development processes, and maintaining high scalability and reliability of those systems. What would you say is missing from the canonical definition of platform engineering?
Dave:I think the biggest thing is that product mindset. I think one of the cool things that happened going through YC was that mantra of build things that people want. And I think approaching the management of infrastructure and managing all of the requirements of security and compliance as a product that people are compelled to use is a really important facet of that move to platform engineering.
I think it's one of the big differences between kind of that DevOps mindset of build it as we need it. It's like build a thing that people are compelled to use and want to use.
Cory:And as a developer joining a company that has platform engineering in place, I feel like that's actually a really exciting opportunity to know that you'll truly be able to focus on your programming languages, the dependencies you need, and just getting the work done that you're excited about and knowing that you don't have to deal with the tedium of operations and kind of where you touch on some of these tasks for managing infrastructure.
Dave:That's a really good point. And I think, you know, my career was funny. I came in through kind of microcontrollers and embedded systems and like in the beginning of my career, the hardware was so important.
And I think getting into some of these companies that were providing software over the internet, the culture was a lot different. It was about kind of like building the product and the thing that people touch and people were very disconnected from the hardware. And I think that kind of gave rise to that DevOps movement of like this stuff matters and we need to be able to manage it effectively.
And now like the longer that I go and now being a founder and entrepreneur, I'm kind of back to this point where it's like I just need a thing that's going to work and I need it to be modular enough to handle any changes to the market that I need to address.
Cory:How did your software development background prepare you for the challenges of platform engineering?
Dave:Yeah. So, I mean, I ended up kind of being a software engineer that straddled the line between operations and software because of that embedded systems background and because the hardware mattered. And I've always had a product mindset when it came to like building automation and things like that.
But the tooling just isn't there to like build the product until quite recently.
Cory:Yeah. It sounds like the product mindset is really one of the key things in developing internal development platforms. But what principles or practices have you found particularly relevant for or critical even in platform engineering?
Dave:Yeah. I think the biggest principles are like that the need to abstract away the hard stuff. I think operations is really, really challenging.
I think security and compliance becomes even more challenging as we distribute systems. And these things are non-negotiable for most businesses. You want to keep people's data safe.
You want to make sure that like any governing body is going to be okay with the infrastructure that you're supplying. And I think for software engineers, the need to kind of scale, right? The ability to take action at 1 a.m. when the alarm goes off is really, really important. And so I think that's the biggest thing is like that kind of mindset of like giving people what they need and only what they need and not bother them with the kind of complexities of the domain that they're operating in.
Cory:So it's interesting to think about abstraction because I feel like when we first started using the cloud, the idea was it was this thing that was going to make things simpler. It was simpler than a data center. It was simpler than racking.
But the cloud has really grown to be fairly complicated. You can even look at something as simple as running maybe Postgres on RDS Aurora. And there's just a ton of parameters that are exposed to you as a developer that may never matter.
You have this great quote that you've used around the office that the goal of the cloud isn't to be simple, it's to be capable. Can you tell me a bit about that?
Dave:Yeah. So I think if you talk to any operator that came from data centers, they will never tell you that the cloud is more simple. They're always like, oh, it was really easy to tail the logs in this box and figure out what was going on.
The magic of the cloud is it's a risk-free, right? Not having to buy a bunch of hardware in order to get your startup off the ground is awesome. And it's there to meet a huge variety of use cases and just run at massive scale.
And that fundamentally requires you to open up the system as much as humanly possible. And so the simplicity has to be on your operations team and your developers and making sure that you're finding these standardized ways to run your cloud infrastructure that works for you and making sure that that's easily repeatable.
Cory:How has your approach changed when developing internal developer platforms versus more traditional software, B2B, e-commerce, SaaS, et cetera?
Dave:So I don't know that it's changed. And I think that's the cool thing, right? If you think about what e-commerce did to commerce, right?
I remember ordering bicycle parts from mail order catalogs when I was a kid, right? And you call somebody and you're like, I need these things. And they probably, if you think about what's going on behind the scenes there, they're probably calling a person in a warehouse and like, do we actually have these things, right?
And then there's some credit card processor that's taking three to four weeks to get your money. And then finally, they're like, cool, we got money. It's time to go ship this.
And you're looking at an eight-week process, right? And e-commerce came in and cut this down to a series of calls to automated systems. And that's really, really neat.
And I think like we need the same thing in kind of operations in the cloud, like that ability to just say, like, I need this piece of inventory and have it be a collection of calls. And it comes out the way you expect it.
Cory:How do you view the relationship between application development and platform engineering, especially given your unique experience in both fields?
Dave:Approaching MassDriver in that product engineering mindset, like we created a pluggable system, right? You can almost think about it like e-commerce for infrastructure, where it's like we have a catalog of things and we have a fulfillment mechanism for the things that you're getting, right? For the cloud infrastructure you're getting.
And we have a way to like kind of audit what has been created and what has been destroyed and where it's gone. I think like it's honestly the same. It's the same principles and the same approach as it would be to building anything else.
Cory:So it's interesting that idea of calling the bicycle place, looking for the parts you need. I feel like it's very similar when you're developing a new feature and you need a cloud service. I frequently know the cloud service that I need to implement a feature.
I need Dynamo versus maybe Redis or relational database. But it's not quite as simple as going to an e-commerce site and pulling it off the shelf. It still very much feels, even though we have these automated APIs for the cloud, it still feels like I'm trying to get a bunch of different parts, right?
I want Postgres, but I need KMS. I need IAM. I need these other tertiary services just to be able to get the thing that I need. And I feel like that's not the way that a lot of developers think or should think, right?
They should be able to think about the dependencies they need, not the dependencies of their dependencies. And we've solved this in the open source world with packages, but we haven't really solved this when it comes to the cloud. What is different about the way that you think about IAC development in this platform engineering world versus how typically somebody might make Terraform modules or Pulumi modules with regard to all these tertiary services?
Dave:Just before we started MassDriver, one of the big things I thought about when it came to building a Terraform module or something like that is something being use case specific, right? It's not like, I don't need a Dynamo. I need a key value store for maybe this kind of scale, right?
And I think that makes it really easy to kind of grab a module and make a really clean API that's like, well, given this is the use case, these are the three or four fields that you really need to be aware of, right? These are the kinds of things you need to monitor. And when it comes to IAM, right?
Like you can craft the policies ahead of time where it's like this DynamoDB will have a policy for read and a policy for write. So you're able to kind of split that stuff without really having to think about it. And it's just like when creating something that's good for a catalog, you kind of think about all those educations and make sure it's all kind of prepackaged and limited enough in scope that a user knows they're, it can be confident they're grabbing the right thing at the right time.
Cory:What drew you to platform engineering from a software development? And what insights can you share with other engineers that are interested in the same path?
Dave:The need to move fast really makes platform engineering appealing. I think my experience on both sides is like as a software engineer, like a product engineer, and as a operations professional, the kind of ad hoc nature of DevOps, where it's like, I'm involved in this project plan and I'm kind of blocking people and people are kind of blocking me and kind of going back and forth. That was really frustrating.
And in almost every operations team I've ever seen, there's a lot of weird animosity kind of throw it over the wall culture, despite the fact that that wasn't supposed to happen in the operations world. And so for me, that ability to know I need a piece of infrastructure and I know I can just grab it and I can know that the company is going to feel confident that it behaves in a way that meets their standards. And I can just start writing my software and like meet the needs of my customers.
Like that's the most important thing that platform engineer could provide.
Cory:And it's the thing that I feel like developers want, right? Like, I mean, you've got 40 hours in your week to build some feature, build some value. Everybody wants to add value.
And I feel like a lot of times as a developer, you can get caught up in all these tertiary things, all these kind of negative engineering tasks that you have to do just to get to the work that your product manager is working on.
Dave:Yeah, absolutely. And then thinking about things like redundancy, like we have higher uptime requirements than in any time in the history of the internet. Everybody expects everything to be fast and up all the time, you know, because of gigantic companies like, you know, Amazon and Uber and all these companies that are just, they're up all the time and handling massive scale.
And I think like, I want to be able to write the software quickly that can handle that kind of scale and can be that redundant. And like, I think getting buried in the nitty gritty of, you know, Kafka and all that stuff is just, it can take a really long time if someone with that kind of brain isn't just focused on that solely.
Cory:As a company that dog foods our own products, how has using MassDriver to manage infrastructure changed your team's relationship with cloud operations and infrastructure?
Dave:Yeah, so I've lived through like a lot of errors of software development on the internet. So like I've had the server room and the air conditioned room and like the office, right? I've lived through kind of the early days of the cloud and I lived through like one of the most like interesting times, which is like that Heroku period, like that beginning Heroku period where they kind of started to really lean into that like Rails deployment environment as a service thing.
That was magical at a time where our software was like limited in scope, right? Like the idea that like something like a technology like Rails could be your security layer, it could be your caching layer, it could be your everything layer. As we kind of like started to realize that like this wasn't actually a thing that could exist anymore and we needed to start distributing our applications.
Like Heroku kind of fell apart and we went into this DevOps world where we're writing a lot of infrastructures, code scripts and things like that. And it honestly just started to feel worse and worse. And I think like the cool thing about platform engineering and like dog fooding mass drivers, it feels a lot more like the Heroku days where it's like, I need to run an application.
These are the six Lego blocks that I need to do like very specific functions. It's just like, it's really compelling as a software engineer. From a team's point of view, it becomes really easy to start to see the kind of like organization-wide patterns, right?
Like this is how we do queuing, right? And everybody's kind of doing queuing the same way because it's just easy. It's just like grab some Lego blocks.
With IEC, it's really difficult to go like comb terraform. It's like, what combination of modules and incantations do I need in order to get this thing that I need? It's just, it's given people a better view of the infrastructure and the patterns while simultaneously being so much more accessible to engineers.
Awesome.
Cory:Well, Dave, thanks for coming and talk to us about platform engineering today. Yeah, it was my pleasure.
Outro:Thank you for listening to this episode of the Platform Engineering Podcast. Have a topic you would love to learn more about? Let us know at cory at massdriver.cloud. That's C-O-R-Y at M-A-S-S-D-R-I-V-E-R dot cloud. Catch you on the next one.
