How to Ship Faster with Feature Flags: Insights from Unleash

Cory: 00:00:39

Welcome back to the Platform Engineering Podcast. I'm your host, Cory O'Daniel, and today I'm joined by Michael Ferranti, Chief Marketing Officer at Unleash.

Michael's career has taken him through Rackspace, portworx, Teleport. He's always right there where big shifts in how we build and run software happening.

These days he's focused on helping teams use feature flags to ship faster with less risk. Michael, welcome to the show. Can you tell us a little bit about your background and maybe how you came to found Unleash?

Michael: 00:01:07

Thanks for having me. First of all, really excited to talk. Love the podcast, love platform engineering generally. So, so eager to dive into some technical topics.

I guess my background kind of, I would call it broadly like a product leader, full transparency. I'm not an engineer. Right, okay. Sorry everybody. Like now, now, you're now your listenership.

Oh, I like to say that up front because I can, I can still talk about technical things. The reason I do what I do is because I love working with engineers and I wanted a job that would allow me to be really, really close to technology.

And it turned out that, you know, working in developer tools companies is a great place to be because ultimately these are, these are the people that are building the world that we live in right now. And I think they're just, you know, interesting people to interact with and then solving really hard problems.

And so I like to work with engineers who are building the technologies that, to solve those hard problems and then just help people understand how products and solutions can solve what they're working on.

And so like kind of last five to 10 years, it's really been all around containerization, Kubernetes DevOps infrastructure as code, like, you know, shift left security, like all of, all of these things. Hopefully that's not just like, man, this is just a bunch of buzzwords. But that's, that's been kind of the space that I've lived in.

And so, yeah, just eager to kind of talk through how some of these things intersect with platform engineering.

Cory: 00:02:31

Yeah, I love that. I was actually Very excited when I saw your email come into the inbox.

You know, I think one of the things that I've always been excited about and that's attracted me to the world of DevOps and platform engineering is just I existed in the pre DevOps world. I was a developer and an operator back then. Depending on which job I was in, there's a couple different ones.

And I felt that friction as a developer to getting things out and trying to get things into a release that was going to go out on, you know, next Monday. And if you didn't get it in by Friday, you were just kind of shit out of luck.

And one of the things that was always excited about DevOps is this idea of making the teams more efficient, getting changes in. That's what it's all about at the root of it.

And platform engineering to me was always an extension of that, going further in the automation, going further in making it more fluid and easier for developers to deploy.

And one of the things that I think that I've seen so many times at so many organizations is even for many companies that have gotten into the point where they're doing continuous deployment, continuous delivery, they're still hampered on testing things out and testing features in a staging environment or a series of staging environments. And I feel like that has always been just an antithesis to moving quickly. Right, because you're still getting in a queue behind other people.

And so when I first saw your email come in, I was like, this, this is great because I think that feature flags are just paramount to moving quickly whether you're doing platform engineering or not.

But like, once you're at the scale of platform engineering, I think this is something that teams absolutely have to be considering as a part of their product. And I think that many people just kind of see feature flags as a way to test new UI features.

And I think there's a, there's a bigger story to feature flags than just like, what do you show in, you know, a react app? Like, how do you see them with, with relation to today's software delivery?

Michael: 00:04:28

You're coming with the big questions right off the bat. Corey, it's like, sorry, sorry, this is.

Cory: 00:04:34

I was just like, man, I've had. Normally I start with some softballs and we bullshit a bit, but this is something I love. So sorry, sorry about that.

Michael: 00:04:41

I was expecting to get into like beard routines and things like that, but like, we're going, we're going right here.

Cory: 00:04:46

Dude, this is, this is three weeks of growing. I can't make a beard I don't even know how to talk about that stuff.

Michael: 00:04:52

Yeah, for another podcast, it's a great question.

And I, like, I don't want to get too serious too fast, but you said the thing that I was hoping we could get to at the end of the conversation, which is feature flags.

It's kind of a misnomer because, like, unless we overload the word feature, when we say feature, we kind of mean like functionality and usually end user functionality. It's, you know, the signup process for a mobile application. Or it's like, you know, I want to do a bank transfer and I click somebody like that.

That's generally what call a feature. And feature flags are really good for that.

And we'll get into reasons for kind of exposing those features gradually to users, rolling the back if there's some bug. But really any code, any software is like, also features.

And so if it's a backend API, like, you should also be thinking about that in the same way that you would for a front end quote, unquote feature. And there's an anecdote that came up recently and this is like, I don't know, you know, it's just, it's a, it's a reality.

Like cloud providers have outages. Like, it's not a news flash to anybody. And Google had one recently and you know, Amazon has one.

So it's not about Google, it's rather they had a major outage. Half the Internet had an unscheduled bout of productivity because like all of the major social networking sites were down.

And in the postmortem for that, Google identified that they had a critical binary that they shipped normally. When they ship this type of update, they wrap it in a feature flag. For reasons that they describe in the release, they didn't this time.

That meant that what would have been caught in staging wasn't caught and cascading effects of, you know, a global service provider had, you know, Gmail down, like, you know, popular social media sites and things like that. And kind of the outcome of that postmortem is like, yeah, we're really going to always use things that put things in a feature flag.

And basically they're saying, right, the most sophisticated, like DevOps platform engineering team, like they invented SRE, they essentially invented platform engineering. And they're saying like, feature flags are just a way in which we ship software.

Because even if you're really, really good at deploying software in an automated manner, when you have kind of these distributed systems like it does with caches and all of these things, even an automated deployment is going to take time to show up in the end user's device. And flags give you a runtime control mechanism to change things instantly. And we really see it as complementary to DevOps.

It's, you know, you shouldn't just like, you know, one, not do CI CD, not do you know other DevOps practices and only use feature flags instead. Its feature flags are a complement kill switch is one like very obvious example of why you would want one of these things.

But also you mentioned kind of deployments and oh, if I don't get it in now it's like it's not going to have to wait a week. Well, it's like feature flags also allow you to do trunk based development where you're regularly pushing to trunk.

And so it's like hey, I just finished my cup of coffee, I'm going to go to get another one.

You can push things up and it's okay that it's completely buggy, that it would like be a terrible experience if anybody used it because it's off, it's wrapped in a feature flag. And so I liken feature flags to things like source control for things like cicd or things like, you know, infrastructure as code capabilities.

These are just, they're just part of how teams should build software. They really should build software.

You should, these are primitives that they just make sense, they make things easier, they reduce risk, they increase velocity. And now how you do feature flags is like that now starts to say okay, well you know, we're going to do it this way.

We have our own system that we use or hey, we're going to use, you know, unleash an open source provider, we're going to use, you know, launchdarkly, which is another feature flag leader.

It like there's not one great one solution for everybody however, like I don't care who you, if you use GitHub or GitLab or BitBucket but like you really should be doing source control. That's kind of how I see feature flags playing into modern engineering.

Cory: 00:09:08

Yeah, I think that's one of those things that's pretty interesting.

Like I'll see teams, they're earlier like in their journey, you know, maybe they're working on their MVP and it's like yeah, maybe you don't need feature flags.

But then I'll see people like just, they kind of, they get bought into the idea of a staging environment and they kind of get locked in this flow and then feature flags might not come up Again, until, like, we're close to being a big boy enterprise or whatever. And it's like you really can start to yield value from them very early.

I mean, I wouldn't say necessarily throw them in your mvp, but like, as soon as you start having customers using your product, like, that's where I think they start to be really beneficial around UI to make sure you're like, releasing features slowly to people. But, like, as you reach scale, like, I feel like they are so important for being able to maintain that fast shipping cakes.

Like, just like you're saying, like, even just something as simple as a kill switch, right? Like in our product, we use feature flags pretty exhaustively. And, you know, we dog food. We're a platform for managing platforms.

We run our own tool on our own tool. And so we feature flag ourselves into everything first, right?

And it's just like, let's make sure that it works for us before we start rolling it out to the entire audience. And there's been plenty of times we've seen a feature go out. 10% of the custom hits 10% of the customers, and it's like, oh, this, this works.

This is great. We did it. Yay. We shipped software. And then it hits 50% of the customers and all hell breaks loose, right?

And we have pretty quick rollbacks, but there's nothing as instant as a checkbox, right? It's just like, oh, whoa, whoa, whoa. That's what's breaking things. We can see it in our telemetry. Boop. Turn it off. Great. And it's gone.

I don't have to wait, you know, three minutes for a container to pull and a bunch of pods to rest. That three minutes, depending on the importance of your software, could be pretty damaging to somebody's environment.

And so it's one of those things I would love to see teams starting to invest in earlier. But what is the common friction that you see earlier teams or smaller teams kind of stumble into?

Or what are the pushbacks that teams have against adopting something like feature flags as a primitive in their software alongside the other things you mentioned that we still skip out on teams that skip on IAC and skip on cicd. What's the biggest pushback you see there?

Michael: 00:11:15

That's a great question. There's kind of two sides of it. One is there's the pushback, which is like, this is more complicated than it's worth.

We're a small application or a small team, and if we add in this, you know, additional capability, like, it's something either we're Adding a dependency because you're using a third party and like, you know, there's a kind of a cost to doing that. I'm not talking about monetary costs so much. It's, you know, there's performance, there's, you know. Well, now, like from reliability, like, do it.

Do I really trust this library that I'm adding? Because I'm kind of, I'm putting into my code base, I'm responsible for it. And it's like I kind of can't think about that right now.

There's the, like, I really don't get why this would be valuable. Why. Why is this better than, for instance, you know, having a feature branch and then I test that, you know, in staging and then I promote it or.

Yeah, everybody says merge conflicts, you know, or pain in the butt with feature branches, but like, I've never personally had that. I'm like really good at, like resolving, you know, merge conflicts or whatever.

So there's like probably a hundred different reasons that people kind of don't think that feature flags provide sufficient value to justify sometimes the zero monetary cost. But, you know, adding a dependency or overhead or maintainability, things like that.

The other common pattern, and I would say it's maybe 50, 50, is people that are like, oh, feature flags are really awesome. I'm just going to add if statements everywhere. And now I'm doing feature flagging.

The customers that we talk to and in fact, we have a user conference next week, it's called UnleashedCon. And we've got companies like Prudential and Wayfair and Lenovo and Mercadona, which is the largest retailer in Spain.

Like, these are the kind of types of companies that typically we use Unleash. And almost all of them started with their own feature flag solution.

Because the number one kind of meme on feature flags is it's just an if statement.

Cory: 00:13:08

Yeah.

Michael: 00:13:09

And it's not wrong.

However, it's also wrong because, you know, an if statement that's running in, you know, on Amazon.com, like, that's a very complicated piece of software and, you know, it builds up and this team is like, oh, we can now flip on and off. Well, now like, can we do it to like just 50% of users? Okay, we have that functionality in. Oh, now can we add this targeting rule. Oh, now can we?

You know, I want to track experiments and so I need to take that feature variant and tie it into my analytics system so that I can tell which of my experiences and you kind of build up that functionality and Then you kind of. It becomes out of control.

Cory: 00:13:44

Yeah, I. I'm laughing because you just described an absolute, like, miserable year of my life. Like, a decade ago, I worked for a pretty big company and we, we had a single staging environment and then we had three and you'd have like, lanes.

You're like, oh, I'm in staging lane one, like, waiting stuff to go out. And we started rolling our own feature, Flag. Like, I don't remember what we called it, but it was, yes, two thousand and twelve or so.

And I was like, oh, it's just, it's just an if statement. It's just it's on or it's off. And then there's just a couple of toggles that appear in our admin console.

And then like, the marketing team sees it and they're like, ooh, can I filter this to like a small audience? It was just like, all of a sudden, like, it felt like our job became like, managing all the possible rules that this thing could have.

And it was like, we have this business over here that makes us money that we have to kind of lean into. But, like, it was like we were just kind of like, just roll. Because it just.

It seems so easy, but then like, really to yield the bigger benefits from it. Like, there is a lot of functionality that exists in mature products like Unleashed.

I think one of the other things that's particularly exciting to me about what you all do is that there is an open source version of it. Right. I think that's one of the bigger pushbacks is like, how do I deploy this in an air gapped environment? Right.

Like, there are reasons to air gap or run things on prem that you still want to feature Flag and you might not want to be reaching out to a third party across the public constantly.

And being able to run Unleash, like in a Kubernetes cluster right alongside your app not only gives you that sense of security, but it also gives you much faster response times for hot paths where you have critical features sitting. So you said you have Unleashed Con. I think when this episode comes out, it'll actually be in the past.

So can you tell me a little bit about UnleashCon and how often are you guys running this conference?

Michael: 00:15:27

Yeah, so it's our annual user conference and folks can Google "getunleash", and then our unleashcon and YouTube. There's actually an HR conference right now called Unleash happening in Australia and they're spending a lot of money advertising.

So, like, depending on when you Google it... we're not the HR conference, we're things engineering... add feature flags to it, you'll find it. But it's our annual user conference, and so we do it generally in September.

As I mentioned, this year, just some really interesting talks and a lot of them for platform teams. Not surprisingly, you know, in, I don't know, Prudential, they've got tens of thousands of developers, hundreds of teams and like each one of those teams is not, you know, picking their own stack. And so they have centralized platform engineering providing basically a service catalog to all of their different users.

I think it's really exciting. You know, some people might think it's boring but like how do you... your Prudential and people literally have like their life insurance policies with you and things like that. It's like the cost of failure is really, really high and like "move fast and break things" doesn't really fly in that type of environment.

Yet, you know, they're trying to recruit developers who are also getting recruited by the big Silicon Valley awesome tech companies and things like that. And so if you have a developer who could work at, I don't know, OpenAI and they're going to do Prudential instead, and then you just like bury them in all of this red tape and they're like, "Man, I just like to want to write software." It's a non-starter. So what they've done is like enable a tool like Unleash to do feature management so that people can just kind of develop in a more modern way.

But on the backend, they're taking all of the changes that are going to come to flag state. And that could be, you know, on/off things but also rollout percentages, things like that. Like anything that they deem as an auditable event... Unleash is very composable, so they're kind of taking our APIs integrating with ServiceNow. Basically generating tickets on the backend that are completely invisible to engineers.

And so they just get to use the Unleashed CLI or API or they're interacting with kubectl or like whatever tool that they use is, and all the stuff is happening on the backend. So they're talking about that from a platform engineering perspective. How do you create those types of experiences?

Another great talk is going to be from Wayfair. So Wayfair, many of your audience probably have bought... you know, things maybe the chair that they're on came off of Wayfair. They're a really interesting story. They had a homegrown feature flag system themselves and it just over time became a maintenance nightmare because they're growing, scaling like crazy and, you know, they really want to make the best e-commerce experience for home goods and not the best feature flag solution. And they ran into a situation where they had some downtime due to that piece of software.

We're talking very, very expensive when you're talking about, you know, a sales event or something like that. So they've been using Unleashed for a few years now, talking about that journey and then in particular some of the stuff around AI that they're experiencing as they kind of like use gen AI copilot type workflows, build agents and how feature flags are actually a great complement to that.

Cory: 00:18:43

Yeah.

Michael: 00:18:43

Which is super interesting. And just, you know, bunches of other talks. So I would encourage people to check up the YouTube if you want like a platform engineer's perspective on feature flags. This is the place to go - hear it from our customers, not from me.

Cory: 00:18:57

Yeah, and we'll put those... I mean they'll be live by the time this episode goes up, so we'll put those in the show notes. So if any of those topics sound interesting, definitely give it a click.

One thing that you said there that I love and I think we're coming up on this time of year, so I'd kind of love your take on this. My background before starting my current company was retail and travel.

And retail and travel has a very special time of year coming up in the next couple of months, Black Friday. And there are so many teams that maybe what, three weeks out, two weeks out from Black Friday, they start the freezing. The freezing of software.

And the freezing of software just boils my piss. I'm a big believer in. I frequently will deploy at 5pm on a Friday just to like feel good about it. Like I do it, I love it.

And the reason why is I lean into feature flags. I lean into like really good CI/CD pipelines that.

Michael: 00:19:51

Not to interrupt you, but can I just say that like the number one T-shirt seller in our swag store is a T-shirt that says "Badass developers deploy on Fridays". So I'll send you one Cory.

Cory: 00:20:06

Hold on, I'm marking that too so we can link to it. But I'm also marking that so that I remember to get one.

Host read ad: 00:20:14

Ops teams, you're probably used to doing all the heavy lifting when it comes to infrastructure as code wrangling root modules, CI/CD scripts and Terraform, just to keep things moving along. What if your developers could just diagram what they want and you still got all the control and visibility you need?

That's exactly what Massdriver does. Ops teams upload your trusted infrastructure as code modules to our registry.

Your developers, they don't have to touch Terraform, build root modules, or even copy a single line of CI/CD scripts. They just diagram their cloud infrastructure. Massdriver pulls the modules and deploys exactly what's on their canvas.

The result, it's still managed as code, but with complete audit trails, rollbacks, preview environments and cost controls, you'll see exactly who's using what, where and what resources they're producing what all without the chaos. Stop doing twice the work. Start making infrastructure as code simpler with Massdriver. Learn more at Massdriver.cloud.

Cory: 00:21:12

But so, I mean, this is the wild thing is an industry that is so tied up on so many of their profits being on like this particular day of the year.

And we all know as software developers, the longer that we wait to introduce change into a build, the higher the chances are of there being a problem.

Michael: 00:21:29

Right?

Cory: 00:21:29

That's what sucks about the once a week or every two weeks or once a month deployment is all these changes happen at once. And then when it rolls out, like, "What broke the build?" It's so hard to know.

Faster, smaller features are what enable us to move quickly and reason about the systems that we build and troubleshoot the things that are broken.

And seeing these teams that have been on these teams where it's like you're in a feature freeze for seven weeks and it's just like seven weeks of the year. It's more than 10% of the year, guys. That's a whole lot of the year that we're not shipping Software. And then January 2nd, I don't know.

I don't know if everybody else gets seven weeks off. Like, you're still developing.

Like, there's still things that the product managers want and they're all getting queued up for, hey, once the holiday season's over, we can release this all the first week of January. And I've just seen so many platforms just implode and have problems.

And so hearing like, you know, companies like Wayfair investing in feature flags, exciting to me.

Not necessarily because I'm like, oh, I can buy things faster, but it's just like knowing that those teams, like, aren't getting stuck in that, like, feature freeze.

And so for teams that do experience this today where it's like, oh, it's coming around like the time of the year where our conference is or where our Black Friday is or our big, our big releases and we can't release anything.

How can those teams advocate for bringing in feature flags and introducing it to the Org to make the grander org more comfortable with quicker releases and smaller releases so they don't have to kind of get locked up in this moment of just being frozen in time indefinitely.

Michael: 00:23:05

Yeah. So last year at Unleashed time, we actually had a talk from an engineer called how to Convince the Bosses.

And it was a talk that really, it took an engineer's perspective to say, hey, this is a technology that I really believe in, that I know is valuable, but I need to convince someone else who's not an engineer necessarily, or at least hasn't been hands on keyboard in quite a while about why we should do this.

And I think that's a very interesting perspective because it's not always, I would say the most technical argument is going to win because a lot of times people that are making these decisions about like, hey, we're going to have a release window and they are also thinking about, you know, things that might come across their desk is, you know, should we move away from Microsoft SQL Server to, you know, CockroachDB? Like they're thinking about that. They're also thinking about is Google Cloud a better fit for the Amazon cloud?

And you know, they're also thinking about should we be looking at OpenShift? And the point I'm making is these are three completely different decisions. But at the end of the day, it's the same dollars, it's the same time.

And so they have to figure out how to choose between one versus the other.

So it all comes down to figuring out how do you for feature flags, we call kind of broadly feature ops is how does it compare against other things that you could spend time and money on? And so we use some metaphors.

For instance, like to say, if you really want to go fast and you're building a race car, where are we going to put our money? Right. Obviously we're going to put our money in the engine. That's super important.

Well, did you know that like brakes are actually one of the most important things on a race car? Right. They have super, super performant and powerful brakes. Why is that?

Well, like race cars go around a circular track and if you don't have brakes, you're going to fly off of the track. Instead of going really, really fast, you're going to be stuck burning.

So you can kind of help people who are non technical understand that this is a control mechanism that's going to allow us to speed up dramatically. There are other metaphors or I guess other maybe examples. So one would be like the Google one that I mentioned earlier.

It's like they decided not to ship something with the feature flag when they usually do. Then what happened? Well, half the Internet was down. How many millions of dollars in lost revenue plus reputation risk was that for Google?

Another really interesting example is Sonos's app rollout of a few years back. I can't remember exactly the year, but they basically had a major update that they wanted to tie with the marketing launch.

And so they did what we call a big bang release, where they kind of held back a lot of features until they could kind of reveal it all at once to the world and make sure that everybody wakes up and notice. And unfortunately, everybody woke up and noticed that none of their semless equipment worked anymore.

Cory: 00:25:42

The worst way to notice the feature is that nothing is working.

Michael: 00:25:46

Oh, I feel that exactly. And I really, really feel for the team.

And in their postmortem, what they acknowledge or what they realized was that when you have a really, really complex ecosystem of they had multiple generations of hardware, multiple generations of software running on different versions of hardware. It's a very, very complex compatibility matrix. It's really hard to test all that stuff in a lab.

And a much safer bet is to, for instance, ship that update with just your latest generation software, your latest generation hardware.

These are probably the people that are going to buy the new thing anyway, because, like, someone who invested in the Sonos 10 years ago, like, are they really going to be ready to upgrade? But people that are always on the latest and greatest might be. So like, it makes business sense intuitively. It reduces the scope of the surface area.

And so I would say if you're in a company that has these code freezes, try to pick the metaphor that's going to work for your team to help them understand that we're actually making it more likely that we're going to get bugs in the future by doing it this way.

And then, you know, the nice thing about an open source solution is you need, you can come with a solution too, that zero cost, which is to say, hey, like, I really think we should do it this way. And there are ways in which we can do it. I know Bunny's tight, right? We can bring this software in, it can allow us to do xyz.

Here are these other Companies that we say we aspire to be like, who are doing it too.

And I think slowly but surely like you can maybe you don't put it in like your, your flagship application to start, but you start with subside applications or things that have, which is a little bit lower risk that maybe aren't, you know, subject to that code freeze. And when they start to see the velocity of that increasing, you can say, hey, we've been doing this for the last six months. It's worked really well.

Like our all the door metrics. Right. Our mean time to restore is faster if there is a failure because we've got kill switches in there.

Our change failure rate is lower because, you know, we don't have these big banks. You can kind of start to make an argument that way.

Cory: 00:27:42

Yeah, I think that we said there about the Doris is. I think is, is very interesting.

I mean I'm one of those people, I love reading surveys and I love getting past like the first three pages because the first three pages they tell you all the happy stuff and then when you get down to page 30 you're like, oh, half the world's still sad.

And the door report, when you get to the end of it and it's like breaking down outages and recovery time and you still see that there's companies that are like, hey, we release once a month and when we have an outage it can take up to a week to restore the service. And it's like that's gotta be like smaller than 1% of the folks out there. And it's like 50% of respondents and you're like, oh, oh, so this.

Michael: 00:28:23

Yeah. How the other half lives.

Cory: 00:28:25

Yeah, it's just like this DevOps stuff isn't being picked up at a lot of places yet.

It's almost like I'm not sure what the right word is for it, but maybe counterintuitive for the non engineers, it's like, oh, well, when we release once a month, we break stuff all the time. So why do we want to introduce changes faster? Because we are. We move slow and break things.

Michael: 00:28:45

Yeah, I don't want to break something every week.

Cory: 00:28:46

Yeah, yeah, I don't want to break something every week. I don't want to break some. These people deploy 25 times a day. I don't want to have 25 breaks a day.

I feel like that's, that's one of the things that can be difficult to convey the importance of feature flags to people that are outside the org. That one always gets me because it is it's difficult and I feel like you can reach for.

And I'm sure you guys probably got some great, great case studies around this. But, you know, sometimes as an engineer, like, trying to present these things, like, why is CI CD important to people that are outside, right?

And some people are probably hearing this, some engineers thinking, like, just do it. That's always been my approach, is just do it. It's like, this is paramount and this is table stakes for good software engineering.

We're just going to start doing CICD and we're going to start writing tests, right? But a lot of engineers still have that friction of like, oh, I have to figure out how to get time to do it, right.

And I feel like that's one of the things that's.

That when you're in that role is like, oh, I have to see if a PM's going to let me or my lead's going to let, or I have to argue for, like, why we want to do this. And good case studies that make sense to the rest of the org, I think can be very beneficial there.

A lot of times we'll come up and we'll be like, oh, it's going to make us faster as software developers. Well, if you guys are faster, you're going to break shit faster because we're already breaking it when we release once a month, right?

But then the other side of it, where you start to see companies like Wayfair maybe releasing new features right up until Black Friday and not having an outage, right? And being able to talk about, you know, last year we had a big outage and we froze for seven weeks.

And then we had a huge outage in January and this year we didn't and we made more money, right?

Because, like, at the end of the day, like, that is all that we're here for, right, Is to build software to build companies money to pay us to do our jobs.

And I think that that can really speak to a lot of people outside the org, as long as we can get them past that kind of choke point where they think like, oh, if we release faster, we're going to break faster and they move fast and break things. I don't think it was ever about moving fast and breaking the product.

I believe it was about moving fast and breaking norms and breaking rules and shipping a good product. I don't think Facebook was ever, like, the site should down for like nine months, right? So a topic I love to bring in.

I, I feel like every, every time I bring this topic up, it's It's. Some people have very different takes on it, but it's everywhere right now. And that's. That's AI.

It's kind of getting kind of laced into every conversation.

Michael: 00:31:00

And so I personally welcome our AI overlords. Just for. Just for the record, I don't.

Cory: 00:31:06

If you look at my chat history with ChatGPT, our AI overlords are not going to be very happy with me. I've been meaner to chatgpt since Sam Altman told us to be nice to it. I'm like, no, dude, I don't listen to you anymore.

But with AI kind of getting layered into everything, what do you see as the real opportunities when it comes to combining AI with feature flags and good DevOps and release practices?

Michael: 00:31:28

It's a topic, believe it or not, that we spend a lot of time talking with customers about. In particular, since we're on the topic of UnleashCon, check out Wayfair's talk on this.

I was just speaking with them recently about what they're going to be talking about and a big part of it is AI. Some of the insights that they share actually come from The State of DevOps report, the most recent one.

Cory: 00:31:47

Awesome.

Michael: 00:31:47

That looked a little bit into kind of like AI coding and things like that.

So what we see it's our own experience as kind of engineers writ large, working on our own product, talking with other people in the community is that AI generally is increasing velocity of output. Right? Measure that in lines of codes.

I know that's a stupid metric, but like you just like there's more software that is able to be written that's accelerating and that's confirmed in the State of Job Ops report. But there's a slight decrease in the reliability of that code. This is not surprising.

It's not that AI is worse at software engineering, it's just like, that's probably a bigger topic about why that is. I've talked to some engineers recently where it's just like code reviews themselves are like, like it can be harder.

There's an assumption that maybe it's okay. And so it's like, well, like, I'm not going to look at that.

Or it's like you asked for like one change and it's like, well, you know, let me refactor the entire application and the PR is massive or whatever. And so like stuff slips through. There are lots of reasons why that there might be a slight decrease by the report.

I might be getting these numbers wrong, but like in general, it was kind of like for a doubling of AI. There's like a 7% decrease in code reliability, something like that.

And so there's more code is being shipped and it's slightly less reliable, but the volume of those changes are increasing to such an extent that even though kind of a QA based way of testing is not really scalable, even if you're not using AI, it's certainly not scalable if you are using AI.

And so what we come back to is this idea that like AI changes a lot, but it doesn't change fundamental practices around how to build and run software.

And so just like we're not going to have, you know, an agent that is making changes without going through source control, just like we're not going to all of a sudden make changes manually, be sshing into servers in order to do deployments, right? There's, there's automated builds and things like that, automated tests. We're going to keep that running. Same thing for feature flags.

Like you want any code that ships with AI assist or any agent that's running to be wrapped in feature flags. So that one, you can just accelerate the delivery of that software, right? It's probably better or it's probably good, right?

There's a chance that it's bad, but it's probably good.

And you want to get that out and you want to see if it's working, you want to run experiments, you want to do all the things that like we're in the business to do. And if it's not right, you want to be able to roll it back quickly.

And so you can imagine how in a truly agentic environment, if you have this runtime control mechanism, you can be ratcheting up a rollout based on metrics, right? We call these things impact metrics. So those might be engineering metrics. I'm monitoring my Grafana dashboards. What's my error rate?

What's my memory and CPU consumption? What's my infrastructure cost? Like all of these things that engineers care about, right? I'm rolling something out based on those.

I'm also rolling it out based on, we call Voice of Business metrics. Are conversions increasing? Are people buying more things like that? And then also what we call voice of Customer metrics.

So are we getting thumbs up when people use this new feature? Are they saying, yes, I love this or no, I hated it, right? Or they, if you give people the ability to do free form answers, tell us what you think.

I hate it, I love it. Right? You can roll out things based on that.

We call this by the Way full stack experimentation where you're taking a basket of metrics across many different systems and then basing progression of your rollout off of those metrics. You can imagine how agents can do this very, very quickly.

And with that control mechanism of being able to ramp up a rollout, ramp back, a rollout proves very, very effective way to increase the velocity which you're learning. And that's the key to all of this stuff. If you don't have that control mechanism, then it's just, it's not impossible.

It's more difficult to instrument the ramping up than the ramping back. If something goes wrong, the kill switch. And so feature management is one of those things that makes a ton of sense if you're not using AI.

And it makes even more sense if you are using the AI.

And to go back to your question about how do you convince the bosses, this is one of the things that we're hearing from our customers which is to say I could talk to him blue in the face about the benefits of trunk based development and gradual rollouts and all of these things, blank face. If I tell my boss that hey, like this is an investment that we can make that's going to accelerate what we're trying to do.

What we've said to, what we've said publicly to Wall street, right, that we're going to make this amount of progress on AI by twenty twenty-seven. Feature flags are a way in which we can, I'll say, guarantee the success of that.

Just because it's the control mechanism allows you to speed up and also mitigate the risk at the same time, which I think is really powerful. And it just points to the fact that like, like you know the book, you know, how to make Friends and Influence people, right?

Like still a lot of really good advice today. It's because there's just certain trisms and like we've as an industry developed great ways to build software.

And not everything changes with AI, it just changes how we can use it. I think feature flags are in that bucket.

Cory: 00:36:57

Yeah, I think that's really interesting because I think one of the things that's particularly interesting about using AI to code is I feel like particularly people outside of software tend to see like software's like, oh, it's very technical, you guys all must be math whizzes to write some code. And it's like, dude, our job is so socio technical.

And it's so much more, to me it's so much more about like culture and how teams fit together and literally we're collaborating on a giant code base. Like, collaboration is key. And where does that happen? It happens in a code review, right? It happens when I get a pull request.

And when you start getting pull requests, it's funny when you see some teams are using AI to approve their pull requests and other ones are using it to generate their code. This is like, oh, a bunch of bots are just doing some stuff together. Where's the trust, right? Like, a lot of DevOps is built on trust.

And that was, I think one of the keys early on is like, these teams have to trust each other to introduce these changes. Like, you have to build trust, you have to focus on culture.

Even in the Google SRE era budgets, we have trust and then we have metrics saying, hey, we're moving too quick, we have too much debt. And with AI producing so much more code, what's interesting is when you're reviewing code, a lot of times it's like, yeah, I'm looking at the code.

Yeah, some PR bot caught some linting things. We don't care about lint anyway. We don't want to complain about it.

I'm trying to focus on the really important parts, but viewing a diff, you're just getting a slice of the code, right? And I'm always expanding the diff so I can see like, okay, here's the slice that you introduced. But what was going on around this, right?

And it's still a lot of context to load. And I would love, I would, I would. I wish it was a great way to measure this because I would love to know this answer.

But when I'm looking at a pr, a lot of the time I spend on that PR is grounded in my trust of that engineer and their familiarity with the code base at that point in time, right? So if it's like, hey, I'm talking to somebody who's brand new to the team, they might have 10 years of experience in the programming language.

It's like, I'm not going to like, nitpick them on the code or whatever, but I might give them some guidance on like, hey, here's how this thing, like kind of works. Just so you know, like, why this debt's here, right?

But then when you see the guy that's been there for like 10 years, like, started the thing is like, oh, he knows that, like, I'm going to look for like some obvious stuff, but I'm not going to like grind and look through like thousands of lines of code around the three lines he put in there. And Then when it's AI, like, I mean even Sam Altman said recently that there is, there is no model. That's right. There's models that are useful. Right.

Like, like the. You cannot innately have this trust with AI. It can produce stuff, but we don't ever know that it's right.

And the one thing that sucks about software is even if it looks right, you don't know until it's in prod. Right. That's why test in prod. Like you, you don't know until it hits production. And even if you have that, we'll put it in staging.

I mean there's that age old meme of worked on my machine. Right. And I think that's one of the things that's really key is like introducing so much change so quickly.

Like even quicker than we dreamed when we saw the early Flickr video about how they're releasing 12 times a day. That's crazy. Like the amount of code that we're producing now and to just lob it into production because AI wrote it.

Some tests passed and I thought it looked, looked good without the ability to roll it back quickly. Like even if you have a quick rollback, the merging story after that quick rollback is not a great story.

A human's probably going to spend time on that and that's even going to be more cognitive load to roll that back. So I think, I really like that. I think it does become a lot more important.

I would love, man, I might actually, I might update my cursor rules to include feature lock in. Pretty much not feature, but putting feature flags around pretty much anything that AI spits out. Because I feel like it's even just easier.

Like one of the things I always saw difficult is like we leaned into it so hard at this company I worked at previous, my current company and it was like one of our big things was like, how do we clean up our flags? Like we had so many of them. It was like finding the ones that we weren't using anymore.

And it just, it feels like this is even that pain can kind of go away with AI. Like, hey, make this and then make sure to follow it up with a PR to remove it and just let that PR hang there until it's out.

Michael: 00:41:10

Yeah, we've been doing a lot of work recently on the cleanup piece. This is one of the, like we, we say that feature flags are tech deck debt and they are.

It's like normally not something you would be like, you know, theoretically, the more feature flags people have, the better it is for us. But like, like the reality is they are tech debt. They're ticking time bombs and they're incredibly valuable, just like regular debt.

But like if you don't clean them up and we're working on instrumenting and using AI to be able to clean up. Not not only like we know for instance, hey, you have this, this feature flag that's in production but that hasn't been used and in seven days.

Yeah, it's like, you know, this is a candidate for cleanup. It's like we see that something hasn't been touched in 60 days, so you should really remove this thing.

And in the process of removing in the code, it's a well structured problem that AI is theoretically good at. It's. It's not completely trivial either. But yeah, absolutely. It all comes full circle and that's a really, really important part of it.

Cory: 00:42:07

Awesome. Well, this has been a super fun conversation. Like I said at the top, like this is an area that I love. I'm super excited about what you all are doing.

Where can people check out the open source project and learn more about Unleash?

Michael: 00:42:19

So we're on GitHub, just Google, unleash, Unleash and you can also go to gitunleash IO. We've got kind of the open source project.

We have an enterprise version you want to add kind of, you know, all of the security and compliance and scale and all of those types of things so you can use it for your entire company with no worries. We've got that as well and happy to help you understand which would be right for you.

But we've got a lot of documentation and tutorials and things like that on our GitHub and our website as well.

We've also got, if you just want to try Unleash, we've got a free version that you can use or you actually don't have to spin it up yourself, which can be useful.

Cory: 00:42:54

Nice.

Michael: 00:42:54

You can find that both on our GitHub as well as on our website.

Cory: 00:42:57

Very cool, Very cool. A lot of different options to get started with feature flags there.

We'll put the links to open source and unleash links in the show notes as well as we'll put all the talks that we mentioned earlier.

So if you're hearing this and this is all exciting to you, you haven't started rolling out feature flags yet and you're or trying to look for how to advocate it to the rest of your org, definitely check out those talks from the conference and maybe search for the conference and bookmark it for next year. Michael thanks so much for coming on the show today. I really appreciate it.

Michael: 00:43:23

Yeah, this is a lot of fun. Cory, thank you.