Meeting Developers In Their Existing Workflows: The Terrateam Advantage

Welcome back to the Platform Engineering Podcast, I'm your host, Cory O'Daniel. Today we're diving into the world of Infrastructure as Code and platform tooling with someone who's spent decades at the forefront of software engineering and problem-solving in some truly fascinating domains. 

Also a buddy of mine, Malcolm Mitalka, co-founder of Terrateam. They're working on enabling teams to deliver infrastructure faster by simplifying and scaling Terraform, OpenTofu, and CDKTF workflows.

Sorry, that was a question, but it was also a statement. That was right. I got them all, right?

Terragrunt, too.  

And Terragrunt. So Malcolm's been deeply involved in the open-source community, including OpenTofu, and has 20-plus years in the software space. So Malcolm, welcome to the show!

Thank you very much. 

It's good to talk to you on a podcast, finally. I've been waiting to do this since the last time.

Am I allowed to make ad hoc announcements? 

Hell yeah. You can do whatever you want. This is your show today, bud. 

For all the listeners, I'm legally changing my name to Cory, so that this podcast can be renamed to The Two Corys, and I'm going to be on every episode going forward. 

Okay. There we go. Everybody. But are you going to spell it the right way, or are you going to spell it some bastardized way?

Well, I think we should have both spellings, so we're independent, too. 

Okay. Well, there's a lot of spellings. Depending on how often you go to a Starbucks, you wouldn't believe the way that people spell a trivially simple name like Cory. 

Don't they misspell it on purpose? 

Dude, I don't know. I got K-O-R-I the other day. I was like, what are you doing? Look at me, you can tell I'm not a K-O-R-I kind of guy. 

So I do have to say, I showed up prepared. I don't know if you showed up prepared, but I've just seen this in so many pictures and episodes, and I just wanted to take it to the next level [Malcolm laughs as Cory puts a red, white and blue sweatband around his forehead on top of his headphones] - I don't know if you can go around. I might take this off in a minute. You don't have yours, handy?

No, I don't have it within reaching grasp. But I do like your colors, because I'm in the Netherlands, and that's the flag colors. 

This is also the flag colors of a place called America, if you haven't… So for people that are listening, I just donned a ... Actually, I donned my 4th of July party sweatband, and I'm going to take it off, because it actually makes the headphones much more painful. But Malcolm, I feel like every time I see a photo … Did you wear that in the podcast with Soren? 

Yeah, I did. 

Yeah, you did?

It was an exhibition match. I had to come prepared, right? 

That's why I grabbed it. I was like, oh shit, I don't know if he's going to show up in that today. So I ran and grabbed this out of the house.  Well, okay. So that's my favorite intro ever. Awesome, man. 

Software Beginnings: Real Estate to Investment Banking to Biotech

Well, it's great to have you on the show. You have actually had a pretty awesome background. You've been in all sorts of places. I'll let you tell a little bit about it, but I'd love to hear about where you started in engineering, where you went to school, how you got into the infrastructure space. 

Yeah, I got super lucky in that first I got a computer pretty young through my dad's job. He worked at a university, so he came home with computers every now and then.

When you're a kid, you start tinkering, and then learning how to program, so on and so forth. And then the big break I had was when I was still in high school, and a friend of a friend was just looking for some cheap programming labor. And I said, sure, I'll do it. And so this is the first like real job. 

The guy who was running as a small little startup, it was just two people, and I came on as the third person, as an engineer. And they're like, “I don't know what to do, just make this thing work better.” And so I had got this complete freedom to implement it however I wanted. 

So at that point in time, my stack was Python. So it was a Python backend server - not doing web requests or anything. The big thing was we were doing websites for realty agents where you can just like populate it automatically given the database that they have access to. So I did the complete rewrite of the backend job to every night pull down all of those listings then populate our database with it, and then did a lot of like sysadmin stuff at the same time. 

So that was like really young, a really good experience. I was very lucky that both they had trusted me to do it the right way and then also I didn't screw it up. So I got rewarded as well. So that was... 

Heck yeah. I heard a little sysadmin in there, not screwing that up off the bat. Like he's… So most people don't know this, but Malcolm actually founded the idea of DevOps back then. That was...

It's true. It's true. I was a developer operations person. You know, this was late 90s.  

Woo! Okay, but you didn't stay in the real estate business long, you moved on to all sorts of other interesting things. 

Yeah. So I went to work at an investment bank for about a year and a half or so. And then 2008 hit. And I was already on my way out, but that was, you know, the last push of like, this place sucks. 

So, at that bank, I had a friend that worked there and once a week we're like, “How is this place still in business?” And then in April 2008, it wasn't. And we're like, “All right, well, that makes sense.” There you go.

So those moments where you're like, “Oh, we've figured out why, finally.” I mean, it's not necessarily great. 

This is when the global housing market collapsed. Just a little fun history jaunt there.  

So I wanted to go back to school at that point. So I decided to go back to school for biotech stuff. And I worked in Maryland for a while doing biotech work. So I didn't really like the bio side all that much. I liked the pure software side. But they needed someone who could do that.

So we built this cool VM-based system where you would download the VM, get a little web app on it. You could do specific pipelines for bio analysis. And then you could hit a button, and if you gave it your AWS credentials, we would spin up a huge AWS cluster and you could do all that ephemerally and then bring it back down to your VM. 

Which was really cool. No one was doing that at the time.

Oh, that's cool. 

But then at the same time, again, I decided I wanted to go do something else. So I moved to Sweden. And you actually come from kind of like an Erlang background, right? 

I've done a bit of Erlanging, yeah. 

Yeah, yeah. So I was doing Erlang at a company called Klarna. 

Oh, yeah. Okay. Yeah, I know Klarna. 

We probably went actually to like a lot of the same conferences… maybe.

I only go to re:Invent. I'm so bad about conferences. I go to language ones like every so often, but it's got to be like... 

Really good. 

Somebody has to be going that I want to hang out with. And they have to be like, “Hey, we're going to be in like Colorado” or whatever. I'm like, okay, I'm going. But otherwise, I'm just like, I'll get it on the minor version update. I'll figure out what happened.  

So I worked there. Basically trying to be as far away from a customer as possible, working on database infrastructure. So we did a lot of Riak. And we were also transitioning the existing system, which was all on Mnesia…

Oh, yeah. 

Over to a real database. So the whole company ran on Mnesia at the time.

Yeah. Okay. It has like a four gig table size limit, right? So this just turned into the Erlang show - sorry, everybody. 

Yeah. Well, it's on top of debts, right? 

We can't turn it into the Erlang show right now because I definitely want to. Like you just got me in my guts, man. Like Riak -  that was my foray into Erlang, the first time. It was managing a huge… I think we're on something like a lower level of Riak called like Riak Core.

Yeah. Yeah. 

There's like a lower level that we built a database on top of. And so like that was my first Erlang experience - hey, we built our own database on this thingamajig over here that you've never heard of. And I was like, this is going to be a fantastic time. And it was. It was a great time. It's a fun language. It's a fun ecosystem.

Yeah. Erlang is a is actually a great language. And really, it's about the runtime, right? That's what it's all about there. 

That BEAM, baby. The BEAM. 

Tales From The Trenches

So then you just gave up on Erlang. You're like, “I'm out. I love it, but I’ve got to walk away.” 

Exactly. It's too much. Like, I can't have such a good life. It's not allowed. 

And now you're in Haskell. [Laughs] I'm sorry. Is it Haskell? Is that what you guys are building?

Low blow. Low blow. No, we're OCaml.

 OCaml. Sorry. Oh, my gosh.

 Close enough. They hang out in the same bars. 

Awesome. 

And Erlang is like, can you guys let me in?

No, not really.

We don't really have a type system.

Oh, man. We're getting one in the Elixir community. We have a type system… if you've got 30 minutes to wait for a Dialyzer to tell you what you did wrong. Like, dude, sorry. 

A Dialyzer run that might have a false positive on it. 

Oh, my gosh. It's never wrong, but it's never clear in why you're wrong. It's just like, “hey, meatbag, punching on the keys - you're an idiot.” And you're like, “Why?” And it's like, “Well, you were born that way.” And you spend 20 minutes figuring it out, and you're like, “Damn it. I know what it is.” Oh, my gosh, the dialyzer.

Since Klarna was in Sweden and Erlang came from Sweden, actually, Klarna ended up hiring a bunch of the BEAM folks and Dialyzer folks and all that stuff. 

Oh, nice.

It was cool to work with those guys. I don't really agree with them on a lot of decisions, but it was cool to at least see those people that had done all this work throughout that time. 

Yeah. It is a rad system. 

So you're at Klarna, and then you move on. So when do you start thinking about Terrateam? When is this clicking moment of there's a problem in the infrastructure space, and I need to solve it? 

Yeah, yeah. All right. 

So one more quick stop is I went to Spotify after that for a little while. Again, working on infrastructure level stuff. And that was all like this kind of hot mess of Puppet at the time. And you would do like a pull request, and then they have like a bunch of bots in the background go do some stuff for you. So it was sort of proto GitOps, but it was really like a lot of duct tape in there. 

Yeah. 

Working there, eventually just sort of getting sick of company hopping and wanting to work for other people as well. So you start getting like the bug if you've got like a little bit of ambition… not just ambition, but kind of like vision on how you think things should work. Your manager's like, “We've got to do it this way just because.” Well, that's not the way we should do it.

 “Just cuz. Just cuz” - Tell you what, if anybody hears that, if their manager says that today, just slap them. And when they're like, “Why'd you do that?” Be like, “Just cuz, just cuz.” [laughing]

No, we don't have violence towards managers. We don't do that.

It's unacceptable. 

It's unacceptable, but oh the “Just cuz that's the way we do things here. It's how it's done. It's how it's done.”

A quick thing on the how we do things thing. When I was working at that bank, at one point, I remember being like, “Hey, we could do it a lot better if we did it this way.” And the guy was like, “That's just not how it's done on the street.” And I was like, “Aren't we trying to beat the other guys? Is that why we're here?” Like, I'm confused.

It's not the way that we do it here. How many times do I have to tell you? It's just not the way. But the way sucks. It's like, but it's “The Way”. It's got “the” in front of it. Like, what are you going to do about it? 

All right. So I can hear in this Puppet story… I can hear the threads of Terrateam starting to get weaved. Hearing about Puppets and background agents responding. I've lived this life. The chat ops life, were you a part of that?

No, thankfully no chat ops. Missed that.

That was a moment that I was like, I can't wait for this to be over.

Don't worry. It's going to come back. Sammy Altman is going to make it happen for you. 

They are - That will ruin everything. Like, not because it's going to take our jobs. It's going to create a decade of mess for us to clean up like 10 years from now. Like, we got enough debt to clean up. 

Could you spawn me a Kubernetes cluster? I'll see what you get.

Here's DCOS. It's like, I mean, yeah, but it's not what I asked for.

Yeah, yeah. All right. I guess I'll use that. Fine.

So yeah, those seeds start going. And then I left Spotify to do a little bit of consulting work. Which was nice in that you get to solve other people's problems, but also if you manage your time really well, you kind of get like a lot of free time.

Yeah. 

Depending on how much money you want to make. You kind of like set, here's how much money I want to make per year. And once you hit that point, you're like, all right, the rest of it is my time. 

Yeah. 

And that can like run you down a bit but it's also, if you get like a good cadence to it, it's not bad. 

The Creation of Terrateam

That's really where, with my co-founder, we really decided to look at making our own company. I was like, you know, I can't do this in the long run just because you're always searching for new customers. It's just not a great experience. 

I wanted something where we can build it and then get subscribers. You know, like your results aren't based on how many hours you put in. It's based on the product that you make, which is really what you want in the long run.

Yeah.

And I forget what year it was… I guess it's two years now, probably two years. Just quit that and did 100% Terrateam.

Hell yeah. And it's just the two of you, right? 

Yeah.

Bootstrapped? 

Bootstrapped, lean. 

Dude. I don't know if you know this… I don't know if you've been on LinkedIn over the holidays, I wouldn't advise it, but I accidentally did…

Is that a website?

Yeah, it's kind of a website. There's like seven funny people on it. And then just like a bunch of people that are like dying to like unironically end up on... What is that nightmare page on Reddit? LinkedIn lunatics? Like I want to get on there, but on purpose.

But I heard that 2025 is the year… the year of the bootstrapped founder.

All right. 

Now it was on LinkedIn, so it's got to be true.

That sounds about right. 

So bootstrapped though, I feel like far and few between do you meet a bootstrapped founder. But I feel like even more so, very much rarely in the DevOps and DevTooling space. 

I feel like it's fairly common if I meet somebody who's like, “Oh, we're bootstrapped.” - they're very much in the SaaS space. They're B2B, but they're selling to a marketing team or a sales team.

Yeah.

How has that been, bootstrapping a DevTool? You know there's a lot of work in it just to get it up off the ground where it meets that level that people can start to use it. How did you go about that? Not saying that it's the wrong move, but why have you guys pushed back against looking towards VC? Have you just not needed it? 

Yeah. I think part of it has been naïveté, just thinking how hard could this be? And another part of it is our general view -  I would say, is that we want to get in the workflow that our customers already have. We're not trying to bring a workflow to them. 

So we're really into GitOps, and that means a lot of our UI and complexity is taken care of for us by what our customers are already using. So in our case right now, GitHub, we're planning on adding more integrations for VCSs, but we're really focused on GitHub right now. We really just had to implement a really high-quality backend to handle those web events, run things, post it back to (in our case) a comment. We're also building out the UI so you can see large outputs… because comments suck in some cases, really. 

That simplified the problem a lot, and it also… constraints kind of force you to be creative in different ways. So our UI is basically a config file. You modify in the repo, you tell it what you want to happen, and we say, “All right, good, we're off to the races.” 

That's pretty cool. 

So it really is simple, in that case, of an experience. And that means you can go really lean and accomplish so much. Because the reality is, a lot of development on the backend, you can get… for an hour put in there, you can get a lot more functionality out of an hour put into a frontend, I would say.

Yeah. Very cool. Very cool.

The Terrateam Philosophy: Libraries vs Frameworks

One of the things you mentioned, and I think I've seen you mention this on LinkedIn before... I know I've seen it multiple places… you guys lean into this idea of libraries, not frameworks. So “Frameworks, no. Libraries, yes.” 

Keeping it simple, I'd love to hear a bit more about that and how you landed there. And how you go about deciding what is a good dependency to bring in. Because essentially, you're putting that on your customer eventually too, right? And so you're picking the right tools for the product. So I'd love to hear about the philosophy and how you got there.

So that philosophy comes from me having to fight Django a whole bunch and just being really annoyed with it not working. And you have to fight just these 20 layers of meta programming magic to figure out it's just… like you had a typo somewhere. And just that being miserable.

So probably about 10 years ago or something, I was... So I've been a big OCaml developer for a while. And for people who don't know, OCaml is a functional statically typed programming language and its general purpose. That effectively means it looks kind of like Haskell - If you're aware, comfortable with that. It has a similar type system there, but it has some big differences.

One of the biggest differences is it's more, I'll say, practical than academic. So for example, they have a really simple runtime. So you can often read some code and mentally compile it in your head and get, this is what the runtime effect will be like. Like it's really clear where a GC pause is going to enter and whatnot. 

Okay, very cool. 

Now it's a really small language, so a small community. So for things, for example, doing concurrent programming - which in Erlang world you get for free - you have to choose which framework you want to use. 

For the sake of this conversation, a framework is something where the flow of control is decided by a piece of code you did not write. 

Okay.

So you can think of… like Django is like that. 

Even deeper than like... we're not talking like MVC frameworks. We're talking like even larger… larger libraries that kind of do routing and whatnot.

Yeah. So you can think of it like… another example would be in the Rust world they have Tokio, which is a concurrency framework. So we have our own concurrency framework.

Again, that's one of those things that started 10 to 15 years ago where I thought, oh, how hard could this be? Because I wanted certain functionality that was lacking in the existing options in the OCaml world. 

Coming from Erlang, I really wanted to be able to cancel a computation if I decided I didn't need it anymore. And none of the existing concurrency options in the OCaml world let you do that. So I was like, all right, well, I really want this. And it sounds like a fun project, so I'll do that.

Then, over time, it just sort of grew and I started using it for more personal projects. Really battle testing it on things I was working on. And so by the time that making Terrateam came around, I was like, I think this is good enough for us to build a product on top of. I think it's solid and it works.

On top of that too, the rare times it does have a bug, I literally know that code end-to-end. So it's like, all right, that's where I'm wrong. I think I know exactly where that is actually.

I love this. I've seen quite a few people on Twitter talking about not taking dependencies anymore. I feel like it's gotten… You look at the left-pad Node.js fiasco a few years ago, where it's like, we just got to the point where we're just grabbing... free software, man. It's all over the place… just grab it and stick it in. 

But there are plenty of times where... I have this code that I work on where I'm like, I go look into it and I'm like, “Shit, I'm separated by a release now from fixing this.” I open a PR, wait, now I'm on the bleeding edge. But then I'm looking at the code and I'm like, I really took a dependency for 100 lines of code, right? 

But that's the rub. It's like, you can do that all day long, “Ah, it's only 100 lines.” Why take the dependency?

What goes into... Is it just totally around the flow control loss? Is that your distinguishing factor of like, okay, I'm not going to take a dependency here, I'm going to build it myself. Or are there other things you take into consideration when trying to decide, do I bring this in or not?

The flow control loss is important in the sense that if you choose to opt into a framework, it's very hard to opt out.

 Yeah.

You need to think about it in terms of risk mitigation. So that's why we say “Frameworks, no. Libraries, yes.”, because if you have a library, as long as something else matches those function calls, you're pretty good, right? You can swap it out. 

If you choose to opt into a framework, it's very hard to opt out.

Mmm-hmm.

Or you can build a little adapter there. You don't have to rewrite your entire program if you decide you don't like your time library or something like that.

Yeah.

That's why we focus on frameworks. 

With everything, of course, there's trade-offs, right? A big trade-off is that we have to implement a lot of stuff that maybe you see, “Oh, I can just get that. I can do npm install and I'm good to go.” 

The benefit for me, based on how I like to think about problems and approach problems, is that the control that we gain, the power that we gain, by inventing these things ourselves, is worth the time spent. We never spend future time trying to debug a new release or understand how some things change or being forced into doing a big refactoring because we need to get the latest version… like a security fix or something like that. 

Especially as a lean, really small team where we want to be very conscious of, here's the time that we want to spend on an engineering problem and here's the time we don't want to spend. And here's the point where we know we want to change some dependencies or something like that. For us, that's been really valuable to be able to have that control and not be forced into someone else's release cycle. 

Yeah, that is rough. 

Open Source Journey and OpenTofu

That is really cool, though. Now, one big thing has changed for you guys, though, in the past couple of months, and that is that you have open-sourced Terrateam. 

Yep, yep. 

How is that communicated to your community? Are people coming along and saying, hey, I noticed this, can I... I know that when I've open-sourced stuff in the past, people just send you crazy commits. Especially around that time of year where everybody's doing commit-a-thon, right? They're just sending you crazy shit.

 But how do you communicate that in your community? Like, hey, we prefer to build things that match this criteria versus take on dependencies. Have you had to deal with that yet in the open-source community? Like, people saying, hey, why are we writing this? Why don't we just use this library instead? 

Not really yet. So, we're so new to open-sourcing. It was really just a month or a month and a half ago that we haven't gotten... A lot of the contributions we've gotten have been more around documentation or pointing out bugs or feature requests.

We haven't gotten a lot of people actually submitting pull requests to code. I think we're also gaining some traction in the OCaml community - So, I think that's probably where it's going to happen first as people are familiar.

So, part of what I like about the OCaml community and also why it's kind of off-putting to some new developers is it is kind of like cowboy style. Where you kind of go in being willing to write a lot of your own stuff because that's just how you think. 

Yeah.

So, I think a lot of people aren't really going to be bothered by it. They might ask a question, but then it'd be like, “Hey, it's because of this.” They're like, “All right, well, that sounds fine.” 

I mean, there's already three other concurrency frameworks in the OCaml world. So, it's kind of like you're used to just understanding that, all right, this is something that there's going to be N plus one of in the future.

So, what drove you guys to open sourcing the product? You've been around for a while and recently open sourced. 

Yeah, so as you pointed out, we're a bootstrap company - small, lean. And we don't have a huge marketing budget, to say the least. And we also just consume so much open source software. 

We thought for a really long time, you know, is this… if we decide… we wanted to open source for a while, and we thought, is this going to destroy the business if we do this? Are people just going to copy it or just run it without upselling to the paid version or something like that?

So we thought for a really long time about how we wanted to break up the feature set. So we are open source, but technically we're open core. And we took three features and we said, these are the ones that we think larger organizations are the only ones that really benefit from these. And in our case, that is a centralized configuration - so a center place you can configure everything. Like all these security compliance things - RBAC is another one. And then our audit trail functionality as well.

And we think that for smaller groups, they don't need those things. They're not going to really benefit that much from them. That's kind of like where we started.

Once we agreed on this is the feature set, we thought about, all right, what's the benefit here to open sourcing? And really it's about, you can just talk to people in a different way, right? 

People kind of feel when you're trying to sell them on your thing because you want them to pay for it versus, hey, you can just use it or not use it, that's up to you.Here's what it does. If you want to make it better, please submit a pull request and that'd be great and we'd love to have you. 

And I think that is just a great way to be able to talk to people and a lot of people end up being like, “All right, well, you're actually doing a great job. I want to get the other features or I just want to find some way to help you guys out to maintain this because that's making my life a lot easier.” 

So how has open source or open sourcing your product like changed the community or community reception? Like are you seeing more inbounds from open source?

Yeah, definitely. Did you see our Reddit post about open sourcing? 

Maybe that's where I saw it. I follow you everywhere. 

Yeah, I know.

I'm just… I'm refreshing your Reddit page. I'm refreshing the LinkedIn page. I got them all.

You're so sweet. 

So we did this Reddit post - which I feel comfortable responding to Reddit posts and helping people out, but I'm still kind of uncomfortable on making a new post because you just never know, you know, which way the wind's blowing that day. 

Yes.

But at first, we had this like marketing style one where it's trying to point out all the features we do. And then we're like, you know what, this just isn't working out. So we deleted the whole thing and just wrote kind of like, you know, like a from the heart type thing - we're doing it. Here's why we're doing it.

I know in this day and age, everyone says they're going open source and you expect them to change their mind in a year, but you know, this is important to us. So we were just honest about what we were doing, why we were doing it and just said, you know, I hope everyone enjoys it. 

And our community channel has definitely gotten an influx of people through that. A lot more people installing it, trying it out, bug reports. So it's been a really… and this is like really new to over the holidays as well. So I feel like we're on a… just, it's one of those things where it's like exponential, right? Like more and more people start talking and then more people start showing up. So I'm really looking forward to a 2025 past there. 

Heck yeah. Have you started to think through like how you'll go about, and I don't know if you've already done this before with open source, but like how you go about managing… like, it's like, you have a second job at some point, right?

Yeah.

Like, like how do I manage community? How do I manage contributions? Like, have you started thinking through that or started working with that at all?

Yeah, this was actually a big discussion point before we decided to pull the trigger. And what we decided is we just don't know what the end… what is going to happen. Like, are we going to get many people trying to contribute or not? And if we do great, and we'll just be honest with people about if we can respond in a timely manner or not, to be honest. 

I think we just said, we don't know what's going to happen. We're just going to be honest at every point and hope the community appreciates that.

‍I mean, it seems like, you know, for all the organizations that are - and rightly so - like concerned about like software supply chain, right? There's not a ton of tooling in the space for managing your Infrastructure as Code that is open source. Where you can literally see… not only can you see the supply chain, but like you guys are doing the work to say like, “Hey, like we're actually going to even own some of this.”

Yeah.

I mean, I'm sure that there's points of time where you're trying to make decisions on even libraries, right? And it's like, “Oh, this is painful to like, think through, like, do we take a library or do we build it ourself?” But that's a feature.

Like in high sec, high compliance environments, like that work that you guys are doing, like skipping out on frameworks and like building your own when you need to, like that's a supply chain feature that I think that - I don't know if you guys talk about that open sourceness on the marketing - but like, that's fucking big.

No, it's huge. Like, and people really appreciate just being able to see what… you know, the sausage factory. They want to know how you're responding to those API calls, you know. And I think…

Wait. Your Reddit name is sausagefeet, right?

Yeah [laughing]. 

Hold on. Can we take a tangent as to why? And then come right back to them looking at the sausage. Or we'll come back to sausage feet. Sorry.

For some reason I knew that, but it never clicked in my mind until I heard you say sausage out loud. I was like, why is his username sausagefeet? Like that is a… ughhh. I like sausage, I don't know if I like sausage feet. 

Let's see. Let's see. So when I was 18, I went with my band's friend on a… they went on a tour and I did like a groupie thing with them all the way down…  started in Massachusetts, all the way down to Florida and back. And in North Carolina, I got so sunburned, like so sunburned, that my feet swelled up and I couldn't put on shoes. 

Oh, ouch.

So everyone started calling me sausage feet.

It could have been a band name. You could have been big. You could have been huge in the punk scene.

So to go back to like the supply chain aspect of it all, like you guys are actually like delivering a product where you don't have to chase down 8 million dependencies to figure out like what you just brought into something that is critical for a lot of orgs. 

Like, this is going to be accessing keys to provision stuff, right.? Like it's going to have the most sensitive of sensitive information. And the way you guys go about your development is in a way that makes that a bit more transparent to people. Which I think makes it a great product for the high sec, you know, compliance space. 

Yeah, absolutely. Additionally, as we are writing so much stuff, it does exactly that thing that it needs to do and no more. So we're not going to get bit by something where some other feature way over to the left, that the right's not even using now as an entry point there. Also, like I'm incredibly proud of how much code is in there and how, in my opinion, high quality it is. But that constraint also means we push off as much functionality as we can, especially around security compliance and integrate against it rather than implementing it. 

So for example, secrets. We depend on your VCS or CI/CD provider to manage those, and we just consume them as we run something. And then they're gone once that compute is gone.

It does exactly that thing that it needs to do and no more.

That mindset means like we try to use these high… I want to say high quality, but I also want to like say that I don't know how GitHub implements it. That's their problem. And we're happy that it's their problem.

Well, I mean, you also get… like in this space, getting across the trust boundary…

Yeah.

… with an operations engineer is one of the hardest things in selling. It’s like once they trust you and like, they already trust their VCS provider. 

Yeah, that's a hundred percent the reason we started doing it that way as well. So pre what we had now, an early idea was - we called it hosted Atlantis. Conceptually, we take a lot of ideas from Atlantis, but we don't take any code. We sort of implemented the idea in a way that we think is fundamentally superior. We sort of like stand on the shoulders of the Atlantis folks and take their ideas and implement it ourselves. But we needed too much stuff for hosted Atlantis to work because we were trying to just run Atlantis. 

So we decided, all right, let's go back and build out this new idea that we had. And one of the early things was we don't want to manage secrets for people. Because that was in the previous one. And we thought it's just sort of like scary to do, and also we didn't know if people were going to trust us. 

So we said, “Can we just like get rid of this problem altogether for us?” And that's where we really got into the idea of just mooching off the VCS as much as possible - VCS or CI provider. 

Yeah, no, I love that. That's great.

So in the open source space, how we met was through OpenTofu. So I would love to know… like, I mean, I know… but I would love for you to kind of share like what originally attracted you to working with OpenTofu? And how did you kind of get into the group of people that all kind of congealed together that first week? 

I don't even know how I actually got into that group. Cause that was a wild week, right? I don't know if you remember, but it started off with maybe an email chain and then a WhatsApp group and then a Slack. 

Yeah. Oh my gosh. I think I still have the WhatsApp group. That'll have to be like… we'll have to turn that into like something that somebody hangs on a wall in a museum.

Got to memorialize that somehow. 

Yeah. We'll put it in the museum of IaC tooling. [laughs] Okay. So you were on the original WhatsApp thread. 

So I think that was my co-founder that was staying in touch with people. And then that all happens. So we got pulled in to the Slack. And of course, I assume everyone who's seen this knows the background there.

I think there's a few ways to interpret everyone's motives there. Sort of the cynical one is just, you know, to stay in business you need to do this thing. And I think there is some truth to that for anyone who's there - Like that really was the way to keep on going. 

But I think the other way is that the reality is Terraform success was built upon the work of people outside of HashiCorp. And that was just not cool, man. Not cool. 

I was in the “it wasn't cool” camp either. I was just like, I'm here purely out of spite [laughs].

It's also, I mean, you can't be like a programmer at this, in this day and age and just not be building your life around open source software, right? It's just not possible. 

Yeah.

This idea of, you know, the Microsoft programmer back in the day who was paid for every thing that they put into that executable they've compiled just hasn't existed for like two decades, right?

You can't be a programmer in this day and age and just not be building your life around open source software.

‍Yeah. I mean, there's always something… whether it's, you know, a higher level library or framework or a tool, right? Like there's so much that… we have so many responsibilities now as developers, right?

I mean, like, even if you rewind back to the nineties, early two thousands and like the LAMP stack era, it's like, we didn't have many tools, but like, we're still doing a ton. Right?

Yeah.

We're gluing everything together. 

To me, it was just like, this thing has kind of become the pervasive ubiquitous tool that we use to talk to the cloud. And like, I get it, I understand the business, you’ve got to do what you got to do to keep your business afloat. If it makes sense, go for it. 

But to me, I was just like, dude, this is… like, I have convinced organizations to use this tool, as a consultant and as an employee. Like, that's my contribution - I helped make you guys money. And like, I feel like that's the best contribution you can get, right? I built some modules. People can use them or not use them. Like I done a lot of like marketing for you in the space. Like, I'm sorry I didn't never add like a single line of Go code.

That they would have said no to anyways.

They would have said no to it. They would have said no. I mean, there's reasons that they would say no, but me in particular, they'd be like, “Your Go code is dog shit, Cory.” And I'd be like, “Yes, yes, it is… but it works.” 

That is the valid reason to close this. 

It works and I showed up with a test. And they'll be like, “It looks like Ruby though.” And I'm like, “Yes, yes, it does.” Oh man.

But anyway, to go back to that week though, like, I think what was really amazing about that is, you know, a lot of frenemies getting together, but still having a common cause that they all legitimately believed in. Maybe not like… again, it's what proportions you want to give to these two motives. But I think most of them - way over 50% - just thought that was just not a good thing to do and it wasn't good for the ecosystem at all.

I mean, I… so me personally for Console and Vault and Nomad, I am less morally outraged at those just because they aren't ecosystems in the way Terraform is.

‍Yeah. Yeah. They definitely are not. I mean, they're important tools, but I mean, I've operated big systems at plenty of companies and have never had to touch Vault or Console. There's companies that I have used that and there's others where it's just not.

But like when it comes to IaC, it's just like… man, this is… I mean, I know Terraform itself isn't a language, but like it is the syntax in which the majority of us are operating, right? I mean, Ansible is another huge, big player in the space. Yes. But for managing my databases and networks and whatnot, like this is the thing that people tend to reach to. 

Yeah.

The other thing that really got me was the amount of people that were uncertain as to what they should do. Whether or not they were directly impacted by the licensing or not. There was so much ambiguity that I was just like, this was ham fisted. Did you go ride along with them Sausage Feet? There's some hammy ass fists out here, some lawyer ham fists. Right? And so that, that was what really drove it for me.

But so, you know, as far as like adoption goes… so you are a product that has Terra in the name… when are you guys rebranding to Tofuteam? 

Zero chance ever. 

I'm sitting here hoping that we really… with the CNCF… we rebrand OpenTofu to any other name.

Please, please, please. 

There was a list of 200 names, people. And this was… this was the one that we ended up with.

Do not let the internet choose your names. 

Is that what happened?

Yeah.

Community Building and Education

Oh, ehh. But so what has that been like in your community? Like, are you seeing… were people in your community like concerned about this? Or were they like, “Oh, the license change is outside my scope. It doesn't matter.” Like, were you seeing people concerned about it?

Have you seen people starting to switch? Which I think is… our current biggest concern in the OpenTofu space is that adoption, right? 

Definitely in the Terraform world or Infrastructure as Code world, I think people are relatively slow to adopt new things and that includes new versions of the same product. So for us, it's really been in the last four months or so we're getting people talking to us saying, “We want to move off 157 and we want to keep on using you. What's that look like?” Or just asking about OpenTofu in general. 

And I think one thing that is kind of a bummer is there's not that consulting company where you can say, if you have questions, talk to these guys and they'll help you through your migration. They'll explain to you what your options are, what the things to watch out for. 

I think we need to do a lot better job in that community. We're trying to put something together to just… information to make people feel better about that transition. Because it's really like a no-brainer transition, right? You just change the command you use and it works. 

We switched a few hundred companies by changing a single image name for them. And it was just like, nobody noticed. I mean, not that they didn’t notice, they knew it was happening, but like there was, there was no… I mean, it's a lot of the same code up until 1.6 or whatever.

It just works. 

Yeah. Just works. 

We definitely have a few customers that are the pioneers and they switched over almost immediately and just never looked back. Just this works fine. I have no problems. The end.

I think one of the things that makes it pretty comfortable for many people I speak to - I do think your consulting point is very valid. I want to come back to that in just a second - is the fact that like the code isn't changing. 

This is a huge win, but it's also a pain point.

‍I think this is a huge win, but it's also a pain point. And that is the amount of people I talk to that are like, “Oh, we thought when the fork happened, that the syntax got all wishy-washy and changed a bunch.” And it's like, no, literally everything just… you just change the binary name for the most part and you're good to go. 

Like you got to do an import or whatever… not an import. What is it?

 An init again, right?

That's it. An init -upgrade to pull the right modules down, but like, you're good to go. You're off to the races. And like, as long as you're not encrypting the state file - state file is exactly the same too. 

Yeah.

So it's like a no effort migration. And so I've suggested to a lot of people just tossing it an extra plan in their pipeline. Just so they can see both plans side by side and start to get comfortable with it on like a non-trivial set of infrastructure.

We've definitely seen a lot of people that… so I don't know where this is coming from exactly, whether it's just a poor communication thing, or somebody is trying to poison the well a little bit… but they think like, “Oh, I thought the providers didn't work in Tofu.”

Oooh.

And it's like, no, that all works. Like a lot of these things that just are flatly untrue. Luckily it's easy for us to disprove. And you're like, “No, just look here. Everything's there. It's everything.”

Definitely there's an information distribution challenge for the Tofu community. 

Yeah. And I think the consulting stuff is key, right? I mean, you know, a lot of organizations today still don't have like well-staffed operations teams. I'll leave it at that.

Yeah.

There's a ton of companies that today tap infrastructure consulting organizations. You've been in this space. I've been in this space. It's a very lucrative space. But the reality is like, we're making software developers with no cloud experience at a pace in which we are not making people that have production and cloud experience.

Yeah. 

And so like, you have to go and find these sources and a lot of orgs struggle to get that extra headcount. So it's like, these people are underwater. How do we get them to these really great practices? 

It's like, well, they're underwater. You don't want to hire more people. They start tapping consultants. And if consultants come in and go, “Oh, Terraform.”

Right. 

They're going to deploy… they're going to use Terraform. You know why? Because they just paid an expert…

To tell them to do that. 

… and this is, and this is what they showed up with.

 I mean, I think there's some really great companies like Masterpoint.

Yeah.

I don't know if you're familiar with them. They're in the space. They're doing the Lord's work on adoption and talking about OpenTofu as an alternative. But I think that is… one of the key things will be like consulting agencies. Like that's unfortunately one of the places that we're going to have to see this start coming from to get companies in on it. 

I think one of the things that's kind of catch 22 with that though is it's scary as a consultant.‍

Yeah. 

You know, to say like, “Hey, I chain my business opinion…” - the professional - “to this library.” So it's scary for them too, to recommend it.  Like nobody ever got not hired again for picking IBM.

Exactly. 

But you guys are doing really awesome stuff with your workshops. 

Yeah. I mean, that's… like I said, you know, we wanted to contribute. We're a small team as well. Like we don't have the money to put a full-time role on OpenTofu. So we're like, okay, we just need to get more content out there. 

Yeah.

So, you know, I think we've done… shit… I think 14 or 15 OpenTofu specific workshops. And it's great. Like, I mean, you know, it's not necessarily something that's a… some of them are definitely like… create pipeline for us as a company. That's not our goal. 

And it's funny when you're talking about Reddit earlier - like we hesitated to put our workshop on Reddit. Because I was like, I don't want to sound like the CEO of a DevTool company coming and annoying a Reddit group or subreddit… I know how fucking obnoxious that is. 

Yeah. 

And so we almost didn't put it up there. And then we put it up on, I think, the DevOps one. I don't think we put it on the Terraform one because I didn't want to piss anybody off. We just put it on the DevOps one. A bunch of people talk shit. Like, “Ah, you fucking shill.”

Of course.

I knew… fuck it, I knew it. But then like 400 people registered for it.

Oh, nice. 

And I was like, holy shit. But that's important. And this was the one that was like getting started. It was crazy.

We're on like the third - it was a 10-week session. We did one hour a week for 10 weeks - We got to like the third session and people are like, “Wait, this is literally, it's exactly the same.” And it's like, “Yeah, yeah.” And we're going to talk about practices and other… because it was a zero IaC to OpenTofu expert kind of situation. 

But people were signing up that knew Terraform because they were like, “Oh, I thought it was so different.” And they were at three and they're like, “Wait, is anything different?” And we're like, “No.”

No.

For the most part, no. I mean, we're going to get to some stuff that is, but like… and they're like, “Holy shit. Like we had no idea that it was a literal fork.” 

And it's like, that's what everyone was mad about. Like what part did you miss there? I don't know.

I don't know how we're doing such a bad job talking about this - but it's a fork. We just found and replaced a bunch of words and then are building our own features now.

Is that specific to the MassDriver community? Or is that something where like I could hand that over to someone who's got questions and say here's a workshop on how to accomplish this with Tofu. 

We try to keep as much Massdriver… so the 10 week is… I don't think Massdriver is mentioned until like the 10th. 

Okay. ‍

The 10th episode. And it's just when we're getting into like the different ways of like running it. So it's like, “Oh, you can run it in Atlantis… I've seen this, that, that… Like this is one way…” but it's like five minutes and we're off.

And then we have a couple of other ones, like more deep dive. So it's like… the importing existing infrastructure one. So these are more like very specific, like, hey, here's how to do this complicated thing in OpenTofu.

But these are also resources that aren't… like, I went around, I was like, what isn't there great resources on in the Terraform space? And that's how I got the idea for the importing one. Every blog you read is like, I run Terraform import. And it's like, okay, but it generates the shittiest code.

And for somebody who's never written IaC before, you're like, “Oh, it generated code for me. I'm good.” And it's like, “No! No, you are nowhere close to good. You're 5% of the way there.” Like you have a lot of refactoring to do to make this thing be able to produce multiple environments. You literally have code to manage that specific resource.

So, you know, we wanted to be able to do stuff that like would not just benefit the OpenTofu community, but also like, hey, if you're in the Terraform space and you're trying to figure out how do I get all my existing cloud stuff into Terraform and generate some Terraform code -  Like I want them to benefit too.

Yeah.

That was just kind of ours… it's like, where can we do stuff? Like we can't throw out, you know, $50,000 to sponsor some stuff here or whatever. We can't hire a team member, then just give it away to an open source project. So we were like, okay, well, if it's just costing us like four hours of our time a week to put together a workshop, like why not?

Yeah. Yeah. And are all those on YouTube?

Yeah, we put them all on YouTube. 

Nice.

‍Yeah. We're trying to think which ones we want to do next, but… I mean, I think that's key. And like a lot of that stuff also is like… it sucks because like so much stuff changes with software all the time. 

But one thing that I think Mitchell did from the get go with Terraform was -  they were so diligent about like changes and backwards compatibility. I mean, like there was a couple of points in time where like things were rough, but even the CLI tool had stuff to like help you refactor and whatnot. 

And those were really like the zero dot days. 

Yeah. But I feel like, you know, if you were to go watch a Ruby on Rails 3.0 workshop, none of it's going to work…

Yeah.

… in set version seven plus or whatever, right? But like the Terraform resources and OpenTofu ones that we do, like they tend to be evergreen for like quite a few minor versions.

Yeah.

Probably for the foreseeable future and major versions potentially. 

But I feel like that's one of those things that's like so hard on the software side versus the Ops side. This is one of those places where like the community around this tool have done a really good job with backwards compatibility. We can have a pretty good idea of how this thing's going to work for a while. 

And like you have evergreen content that works, but people still look for it. We see stuff like, “Oh, how do you do this? Like now though? In 1.9”

Exact same way.

It's like the exact same way. 

The community around this tool have done a really good job with backwards compatibility.

You know, we talked a bit about how Terraform is where it is now, thanks to the community or, you know, in a large chunk thanks to the community… and one of the benefits there goes to exactly what you just said, where the provider protocol can't really change that much over time. Because you can't get that many people to rewrite their providers because there's so many out there.

And I think we really benefit from that in that that means there's no strong incentive to change the code you write very much because you're not going to be able to benefit from that that much in the first place. 

Advice on Entering the DevTools Space

Well, I know that we're coming up on time. I got a couple more questions for you.

First, what I'd love to know, like… it's just like, again, you're a founder in the DevTools space. You have bootstrapped your business. You've open sourced a previously closed source project. You're involved in a bigger open source project. Like, what advice do you have for builders that are sitting at home, they've got this idea in their head of a DevTool or infrastructure tool… What advice would you give them getting into this space? 

One, don't do too much research. Unless you feel motivated when you do a lot of research. That's kind of a joke, but I think on one hand, it's, you know, the naïveté of just, “I think this is the right thing to do. I'm going to build it and see what happens.” Because a lot of times you can either talk yourself out of it or you don't know what it's going to be like after 20 iterations and you get some player that people look at and they say, “All right, well, that makes a lot of sense. That solves a problem I actually have.” So I think partially, you know, just do it.

But also post on LinkedIn a lot because that'll give you a lot of high quality feedback. 

I'm going to need you to clarify that one. 

Malcolm does great shitposts on LinkedIn. So I can't… I can't tell. He kept an extremely straight face when he said that. I can't tell if he's being serious because he shitposts, but he also posts thoughtful stuff about the space sometimes.

So I can't tell if you actually get good feedback from LinkedIn or not… or if that was a joke. But I'm sure the listeners would love to know. 

No, that was an absolute joke as well. 

But depending on what you want to do, there's actually a lot of really good community support out there. Specifically, if you do want to be involved in the OpenTofu world, there's the community channel. And not only is that dedicated to doing OpenTofu work, but also there's a sub channel in there called Tofu Utils, I think, which is really just any thing in that space that you want to contribute. And I think that has a pretty high concentration of people worth talking to and bouncing ideas off of.

Really bounce a lot of ideas off people unless you have this absolute clarity and vision of what you're interested in doing. 

Awesome. And then this is a new question that just popped into my head - Do you have some Ninja Turtle stuff going on in that room? I just saw the door open and there was like a neon green glow coming out of it. Is there ectoplasm in there?

Well, it's called the Ooze, and this is a very strong Teenage Mutant Ninja Turtles household. 

Okay. And then last question. So a lot of platform engineers, DevOps, operations engineers, whatever people are calling them these days, listen to the show. One of the things you said at the beginning was Terrateam likes to meet people where they are in their workflows. So what I would love for you to do is just say, for the people listening that are in the market to either build or buy something for orchestrating their infrastructure, who is the right user for Terrateam? And who's listening that should definitely come… besides everybody should go check out Terrateam… but who do you think is the right fit for you guys? 

So if you are a user that is very comfortable inside of a sort of standard development lifecycle flow, doing pull requests, all that, and you really want to use your infrastructure as the launchpad to do your more complicated application work, and you want just a very simple, “I do this change, it goes out.”, and you're not really trying to complicate how your infrastructure works. You just want it to work, the end. And like this invisible flow that's just there in your pull requests. That is the person that we definitely see has the most success. They just set it up. It works.

We have a lot of people that have set up Terrateam and just given it, and like their co-workers don't even realize that they're using a product. They just sort of get the messages in their pull requests. They're like, “Oh, I guess I'm supposed to apply now. Great. Done.” 

Awesome.

Well, Malcolm, thanks so much for coming on the show. Everybody check out Terrateam. I'll put Malcolm's LinkedIn - which you’ve definitely got to follow him on there. You need a L.O.L. in your life. Not a lull like L-U-L-L, lull like L-O-L. Sorry. I did just pronounce it, that was weird. I'll put his LinkedIn in the show notes, but check it out. Terrateam dot...

I-O. 

I-O. 

I-O.

I-O. Awesome. Thanks so much for coming on the show, Malcolm, and I'm sure I'll see you very shortly.

Thank you very much for having me. It's been great.

‍

Important Links:

• Malcolm Matalka - LinkedIn
• Malcolm Matalka - Reddit
• Terrateam
• Terrateam - LinkedIn
• Erlang
• Riak
• Mnesia
• OCaml
• Puppet
• Tokio
• Terrateam’s Reddit Post about open sourcing
• Atlantis
• OpenTofu
• Masterpoint Consulting
• OpenTofu Foundations - 10 part workshop by Massdriver

‍

‍