When he worked as a product manager at Google from 2014 to 2017, Chris Stephenson was an eyewitness to Google’s huge, growing, and powerful application development environment. He watched with fascination as developers thought up new software ideas and quickly moved forward to build them for use by 100 million users.
The amazing thing that he noticed about it was that those Google developers simply concentrated on dreaming their software dreams and never had to think about the backend on which they would run and be distributed.
That was liberating for them, Stephenson told The New Stack.
“It means that developers, software engineers at Google, only really need to worry about the problem that they are trying to solve, and they do not need to solve problems about how to run software at scale,” he said. “So, what happens at Google is you come up with a product idea and you build a product to automatically target 100 million users. That is just Google scale. It is just how it operates.”
While Google did not formally call those internal development machinations Internal Development Platforms (IDPs) at the time, that is essentially what they were, he added.
These were powerful lessons for Stephenson, who in July of 2019 became the CTO for then-nascent platform engineering vendor, Humanitec. In his new post, he took those indelible impressions with him from his experiences and observations at Google.
“Fundamentally, engineers at Google just focus on the business logic and what they want,” said Stephenson. “The platform, by its collection of tuning and supporting infrastructure, means that they can focus on solving the business problem, and then the infrastructure just kind of solves the distribution problem.”
These observations led to more questions in his mind about Google’s development processes. How do you get the software out there? How do you make it scale? How do you make it resilient? How do you make it available to 100 million people without falling over?
Other technologists, of course, were also thinking about new software development methods and how traditional systems could be revamped. These technologists were also considering how large enterprises like Google were developing and distributing vast amounts of software and products, and how lessons could be learned from that work.
Much business software development in recent years has been moving to DevOps methods, which brings together software development and operations with the goals of increasing development speed, efficiency, and security. But a drawback of DevOps is that it cannot be scaled beyond about 50 developers before it begins tripping over itself and becomes difficult to manage. With that serious shortcoming, platform engineering and its strengths and flexibility was gaining more attention in the marketplace.
Emulating Google’s Processes for Enterprises
When Stephenson joined Humanitec, his observations about Google were also being discussed and imagined by other innovators in the world of technology, including platform engineering luminaries Manuel Pais, the co-author of Team Topographies, Florian Lipp, Gregor Hohpe, Ajay Chankramath, Jason Warner, and others. These discussions continued about new kinds of efficient and scalable development processes for businesses of any size. Bright minds around the world were talking about it, thinking about it, and finding ways to use it.
“How can we bring this idea of having the kind of power at Google — and incidentally also at Facebook and Netflix and other big tech companies?” said Stephenson. “They have solved the problem of making their software engineers much more effective. It was a force multiplier for the engineer, because they only have to worry about solving their problems.”
In those early days, the ideas expanded to figuring out how such development possibilities could be brought to people at any companies, including startups, mid-sized banks, HR businesses, and even companies that may not be technically advanced. The possibilities were almost endless.
The Enablers Arrive
As these issues were the talk of conferences, meetings, research papers, and coffee shop discussions, two other technologies that would arrive to drive platform engineering — cloud computing and Kubernetes — were evolving, growing, and becoming more powerful on their own. No one knew it at the time, but they would become the needed enablers that could get platform engineering off the ground for any business, said Stephenson.
“There is always the question, ‘Why does something happen at a specific point in time?’” said Stephenson. “But something caused everything to happen at the same time. Fundamentally, it was the development of cloud technologies that went from being something of a toy that people liked to talk about, to being something that has been broadly adopted. And you can line it up pretty well with the inflection point of Kubernetes.”
That is the moment when Kubernetes “switched from being a geeky toy to being something that Fortune 500 companies were using to run their production infrastructure on” starting between 2018 and 2020, he said. “Kubernetes really kicked off, and that enabled you to run cloud apps at many different scales, with consistency and standardization inside the cloud ecosystem for the first time.”
Prior to this combination, “you needed an engineering team the size of Google or Facebook or Netflix to actually support and build out in terms of a platform,” said Stephenson. “They invented all that technology to serve Google, and similar systems at Netflix and Facebook. And what happened is that with Kubernetes and this platform you could start doing this at other scales. It was, I think, an important inflection point.”
In the end, it was all these technologies coming together at once that was the key.
“This mirrors very closely with the rise of Kubernetes as a dominant compute layer within the cloud,” he said, “And that is why all of these people were talking about the same things at the same time, why we all met each other and started having these ideas bouncing off each other.”
The Evolution of Platform Engineering Continues
It has been an interesting journey so far for the platform engineering movement, Kaspar von Grunberg, the CEO of Humanitec, told The New Stack.
“It is an important thing to point out that we are in the very early ages of platform engineering,” said von Grunberg. “There are no best practices yet. Many things do not have names yet. This feels to some people like it has been around forever, but the truth is that we only know maybe 20% of the full practice.”
There is still a lot of learning to be done about the field, he said. “You should jump on the IDP train,” said von Grunberg. “Developers should join the communities and they should share their failure stories.”
Stephenson agrees.
“We are looking back and we are aware of all of the dead ends that we went down to get here,” said Stephenson. “The only ideas that you see are the ones that all worked out, right? Well, we as a community and as an organization, we are going to go down many other dead ends and we are going to backtrack if we learn the dead end did not work. We need to tell everyone else about it so that other people do not go down the same dead ends.”
The post The Birth and Continuing Evolution of Platform Engineering appeared first on The New Stack.
How tech trends involving Google, the cloud, and Kubernetes came together to inspire the creation of platform engineering.