This November we hosted our first DevOps Leaders Roundtable event. Gathering together 14 of the best DevOps minds and managers from multiple different companies (from banking to fintech, from gambling to sports, from start-ups to consultancy’s). The aim was to provide a platform for them to discuss freely their views surrounding the two hot topics Kubernetes and Cloud security. We gave them an opportunity to dive deep into the technical challenges they face daily with their teams.
Grant Smith, Ex-Director of SRE at Just Eat and Author of Next Gen DevOps, kicked-off the evening by talking about the purpose of the meetup and why it’s important for peer groups in tech to share their experiences. Saying “Leadership is lonely and is an often repeated cliche. I think that cliche is particularly true for those of us operating in and around the SRE space”.
Grant continued “I’ve been in this very situation throughout my career and when the meet-up scene took off I really hoped I’d find other people like me. What I found was a bunch of engineers sharing their successes and a bunch of recruiters looking to pick a juicy engineer out from the crowd. I didn’t find managers, directors or heads of engineering. These people seem to choose to go to reinvent or maybe velocity, but these are strictly one-many foru
ms. It didn’t help me.”
Concluding this, Grant said, “So, when Adam asked me what sort of event we wanted, we described this roundtable style meetup. I wanted to gather a small group of my peers who can share their experiences and better still share their questions and concerns. This is somewhere that I think we can all learn something useful.”
The evening was also chaired by Chris Livermore, Head of DevOps at Hive Centrica and he led us into the first topic of the evening.
Kubernetes
Choosing the right approach for your organisation: self-hosted clusters or a PAAS implementation (GKE, EKS, AKS, etc.)
This was the first topic of the evening and sparked a lot of interesting conversations, providing insights into how a container strategy may be more interesting than pure Kubernetes within certain companies. Some of the key insights that came out of the initial topic were:
• Using AWS ECS and Fargate rather than Kubernetes for certain workloads where speed and agility is key without needing the skills or having the time to stand up Kubernetes
• The importance of high performing teams and how this should be a focus rather than worrying about which tech to use
• Why businesses should refrain from creating DevOps ‘teams’ – DevOps should be something that is across the organisation rather than one team
There were also discussions around how businesses need to have more understanding of DevOps, the importance of understanding the business you work in and deploying based on the business needs.
Finally, the group also discussed the struggles and challenges of DevOps candidates during the interview process. There was a common theme among the leaders that candidates are generally given very specific (usually business specific) issues to fix, rather than a focus on the actual skills and experience necessary.
Kubernetes part 2
How to get a Kubernetes project established and how to set-up a team for a successful implementation
For the second topic on Kubernetes our speakers, Grant and Chris, wanted to discuss how individuals have gone about getting a Kubernetes project established and how they’ve previously set up a team to ensure successful implementation. Some of the topics the group covered, included:
• How to build a team that both provides support and is capable of developing the system – SRE vs DevOps or SRE and DevOps?
• How to manage vendors so that they are positioned to bring value to your technology projects
• How to use technology roadmaps to drive innovation strategy
• How to create and recreate dynamic capabilities within an organisation
• How to recruit for the best fit in DevOps/DevSecOps teams – this is always an issue
To finish the topic of Kubernetes, our leaders discussed how they carry our platform build projects in an Agile way. The group discussed how they each approached this, including running the platform build as Waterfall with Agile feature sprints on top.
Kubernetes highlights
These were the key takeaways from the discussion about Kubernetes:
• Abstraction has been a bad thing for developers – Kubernetes exposes this underlying complexity in a way that helps educate them and unblocks them. Counterpoint is a very mature Kubernetes implementation with significant abstraction and helps developers make progress without needing deep level of knowledge of infrastructure and security.
• Under Resourced teams are one of the root causes of the struggling with Kubernetes management. Google claim 10% of their engineers are SRE and this might be a guide as to what’s needed to run a sophisticated system.
• Abstraction wrappers need maintenance, when people leave, they create a lot of pain. Kubernetes also adds value but requires huge engineering effort. Often companies could have used something like ECS but that isn’t ‘cool’.
• Running Kubernetes in prod can be easy but only if there is significant governance of the workload. Also a willingness from Developers to use the system ‘by the book’ the more customisation of the system takes place, the greater the maintenance burden is.
Cloud Security
How to design and implement a coherent security model in the cloud and how to ensure all the security stakeholders have appropriate access, responsibility and accountability.
Finally, the group covered the hot topic of Cloud Security. Grant and Chris wanted to talk about this as it’s a must for all tech teams. There were conversations around automated controls, for risk and compliance. Also identity and access management, and zero trust security / BeyondCorp.
Below are the highlights from the discussions on Cloud Security:
• Cloud Security is broken – testing for security is like unit testing, you write tests that you know will passStrong governance auto deny, IAM keys banned by default, but IAM roles can be retrieved and accessed
• Humans are the weakest link – it’s difficult to prevent a human accidentally exposing private keys
• The pressure release valve is a sandbox environment that isn’t connected to the main data network which allows much more freedom
• Use policies to create governance that surround the configuration of network components. This way developers are free within the policy but the policy is the ultimate arbiter of access
• Start ups do have to care about PCI and FCA and so governance is required
• Zero trust model where device/browser is scored and access is granted in a phased approach – there isn’t much of a commercial response currently which means it needs to be built from scratch
• Vault is a great solution to automating management of keys, rotating, temporary use, finite use keys but then the concern is access to vault
Thank you
The roundtable event was a great success and we will be looking to do many more moving forward. We were able to provide an evening of fantastic insights, networking and the opportunity for these 14 DevOps leaders to learn from one another about the current trends in tech. We’d like to thank everyone who attended, with special thanks going to our fantastic hosts Grant Smith and Chris Livermore.
If you’re interested in speaking at our next event or even attending one, please don’t hesitate to contact Third Republic or even the host Adam Elliott-Smith directly.
This is just the beginning of something special within the Senior DevOps Market!
