Before diving into this post, I want to clarify my specific beliefs regarding sports betting and prediction markets. In a vacuum, I have no moral qualms with the principle of gambling. I’ve certainly laid down some sports bets while in Las Vegas. That said, I do not play daily fantasy sports (DFS) or engage in any sort of online betting or prediction markets. Mirroring many of my thoughts regarding social media, I personally feel that unfettered online access to the dopamine hits of gambling is deeply unhealthy for the human psyche. In addition, I believe that the potential for sudden personal wealth via highly speculative risks (much like cryptocurrencies) attracts so many willing participants due to the rapidly vanishing middle class and the increasing scarcity of quality job opportunities. As the already massive global wealth inequality continues to widen, I view these as yet another transfer of capital from the middle and lower classes back to corporations and executives. As a diehard sports fan, I am also profoundly disappointed at how intertwined sports media and betting have become. People today simply cannot watch any sports coverage without being bombarded with online betting ads. In fact, I fear that new sports fans are now unable to distinguish between betting and the sports themselves. Kids and teenagers today only know betting to be woven into the very fabric of sports.

Then came the prediction markets. Let’s start with the definition from Wikipedia:

Prediction markets, also known as betting markets, information markets, decision markets, idea futures, or event derivatives, are open markets that enable the prediction of specific outcomes using financial incentives (gambling on real-world events). They are exchange-traded markets established for trading bets in the outcome of various events.

Think about why sports are so often the target of betting. It’s a controlled environment with clearly established rules. Each player and coach has their own incentives to perform to the best of their ability, regardless of circumstances. The leagues themselves have vested interests in ensuring that the games are refereed and scored fairly in order to maintain public trust that these are competitions rather than scripted entertainment. Even with these guardrails and incentives, there have been several cases recently of players engaging in game manipulation across the NCAA, NBA, and MLB. Now, throw out virtually any guardrails. Add in the fact that folks can now bet on damn near anything.

On January 3rd, 2026, the United States illegally invaded Venezuela after committing various war crimes throughout the region in the months prior. The target of Operation Absolute Resolve (other than Venezuela’s oil) was Nicolás Maduro, the leader at the time. While this news sent shockwaves around the world, a story about prediction markets may have been lost in the mix for some. Someone, almost certainly with inside information, created a Polymarket account in order to bet on real-world events. Specifically, they bet on Maduro leaving office, which had odds of approximately 7% at the time. The user pocketed over $436,000. Real-world insider trading with classified military intelligence.

The Common Person Will Not Win​

While this is by far the most public example, it shows just how rigged the system is. Polymarket and Kalshi, two of the largest players in the predictive market space, are not who you are betting against. Rather than a traditional betting model where the customer bets against the house, these two simply facilitate the bets and take a cut of each. As such, they have no incentive to prevent insider trading unless it decreases trade volume (i.e., their revenue). In fact, Shayne Coplan (Polymarket CEO) said the following in a 60 Minutes interview:

I think that people going and having an edge to the market is a good thing. Obviously, you need to curate them, and you need to be really clear and stringent on where the line is drawn...

Aside from this being virtually impossible to enforce, this clearly spells out the game. If you have insider information about world events, you are almost certainly wealthy already. Generally speaking, the average person is not high enough in their company's org chart to obtain sufficient information to commit insider trading. This leaves two winners and one loser in the predictive market game. Prediction market companies derive revenue from trades with no risk to themselves. The insiders have a significantly higher chance of winning than outsiders. This leaves the common person to foot the bill. Their money is then funneled to the elite and corporations. Game over, sucker.

If you’re hoping for regulatory oversight, you will likely be disappointed. Donald Trump Jr. is a paid advisor for both Polymarket and Kalshi. Ignoring the obvious conflicts of interest (as well as the other opportunities that the Trump family has capitalized on to profit at least $1.4 billion from the presidency), the Republican administration has yet to show any initiative in actually regulating these industries.

The Impossibility of Avoiding Gambling in Modern Life​

With all this context, you might think that one could just avoid sports betting and gambling markets altogether. Believe me, I’ve tried. Even outside the digital realm, I walk past a giant Polymarket ad every single day as I walk around my neighborhood in downtown Los Angeles.

While walking past this poster one day, I thought about all the Twitter accounts in my feed that had suddenly become affiliates of Kalshi or Polymarket, noted by the respective company logo next to their username. I still haven't been able to figure out exactly what being an affiliate means or what benefits prediction market companies offer to these accounts. I even reached out to a few via DM for more information, but none responded. It’s no secret that I’m a Los Angeles Lakers fan, so here’s an example of an account that I do not follow but often appears in my feed:

A simple idea came to mind. What if I could measure just how much of my feed are affiliates of gambling companies? The experiment begins.

The Plan and Implementation​

• Since Elon Musk ruined Twitter, I’ve noticed that API call limits are much stricter
• I have no interest in paying money to Twitter, so capturing data client-side (i.e., via the web browser) made a lot of sense
• Blocking accounts would impact my algorithm
• However, simply hiding this type of content via CSS would allow me to observe how often it appears in my feed in a controlled way, without having to physically see the actual content
• I had never taken the time to build a Chrome extension, so it seemed like some fun JavaScript hacking

The first steps were to parse the HTML from Twitter and figure out a clever way to determine if an account is an affiliate of one of the gambling companies. Surprisingly, this took a bit of research. One might think that there is some obvious text indicating the affiliate company. The best way that I found was to parse the URL for the affiliate image (i.e., the little picture with Kalshi or whichever gambling company). I noticed that the URL contained the gambling affiliate's Snowflake ID. After reviewing several of the most prominent affiliations, I built a simple mapping.

// far from a complete list
// these are only the ones that I detected in my feed
const BLOCKED_AFFILIATE_BADGE_IDS = {
  '1998108505408999424': 'Kalshi',
  '2000660110239023106': 'Kalshi',
  '1994986527341113344': 'Polymarket',
  '2005664281002491904': 'Polymarket',
  '1961093049913851904': 'ProphetX',
  '1954932272378925056': 'FanDuel',
  '1996078471097761792': 'Onyx',
  '2007116898677243904': 'Underdog',
};

With this mapping, each tweet can be easily categorized by parsing the HTML using JavaScript.

function processTweets() {
  const tweets = document.querySelectorAll('article[data-testid="tweet"]');
  tweets.forEach((tweet) => {
    const container = tweet.parentElement;
    if (!container) return;
    if (container.style.display === 'none') return;
    if (!tweet.dataset.scanned) {
      tweet.dataset.scanned = 'true';
      // function to count total Tweets scanned
      incrementScannedCount();
    }
    const affiliate = getAffiliateType(tweet);
    if (affiliate) {
      const userArea = tweet.querySelector('[data-testid="User-Name"]');
      const nameLink = userArea ? userArea.querySelector('a') : null;
      const username = nameLink ? nameLink.textContent : 'unknown';
      const tweetText = tweet.querySelector('[data-testid="tweetText"]');
      const body = tweetText ? tweetText.textContent.substring(0, 100) : 'no text';
      container.style.display = 'none';
      if (!tweet.dataset.blocked) {
        tweet.dataset.blocked = 'true';
        // track stats in local storage
        updateBlockedStats(username, affiliate);
      }
    }
  });
}

There’s a bit more code for the UI and storage logic, but you can find the complete repository here. I rarely write JavaScript these days, so I’d hardly consider this my software magnum opus. There’s probably some edge cases remaining, and I slapped this together over some coffee on a Saturday morning. Pull requests are welcome. Here's the extension in action:

The Results​

As mentioned above, I intentionally do not follow any account that is an affiliate of Polymarket, Kalshi, or any gambling company. Given the opacity of any recommendation algorithm, I can’t explain or reproduce exactly how or why certain accounts appear in my feed. This was far from a perfect experiment, but the directional results were still surprising. Below is the breakdown (with $n=2200$ tweets):

Of the affiliates, Kalshi and Polymarket had the lion's share of influence on my feed:

Obviously, everyone’s feed is very different. Mine centers on sports and politics, which now seem more geared toward gambling, unfortunately. In addition, many of these accounts tweet frequently, especially during live games, trade rumors, and major world news. For example, during testing today, several of these accounts tweeted about the impending NBA trade deadline, Austin Reaves’ injury, the Lakers/Knicks game on national television, and the Grammys. Even with all the caveats, there is no denying the web of influence that these companies have in social media. While it is certainly not the case that these accounts are constantly spewing gambling ads, there seems to be something much larger at play as these companies continue to prey on people through constant marketing, addictive products, and promising the hope of making money.

Try It Yourself​

As mentioned, this code is available under MIT on GitHub. At this time, I have no plans to publish in the Chrome Web Store due to my unfamiliarity with the rules and requirements. That said, it’s straightforward to install extensions manually:

• Navigate to chrome://extensions/
• Enable developer mode
• Click the button to load the unpacked extension
• Select the directory where you cloned the GitHub repository (specifically where the manifest.json file resides)
• Go to Twitter
• Scroll through some content
• Open the extension to view the UI

Disclaimer​

The opinions and views expressed here are my own and not those of my employer or school.

I am living proof of the American Dream, though I fear that my story is becoming less and less common. I was born and raised in Victoria, Texas, a city of 65,534 as of the 2020 census. In this blue-collar town, I grew up in a working-class home. My father is one of 14 children to immigrant parents and spent the vast majority of his life working in the brutal oilfield elements to provide for his family. My mother, also the child of an immigrant, worked as a substitute teacher. They often reminded me to pursue what my father called an 'office job,' which, in hindsight, represents socioeconomic betterment. Although I have now worked in the technology industry for 13 years, being a software engineer wasn't a vocation that I even knew existed when I was growing up. I was fortunate enough to take a Java programming class in 2004, which radically altered the path of my life. So much so that I spent four years volunteering by teaching AP computer science at various high schools in Houston from 2018 to 2022 to pay this forward. Education is core to my beliefs and identity, and in many ways, that started at the University of Texas at Austin.

DEI is core to the American Dream. Socioeconomic and racial imbalances must be countered for the American Dream to be attainable for everyone. I likely would not have been accepted to the university without the Top 10% Rule (Texas House Bill 588). While admission alone did not provide equity in the university experience, it allowed me to demonstrate that I belonged there, despite traditional socioeconomic barriers to higher education, such as the SAT, which naturally favors high-income families with access to private tutors. The Trump Administration states the following in its Compact for Academic Excellence in Higher Education proposal:

"Treating certain groups as categorically incapable of performing — and therefore in need of preferential treatment — perpetuates a dangerous badge of inferiority, destroys confidence, and does nothing to identify or solve the most pressing challenges for aspiring young people... Institutions shall have all undergraduate applicants take a widely-used standardized test (i.e., SAT, ACT, or CLT) or program-specific measures of accomplishment in the case of music, art, and other specialized programs of study."

I write to implore the University of Texas at Austin to reject this Compact. Not only would joining effectively end the opportunity for folks of a variety of ethnic and socioeconomic backgrounds to access a premier education, but it would also rapidly advance the intellectual exodus fueled by authoritarian-like government overreach and rampant American xenophobia. I do agree that "American higher education is the envy of the world." This Compact would tarnish that brilliant reputation through financial coercion. As a proud Texas alumnus and current graduate student at the University of Southern California, I urge the University of Texas to refuse to bend the knee to the Trump Administration, just like the five other schools (MIT, Brown, Penn, Virginia, and USC) have at the time of writing.

Ye Shall Know the Truth and the Truth Shall Set You Free​

This inscription sits above the entrance to the Main Building on campus. The truth is that tightly coupling funding with ideological requirements undermines the curiosity and independence that made American universities what they are. This Faustian deal is another step towards the suppression of free thought. It's a mechanism designed to deter critics of an administration that clearly aspires to be authoritarian. It also perpetuates a false narrative that conservative voices do not exist on college campuses. During my time as an undergraduate student, I recall a strong presence of Republican voices, including the Young Conservatives of Texas, which notably planned a 'Catch an Illegal Immigrant Game' in 2013. Fisher v. University of Texas also sparked conservative voices and debates around the same time. The irony is that this Compact aims to suppress DEI admissions based on socioeconomic factors while at the same time using financial leverage to artificially amplify conservative ideals that the Trump Administration perceives to be underrepresented in higher education. The hypocrisy is blatant.

This Compact is fundamentally and explicitly anti-transgender:

"Institutions commit to defining and otherwise interpreting 'male,' 'female,' 'woman,' and 'man' according to reproductive function and biological processes."

A person's gender and sexual identity have nothing to do with education at large whatsoever. It continues a trend of the Trump Administration demonizing and discriminating against one of the most vulnerable populations in the United States. Including something like this is utterly deplorable. If the University of Texas joins this Compact, it will not only turn its back on its trans population but also fail to recognize their very existence.

"Therefore, no more than 15 percent of a university's undergraduate student population shall be participants in the Student Visa Exchange Program, and no more than 5 percent shall be from any one country."

Setting arbitrary limits on international students is not only intrinsically xenophobic but also diminishes the so-called 'vibrant marketplace of ideas' when there is less variety in perspectives. Limiting international students, who typically pay full tuition, would also create economic shortfalls for universities like USC, thereby increasing their dependence on federal funding. Not to mention that these international students tend to stay in the United States and drive innovation here. Of the so-called Magnificent Seven tech companies, three were founded by first-generation immigrants. It's impossible to quantify the adverse effect that this would have on the United States economy. Why stop the best and brightest from coming to our great nation?

Let's not mince words here. This Compact is a quid pro quo, a tactic for which the President was already impeached. It's a thinly veiled attempt to install right-wing ideals in some of America's most prestigious schools in exchange for a quick buck. It aims to silence protests and dissenting voices. Much like his maligned and defunct Trump University, this is a grift. Do not let him make a mockery of our beloved university. Do not subject the university to politically-motivated compliance requirements.

A Plea for Common Sense​

If you believe, as I do, that the University of Texas at Austin is a world-class institution, then leave what makes it great in place. With a $20B+ endowment, this university does not need to capitulate to Trump's ideological demands and grovel for money. I am one tiny voice. I am merely a proud alumnus, not a megadonor or booster. I have donated to the university in the past, but if the school joins the Compact, I will withhold all further donations. My pride as a Longhorn will be severely diminished, but I will not be deterred by the xenophobia and prejudice that plagues the United States. Education is worth fighting for.

Hook 'em and fight on!

Disclaimer​

The opinions and views expressed here are my own and not those of my employer or any school.

As a senior engineer with nearly 13 years of professional experience, I've reinvented myself countless times. I've seen more technologies than I can count or remember come and go. I've seen bubbles form and burst. I remember when the latest flop was supposed to be the next big thing. I've spent countless hours learning new skills and honing my craft in a knowingly futile effort to stay up-to-date with technology. I've seen layoffs and downturns. Trust me, I understand that the macroeconomics of the tech industry has peaks and valleys. After all, I wrote my first Hello, world! in high school shortly after the .com crash. However, something feels different this time. I'm starting to feel deeply disillusioned about the future of AI.

Goodhart's Law and Vestigial Structures​

I will refer to the concept of Goodhart's Law a few times throughout this post, so I'll quickly define my interpretation. It states that:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Or, in layperson's terms (which I strongly prefer):

When a measure becomes a target, it ceases to be a good measure.

I think about this adage almost every single day. It's no secret that Amazon is one of the most data-driven companies on the planet. During my tenure at AWS, I wrote numerous docs overflowing with quantitative evidence to support my claims and proposals. As a customer-facing solutions architect, data was integral to guiding my customers on the right path and measuring the efficacy of my work. That said, Amazonians are also expected to balance the data with anecdotes. Like Jeff Bezos famously said back in 2018:

The thing I have noticed is that when the anecdotes and the data disagree, the anecdotes are usually right. There is something wrong with the way that you are measuring it.

I remember my professors at university telling stories of how programmers back in the day used to have their productivity measured by various arbitrary metrics like program execution duration, lines of code, etc. When these metrics become targets, other arguably more meaningful metrics like code maintainability quickly suffer.

Exhibit A:

const add = async (first: number, second: number): Promise<number> => {
  const sleep = (millisecondsToWait: number): Promise<void> => {
    return new Promise((resolve) => setTimeout(resolve, millisecondsToWait));
  };
  // waste five minutes of runtime
  await sleep(300000);
  return first + second;
};

Exhibit B:

// replace
// condition ? truePathReturn : falsePathReturn
// with extra lines
// add comments for even more lines
const ifWithExtraLines = (condition: Boolean, truePathReturn: object, falsePathReturn: object): object => {
  // check condition
  if (condition) {
    // return true path object if condition is true
    return truePathReturn;
  } else {
    // return false path object if condition is false
    return falsePathReturn;
  }
};

You get the idea. These code snippets are terrible, but if developers' targets are runtime or the number of lines, this is the type of software that is incentivized. Individuals will always index on their targets.

As the tech industry has narrowed its focus and investments on AI lately, I can't help but feel less human. I'm constantly bombarded on social media with the sensationalized idea that developers will soon be replaced. Programmers will become vestigial structures, casualties of the rise of AI agents, and ironically, the very source of data that will lead to our own demise. While I am skeptical about just how close humans are to AGI (artificial general intelligence), there's no doubt that we are approaching a critical juncture in technological history. I'm also not convinced that we're considering the right metrics.

I'm not going to masquerade as some genius economist, but let's consider a stock price as an example of a target for a moment. Everyone can agree that this is a meaningful target, but I argue that this is a short-term target. Why are some executives paid primarily via stock? So their focus is to raise the stock price, of course. Typically, vest periods of a few years ensure no short-term stock manipulation. However, they have a clear target and timeline. Other leadership now indexes on this as well. Many other metrics, such as the company's long-term performance, employee satisfaction, etc., are often less of a focus because they impact their target far less. Goodhart's Law in action. Now, most employees are constantly laser-focused on short-term growth in perpetuity. Not all companies are like this, but this example illustrates the point. However, I fear this myopic mentality honing in on short-term profitability and adoption, instead of ethics and considerations of socioeconomic impact, is prevalent in AI.

Developer Productivity Is Not for You​

With the advent of large language models (LLMs) and AI agents, there are typically two value propositions: hire fewer developers and/or materially augment their performance. While we can debate just how much the productivity gains are, tools like these undoubtedly help developers like me ship features faster than ever. But have you ever asked yourself who benefits from these productivity gains? In theory, these productivity gains mean more features. More features correlate to more revenue. More revenue sways the market. The market support bolsters the stock price. So simply put, your employer and its shareholders benefit from your increased productivity as a developer. Developer productivity is not for you.

While I use these tools daily for tasks like explaining code I did not write, I have no idea how this will impact my skills and cognitive sharpness in the long term. This is a nuanced question about human psychology. Having an LLM available to generate code for me can make me far less motivated to do my own thinking and research, especially during project crunch times when I am under massive pressure to deliver. These tools certainly make me more productive, but may not make me a better programmer. That is the key distinction because one benefits my employer, and the other benefits me as a skilled worker.

We Gave Away the Knowledge for Free (Under MIT)​

I also won't parade myself around as an open-source purist. The lion's share of the software I've authored resides under the lock and key of my megacorporate employers' private repositories. That said, I never imagined that all the code I (and much more so the actual open-source contributors) published freely on the internet would benefit a select few companies later, back when I signed up for my GitHub account over a decade ago. While access to these models is democratized for a fee, creating these massive models is limited to the select few corporate titans with the means of production (mountains of GPUs, barring revolutionary advancements in the vein of DeepSeek).

While these technological advancements are truly remarkable, I still struggle with the ethics of using these large models at times. Without namedropping any company or model, you can find countless lawsuits from creators who have coaxed significant portions of their works from LLMs and diffusion models without the proper licensing. These cases are ample evidence, in my opinion, that the macroeconomic corporate stance is to ask for forgiveness, not permission, for what content is acceptable to train models on.

Ultimately, we gave away the knowledge for free, but now we have to pay for access to its application. When we look at the jobs that are most at risk of being replaced by AI (or rather by its perceived value), they are mostly well-paying jobs in a rapidly vanishing middle class (e.g., developers) or creative fields like graphic design. To quote the brilliant tweet of @QuinnCat13:

We could automate menial jobs so people have time to make art and music, but apparently, we'd rather automate art and music so people have time for menial jobs.

Software like Glaze, which adds minor modifications to images that are imperceptible to the human eye but makes it more difficult for models to train on, gives me hope for checks, balances, and enforcement in AI ethics. However, these powerful models already exist. Remediation is much more challenging than prevention.

SaaS Is Not Going to Save Us​

One trendy take I've seen from the indie hacker influencers on Twitter and LinkedIn is that instead of working for a corporation, you should quickly build your own products using LLMs and IDEs like Cursor. This way, you own your intellectual property and generate revenue directly. However, I see several problems.

I do not want to gatekeep how we define a programmer, so I'll use artistry as an example. I have zero experience in Photoshop, Illustrator, etc. I am not a graphic designer or illustrator by any stretch of the imagination. However, I can now throw a sentence or two into a diffusion model and produce imagery. Am I now an artist? I argue not. If any modifications need to be made to the output, I am utterly incapable of handling them unless I can do so via prompt engineering. The rendered art will also be soulless, lack perspective, and solely regurgitate existing works of actual artists. Similar problems exist for purely AI-generated codebases by inexperienced programmers. I won't namedrop and pile on to them, but you can find numerous examples of security breaches and unexpected cost spikes due to issues in AI-generated code.

Another key issue is the concept of differentiated value. This is a common argument for using cloud service providers like AWS. You can run your own data centers, manage your own virtualization, build your own provisioning automation, etc. However, if AWS can do it out-of-the-box with better economies of scale than your company, why waste time on work that does not set your product apart from its competition? A common approach is to limit undifferentiated work. If AI primarily writes your SaaS product, then the differentiated value of your business is effectively a set of prompts. While I predict that LLMs have and will continue to plateau, let's assume I'm wrong. Why would a company (or individual power users) pay a recurring fee for your product if they can get the same results by generating and plugging in the prompts themselves? The barrier to entry for developers is lowered with AI appliances, but corporations also have this exact ability with far more resources behind them. SaaS is not going to save us.

No One Left to Pay​

At the height of the Great Depression, unemployment hit approximately 25% in the United States. Most people fail to realize how much the global economy depends on people having money to consume services and goods. Again, let's assume that I'm wrong about the AI plateau. Let's also ignore supply and demand issues for compute resources. Who is left to pay for anything if AGI replaces all these jobs? A global economic crisis would ensue. I don't know whether it's a universal basic income, an oligarchy with access to key resources, or something else that emerges after AGI, but it does not look bright for humanity at that point.

A Focus on Humanity​

Believe it or not, this post started as a therapy exercise. I've been struggling lately to define my worth, identity, and goodness as a human being without relying on my career accolades, the perceived prestige of the company I work for, the quality of my education, etc. When I look to others for inspiration online, I see the same qualities: braggadocious reflections about their many followers, indie makers plastering their incomes on their profiles, toxic elitism, etc. We've been conditioned for these targets in this influencer-obsessed society.

Over the years, I've sought personal validation and measured my worth through various metrics that, unfortunately, are deeply rooted in capitalism and vanity. At no other point in human history have we had unfettered access to data about ourselves as we do now. We can measure how seen we are with social media views, how much people approve of our thoughts via likes, how valuable we are with our incomes and contributions to our employers, etc. Perhaps even worse, we can also see these metrics for our peers. We choose these arbitrary metrics as our targets, and the wrong actions are incentivized. We build and optimize for these targets, and humanity takes a backseat. No longer do we have time to gaze up at the stars, draw constellations, tell stories, have downtime, etc., as our ancient ancestors did. We've replaced this with incessant media consumption, working more than ever, and nonstop reminders that all the numbers that matter to us must keep going up.

I don't blog often, but I felt inspired to write about my frustrations with the industry, society, and myself. I still have a genuine love of technology and am grateful that my career revolves around this passion, a privilege I do not take for granted. However, now more than ever, it's time for me to relinquish some of that competitive, capitalistic vanity and reconnect with my humanity. I do not care about constant improvement, revenue growth, external validation, etc., the same way that I used to. I want to spend more time creating for myself, learning for fun, exploring spirituality or connection with the natural world, and investing time in things that make the world a slightly better place. If AI replaces me in the process, then so be it.

I have no power to make changes on a massive scale. I'm a minuscule cog in this intricate and profoundly unfair machine, like most people. I can offer up my authentic self, though. I can share my tiny voice that questions the ethics and benefits of this technology that we know and use. I acknowledge that the anecdote has much more context than the data. I can share my journey to define my self-worth and goodness without the quantitative pressure of comparing myself to others. I can strive to be more human in an increasingly AI-obsessed world.

Disclaimer​

When I started this blog post, I worked for Amazon Web Services. The opinions and views expressed here are my own and not those of Amazon. I have since left the company.

While I've used Kubernetes professionally in a few capacities (particularly in customer engagements while working at AWS), I wanted to cement my knowledge and increase my mastery with a systematic approach. I decided to prepare for the Certified Kubernetes Application Developer (CKAD) exam. I've taken and passed more than a dozen technology certification exams spanning AWS, Azure, HashiCorp, and more. This exam is unique in several ways. Namely, it's all hands-on in a lab environment. Azure exams often have a coding, configuration, or CLI command component, but even these are typically multiple-choice questions. The CKAD presents you with a virtual desktop and several Kubernetes clusters, making you tackle 15-20 tasks with a strict two-hour time limit. I put together this repository and post for a few reasons:

• I wanted to document all of my hands-on preparation for when I have to recertify in two years
• I wanted to share my knowledge with others and offer a supplemental guide to a CKAD course
• Since the CKAD exam focuses on Kubernetes from a cloud-agnostic perspective, I wanted to fill in the gaps in my own knowledge of running Kubernetes in the AWS ecosystem (e.g., Karpenter, Container Insights, etc.)
• Many courses and guides leverage Microk8s or minikube to run Kubernetes locally, but I wanted to focus on cloud-based infrastructure, especially for things like EBS volumes created via PVCs, ELBs created via a Service, etc.

In summary, this material focuses on hands-on exercises for preparing for the exam and other tools in the cloud-agnostic and AWS ecosystems.

Preparing for the Exam​

While two hours may sound like plenty of time, you'll need to work quickly to complete the exam. With an average of six to eight minutes per exercise (each is not timed individually), ensuring you can work efficiently and ergonomically is paramount. The following items were incredibly useful for me:

• Running through a practice exam to get a feel for the CKAD structure
• Proficiency with Vim motions (since most of the exam takes place in a terminal) to efficiently edit code
• Generating YAML manifests via the command line for new resources instead of copying and pasting from documentation (e.g., kubectl create namespace namespace-one -o yaml --dry-run=client)
• Generating YAML manifests for existing resources that do not have one (e.g., kubectl get namespace namespace-one -o yaml > namespace.yaml)
• Leveraging the explain command instead of looking up resource properties in the web documentation (e.g., kubectl explain pod.spec)
• Memorizing the syntax for running commands in a container (e.g., kubectl exec -it pod-one -- /bin/sh) and for quickly creating a new Pod to run commands from (e.g., kubectl run busybox-shell --image=busybox --rm -it --restart=Never -- sh)
• Refreshing knowledge of Docker commands like exporting an image (i.e., docker save image:tag --output image.tar)

Materials and Getting Started​

All code shown here resides in this GitHub repository. In addition to this content, I highly recommend the following:

• The CKAD courses on Pluralsight for classroom learning
• Killer Shell for practice exams
• This GitHub repository for many useful CLI commands

My preferred approach was to work through the Pluralsight course first. After reviewing the classroom material, I designed and implemented the examples below. If you have foundational Kubernetes knowledge, skip to the most useful exercises. Each one is designed to be a standalone experience.

00: eksctl Configuration​

eksctl is a powerful CLI tool that quickly spins up and tears down Kubernetes clusters via Amazon EKS. Nearly all of the exercises below start by leveraging the tool to create a cluster:

# before running these commands, first authenticate with AWS (e.g., aws configure sso)
eksctl create cluster -f cluster.yaml
# if connecting to an existing cluster
eksctl utils write-kubeconfig --cluster=learning-kubernetes

The default cluster configuration uses a two-node cluster of t3.medium instances to keep hourly costs as low as possible. At the time of writing this blog post, the exam tests on Kubernetes version 1.30.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: learning-kubernetes
  region: us-west-2
  version: '1.30'

nodeGroups:
  - name: node-group-1
    instanceType: t3.medium
    desiredCapacity: 2
    minSize: 2
    maxSize: 2

This cluster can be transient for learning purposes. To keep costs low, be sure to run the destroy-cluster.sh script to delete the cluster when not in use. I also recommend configuring an AWS Budget as an extra measure of cost governance.

eksctl delete cluster --config-file=cluster.yaml --disable-nodegroup-eviction

01: First Deployment with Nginx (CKAD Topic)​

With the cluster created, we can now make our first Deployment. We'll start by creating a web server with three replicas using the latest Nginx image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx-container
          image: nginx:latest
          ports:
            - containerPort: 80

The following commands leverage the manifest to create three Pods and inspect them:

# assumes cluster created from 00-eksctl-configuration first
kubectl apply -f ./
# returns three pods (e.g., nginx-deployment-5449cb55b-jfgnc)
kubectl get pods -o wide
# clean up
kubectl delete -f ./

02: Pod Communication over IP (CKAD Topic)​

The Deployment in this example is identical to the previous: a web server with three replicas. Use the following commands to explore how IP addressing works for Pods:

# assumes cluster created from 00-eksctl-configuration first
kubectl apply -f ./
# 192.168.51.32 is one of my pod's IP address, but yours will be different
# when the pod is replaced, this IP address changes
kubectl get pods -o wide
# creates a pod with the BusyBox image
# entering BusyBox container shell to communicate with pods in the cluster
kubectl run -it --rm --restart=Never busybox --image=busybox sh
# replace the IP address as needed
wget 192.168.51.32
# displys the nginx homepage code
cat index.html
# returning to default shell and deletes the BusyBox pod
exit
# clean up
kubectl delete -f ./

03: First Service (CKAD Topic)​

Since each Pod has a separate IP address that can change, we can use a Service to keep track of the Pod's IP addresses on our behalf. This abstraction allows us to group Pods via a selector and reference them via a single Service. In the Service manifest and leveraging the same Deployment as before, we specify how to select which Pods to target, what port to expose, and the type of Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    name: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      # nodePort is used for external access
  # ClusterIP services are only accessible within the cluster
  # NodePort services are a way to expose ClusterIP services externally without using a cloud provider's load balancer
  # LoadBalancer is covered in the next section
  type: ClusterIP

Using the Service, we have a single interface to the three nginx replicas. We can also use the Service name instead of its IP address.

# assumes cluster created from 00-eksctl-configuration first
kubectl apply -f ./
# 10.100.120.203 is the service IP address
kubectl describe service nginx-service
# entering BusyBox container shell
kubectl run -it --rm --restart=Never busybox --image=busybox sh
# can also use the IP address instead
wget nginx-service
cat index.html
# returning to default shell
exit
# clean up
kubectl delete -f ./

04: Elastic Load Balancers for Kubernetes Service (CKAD Topic)​

A significant benefit of Kubernetes is that it can create and manage resources in AWS on our behalf. Using the AWS Load Balancer Controller, we can specify annotations to create a Service of type LoadBalancer that leverages an Elastic Load Balancer. Using the same Deployment from the past two sections, this manifest illustrates how to leverage a Network Load Balancer for the Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-load-balancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
    # by default, a Classic Load Balancer is created
    # https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/introduction.html
    # this annotation creates a Network Load Balancer
    service.beta.kubernetes.io/aws-load-balancer-type: 'nlb'
spec:
  selector:
    name: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
      - ip: '192.0.2.127'

The following commands deploy the LoadBalancer Service:

# assumes cluster created from 00-eksctl-configuration first
kubectl apply -f ./
# entering BusyBox container shell
kubectl run -it --rm --restart=Never busybox --image=busybox sh
wget nginx-load-balancer
cat index.html
# returning to default shell
exit
# clean up
# this command ensures that the load balancer is deleted
# be sure to run before destroying the cluster
kubectl delete -f ./

05: Ingress (CKAD Topic)​

Services of type ClusterIP only support internal cluster networking. The NodePort configuration allows for external communication by exposing the same port on every node (i.e., EC2 instances in our case). However, this introduces a different challenge because the consumer must know the nodes' IP addresses (and nodes are often transient). The LoadBalancer configuration has a 1:1 relationship with the Service. If you have numerous Services, the cost of load balancers may not be feasible. Ingress alleviates some of these challenges by providing a single external interface over HTTP or HTTPS with support for path-based routing. Leveraging the Nginx example one last time, we can create an Ingress that exposes a Service with the NodePort configuration via an Application Load Balancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx-service
                port:
                  number: 80

The following commands install the AWS Load Balancer Controller, configure required IAM permissions, and deploy the Ingress. Be sure to set the $AWS_ACCOUNT_ID environment variable first.

# assumes cluster created from 00-eksctl-configuration first
# install AWS Load Balancer Controller
# https://docs.aws.amazon.com/eks/latest/userguide/lbc-manifest.html
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/install/iam_policy.json
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json
rm iam_policy.json
eksctl utils associate-iam-oidc-provider --region=us-west-2 --cluster=learning-kubernetes --approve
eksctl create iamserviceaccount \
  --cluster=learning-kubernetes \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve
kubectl apply \
    --validate=false \
    -f https://github.com/jetstack/cert-manager/releases/download/v1.13.5/cert-manager.yaml
curl -Lo v2_7_2_full.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.7.2/v2_7_2_full.yaml
sed -i.bak -e '596,604d' ./v2_7_2_full.yaml
sed -i.bak -e 's|your-cluster-name|learning-kubernetes|' ./v2_7_2_full.yaml
kubectl apply -f v2_7_2_full.yaml
rm v2_7_2_full.yaml*
kubectl get deployment -n kube-system aws-load-balancer-controller
# apply maniftests
kubectl apply -f ./
# gets address (e.g, http://k8s-default-ingress-08daebdfec-204015293.us-west-2.elb.amazonaws.com/) that can be opened in a web browser
kubectl describe ingress
# clean up
kubectl delete -f ./

06: Jobs and CronJobs (CKAD Topic)​

Jobs are a powerful mechanism that reliably ensures that Pods are completed successfully. CronJobs extend this functionality by supporting a recurring schedule.

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
        - name: pi
          image: perl:5.34.0
          command: ['perl', '-Mbignum=bpi', '-wle', 'print bpi(2000)']
      restartPolicy: Never
  backoffLimit: 4

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello
spec:
  # runs every minute
  schedule: '* * * * *'
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: hello
              image: busybox:1.28
              imagePullPolicy: IfNotPresent
              command:
                - /bin/sh
                - -c
                - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

07: Metrics Server and Pod Autoscaling (CKAD Topic)​

Metrics Server provides container-level resource metrics for autoscaling within Kubernetes. It is not installed by default and is meant only for autoscaling purposes. There are other options, such as Container Insights, Prometheus, and Grafana, for more accurate resource usage metrics (all covered later in this post). With Metrics Server installed, a HorizontalPodAutoscaler resource can be configured with values such as target metric, minimum replicas, maximum replicas, etc.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
status:
  observedGeneration: 1
  currentReplicas: 1
  desiredReplicas: 1
  currentMetrics:
    - type: Resource
      resource:
        name: cpu
        current:
          averageUtilization: 0
          averageValue: 0

HorizontalPodAutoscalers create and destroy Pods based on metric usage. On the other hand, vertical autoscaling rightsizes the resource limits (covered in the next section) for Pods.

08: Resource Management (CKAD Topic)​

When creating a Pod, you can optionally specify an estimate for the number of resources a container needs (e.g., CPU and RAM). This baseline estimate should be specified in the requests parameter. The limits parameter specifies the threshold for which a container should be terminated to prevent starvation of other processes. Limits also help with cluster capacity planning (e.g., EKS node groups). Below is the Nginx Deployment from earlier with resource management applied:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx-container
          image: nginx:latest
          ports:
            - containerPort: 80
          resources:
            # estimated resources for container to run optimally
            requests:
              cpu: 100m
              memory: 128Mi
            # kills the container if threshold is crossed
            limits:
              cpu: 200m
              memory: 256Mi

09: Karpenter​

In the previous two sections, we covered how additional Pods are created (i.e., horizontal scaling) and how resources (e.g., CPU and RAM) are requested and limited in Kubernetes. The next topic is managing the underlying compute when additional infrastructure is required. There are two primary options for scaling compute using EKS on EC2: Cluster Autoscaler and Karpenter. On AWS, Cluster Autoscaler leverages EC2 Auto Scaling Groups (ASGs) to manage node groups. Cluster Autoscaler typically runs as a Deployment in the cluster. Karpenter does not leverage ASGs, allowing for the ability to select from a wide array of instance types that match the exact requirements of the additional containers. Karpenter also allows for easy adoption of Spot for further cost savings on top of better matching the workload to compute resources. The cluster defined in 00-eksctl-configuration uses an unmanaged node group and does not leverage Cluster Autoscaler or Karpenter. To demonstrate how to leverage Karpenter, we'll need a different cluster configuration file. We can dynamically generate it like so:

# set environment variables
export KARPENTER_NAMESPACE=karpenter
export KARPENTER_VERSION=v0.32.10
export K8S_VERSION=1.28
export AWS_PARTITION="aws"
export CLUSTER_NAME="${USER}-karpenter-demo"
export AWS_DEFAULT_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)"
export AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)"
export GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)"
# deploy resources to support Karpenter
aws cloudformation deploy \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file karpenter-support-resources-cfn.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"
# generate cluster file and deploy
<<EOF > cluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_DEFAULT_REGION}
  version: "${K8S_VERSION}"
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}

iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: karpenter
      namespace: "${KARPENTER_NAMESPACE}"
    roleName: ${CLUSTER_NAME}-karpenter
    attachPolicyARNs:
    - arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}
    roleOnly: true

iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
  username: system:node:{{EC2PrivateDNSName}}
  groups:
  - system:bootstrappers
  - system:nodes

managedNodeGroups:
- instanceType: t3.medium
  amiFamily: AmazonLinux2
  name: ${CLUSTER_NAME}-ng
  desiredCapacity: 2
  minSize: 2
  maxSize: 5
EOF
eksctl create cluster -f cluster.yaml

Next, we install Karpenter on the EKS cluster:

# set additional environment variables
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
# install Karpenter
helm registry logout public.ecr.aws
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN}" \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Finally, we create a node pool that specifies what compute our workload can support. In this case, Karpenter can provision EC2 Spot instances from the c, m, or r families from any generation greater than two running Linux on AMD64 architecture.

# create NodePool
<<EOF > node-pool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  amiSelectorTerms:
    - id: "${ARM_AMI_ID}"
    - id: "${AMD_AMI_ID}"
EOF
kubectl apply -f node-pool.yaml

With the new EKS cluster deployed and Karpenter installed, we can add new Pods and see new EC2 instances created on our behalf.

# deploy pods and scale
kubectl apply -f deployment.yaml
kubectl scale deployment inflate --replicas 5

In less than a minute after the inflate command, a new EC2 instance is created that matches the node pool specifications. In my case, a c5n.2xlarge server was deployed.

As expected, the node pool leverages Spot instances.

You can monitor the Karpenter logs via the command below. Less than a minute after deleting the Deployment, the c5n.2xlarge instance was terminated. Be sure to follow the cleanup steps when done to ensure no resources become orphaned.

# monitor Karpenter events
kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller
# scale down
kubectl delete deployment inflate
# clean up
helm uninstall karpenter --namespace "${KARPENTER_NAMESPACE}"
aws cloudformation delete-stack --stack-name "Karpenter-${CLUSTER_NAME}"
aws ec2 describe-launch-templates --filters Name=tag:karpenter.k8s.aws/cluster,Values=${CLUSTER_NAME} |
    jq -r ".LaunchTemplates[].LaunchTemplateName" |
    xargs -I{} aws ec2 delete-launch-template --launch-template-name {}
eksctl delete cluster --name "${CLUSTER_NAME}"

10: Persistent Volumes Using EBS (CKAD Topic)​

Storage in Kubernetes can be classified as either ephemeral or persistent. Without leveraging PersistentVolumes (PVs), containers read and write data to the volume attached to the node they run on. Ephemeral storage is temporary and tied to the Pod's lifecycle. If requirements dictate that the storage persists or be shared across Pods, there are some prerequisites before EBS can be leveraged for PVs.

The first step is installing the AWS EBS Container Storage Interface (CSI) driver. The next step is to define a StorageClass (SC) that includes configuration such as volume type (e.g., gp3), encryption, etc. The final step is to reference a PersistentVolumeClaim (PVC) when deploying a Pod in order to dynamically provision the EBS volume and attach to the containers.

In practice, this goes as follows:

# assumes cluster created from 00-eksctl-configuration first
# create an OIDC provider
eksctl utils associate-iam-oidc-provider --cluster learning-kubernetes --approve
# install aws-ebs-csi-driver
eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster learning-kubernetes \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve
eksctl create addon --name aws-ebs-csi-driver --cluster learning-kubernetes --service-account-role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole --force

Once completed, the add-on will appear in the AWS Console.

Next, define the StorageClass and PersistentVolumeClaim:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  csi.storage.k8s.io/fstype: xfs
  type: gp3
  encrypted: 'true'
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.ebs.csi.aws.com/zone
        values:
          - us-west-2a
          - us-west-2b
          - us-west-2c

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi

Finally, attach the PVC to the Pod and deploy:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: centos
      command: ['/bin/sh']
      args: ['-c', 'while true; do echo $(date -u) >> /data/out.txt; sleep 5; done']
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim

As soon as the Pod is created, a gp3 volume is provisioned.

11: Prometheus and Grafana​

The next several sections focus on observability. Prometheus is an open-source monitoring system commonly leveraged in Kubernetes clusters. As a de facto standard, it's widely used with Grafana to provide cluster monitoring. Using Helm we can quickly deploy both of these tools to our cluster.

# assumes cluster created from 00-eksctl-configuration first
# install Helm on local machine
# https://helm.sh/docs/intro/install/
brew install helm
# install Helm charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install v60-0-1 prometheus-community/kube-prometheus-stack --version 60.0.1
# use http://localhost:9090 to access Prometheus
kubectl port-forward svc/prometheus-operated 9090
# get Grafana password for admin
kubectl get secret v60-0-1-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
# use http://localhost:3000 to access Grafana
kubectl port-forward svc/v60-0-1-grafana 3000:80

Using port forwarding, we can quickly access Prometheus:

And Grafana:

12: Container Insights​

Prometheus and Grafana are both open-source and cloud-agnostic. AWS has a native infrastructure monitoring offering called Container Insights that integrates cluster data with the AWS Console via CloudWatch with two simple commands:

# assumes cluster created from 00-eksctl-configuration first
# configure permissions
# change role to the one created by eksctl
aws iam attach-role-policy \
--role-name $EKSCTL_NODEGROUP_ROLE_NAME \
--policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
# wait until add-on is installed and give time for data to propagate
aws eks create-addon --cluster-name learning-kubernetes --addon-name amazon-cloudwatch-observability

It's worth noting that Container Insights can also ingest Prometheus metrics.

13: EKS Split Cost Allocation Data in Cost and Usage Reports​

The AWS Cost and Usage Reports (CUR) are the most comprehensive and detailed billing data available to customers. It offers a well-defined schema that we can use to write SQL queries against via Athena. CUR data offers resource-level time series data for in-depth AWS cost and usage analysis. In April 2024, AWS released EKS split cost allocation data for CUR. Previously, the lowest resource level available was an EC2 instance. This feature adds billing data container-level resources in EKS (e.g., Pods).

Create a new CUR via Data Exports in the Billing and Cost Management Console if required. If you have an existing CUR without split cost allocation data, you can modify the report content configuration to add this.

With this configured, we can use the following SQL query in Athena to gather cost and usage data for the EKS cluster resources:

SELECT
  DATE_FORMAT(
    DATE_TRUNC(
      'day', "line_item_usage_start_date"
    ),
    '%Y-%m-%d'
  ) AS "date",
  "line_item_resource_id" AS "resource_id",
  ARBITRARY(CONCAT(
    REPLACE(
      SPLIT_PART(
        "line_item_resource_id",
        '/', 1
      ),
      'pod',
      'cluster'
    ),
    '/',
    SPLIT_PART(
      "line_item_resource_id",
      '/', 2
    )
  )) AS "cluster_arn",
  ARBITRARY(SPLIT_PART(
    "line_item_resource_id",
    '/', 2
  )) AS "cluster_name",
  ARBITRARY("split_line_item_parent_resource_id") AS "node_instance_id",
  ARBITRARY("resource_tags_aws_eks_node") AS "node_name",
  ARBITRARY(SPLIT_PART(
    "line_item_resource_id",
    '/', 3
  )) AS "namespace",
  ARBITRARY("resource_tags_aws_eks_workload_type") AS "controller_kind",
  ARBITRARY("resource_tags_aws_eks_workload_name") AS "controller_name",
  ARBITRARY("resource_tags_aws_eks_deployment") AS "deployment",
  ARBITRARY(SPLIT_PART(
    "line_item_resource_id",
    '/', 4
  )) AS "pod_name",
  ARBITRARY(SPLIT_PART(
    "line_item_resource_id",
    '/', 5
  )) AS "pod_uid",
  SUM(
    CASE WHEN "line_item_usage_type" LIKE '%EKS-EC2-vCPU-Hours' THEN "split_line_item_split_cost" + "split_line_item_unused_cost" ELSE 0.0 END
  ) AS "cpu_cost",
  SUM(
    CASE WHEN "line_item_usage_type" LIKE '%EKS-EC2-GB-Hours' THEN "split_line_item_split_cost" + "split_line_item_unused_cost" ELSE 0.0 END
  ) AS "ram_cost",
  SUM(
    "split_line_item_split_cost" + "split_line_item_unused_cost"
  ) AS "total_cost"
FROM
  cur
WHERE
  "line_item_operation" = 'EKSPod-EC2'
  AND CURRENT_DATE - INTERVAL '7' DAY <= "line_item_usage_start_date"
GROUP BY
  1,
  2
ORDER BY
  "cluster_arn",
  "date" DESC

AWS also offers open-source QuickSight dashboards that provide a visualization of this data.

14: ConfigMap (CKAD Topic)​

The following two sections focus on configuration management. A ConfigMap is a Kubernetes construct that stores non-sensitive key-value pairs (e.g., URLs, feature flags, etc.). There are several ways to consume ConfigMaps, but we'll set an environment variable for a container below. First, I created a TypeScript Cloud Development Kit (CDK) application to deploy a FastAPI container to Elastic Container Repository (ECR). The API is simple:

api = fastapi.FastAPI()
@api.get('/api/config')
def config():
    return {
        'message': os.getenv('CONFIG_MESSAGE', 'Message not set')
    }

We publish the container to ECR via CDK:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { DockerImageAsset } from 'aws-cdk-lib/aws-ecr-assets';

export class ApiCdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    const dockerImageAsset = new DockerImageAsset(this, 'MyDockerImage', {
      directory: './container/',
    });
  }
}

Next, we define the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-configmap
data:
  config-message: 'Hello from ConfigMap!'

Finally, we reference the ConfigMap in the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: config-api-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: config-api
  template:
    metadata:
      labels:
        name: config-api
    spec:
      containers:
        - name: config-api-container
          # deployed via CDK
          # replace with your image
          image: 196736724465.dkr.ecr.us-west-2.amazonaws.com/cdk-hnb659fds-container-assets-196736724465-us-west-2:afbbd8d7b43a7f833eb07c26a13d5344fa7656c136b1e27b545490fa58dad983
          ports:
            - containerPort: 8000
          env:
            - name: CONFIG_MESSAGE
              valueFrom:
                configMapKeyRef:
                  name: api-configmap
                  key: config-message

With the API deployed, we can verify that the configuration propagates correctly.

# entering BusyBox container shell
kubectl run -it --rm --restart=Never busybox --image=busybox sh
wget config-api-service:80/api/config
cat config

15: Secrets (CKAD Topic)​

Secrets are very similar to ConfigMaps except that they are intended for sensitive information. Opaque is the default type of Secret for arbitrary user data unless you need to store SSH credentials, TLS certificates, ~/.dockercfg, etc. For a complete list of types, see the documentation. Kubernetes Secrets do not encrypt the data on your behalf. That responsibility is on the developer.

apiVersion: v1
kind: Secret
metadata:
  name: busybox-password
type: Opaque
data:
  password: MWYyZDFlMmU2N2Rm

16: Multi-Container Pods (CKAD Topic)​

In the examples so far, Pods and containers had a 1:1 relationship. Two common patterns for multi-container Pods in Kubernetes are init containers and sidecars. To illustrate these patterns, we'll use a PostgreSQL database with a backend that relies on it. Given that the backend container depends on the database, we must ensure that PostgreSQL is available before starting it. To do so, we can use an init container that verifies the ability to connect to the database. All init containers run before the Pod starts. If any init container fails, the Pod fails.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-with-database
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  replicas: 1
  template:
    metadata:
      labels:
        app: backend
    spec:
      initContainers:
        - name: verify-database-online
          image: postgres
          command: [
              'sh',
              '-c',
              'until pg_isready -h database-service -p 5432;
              do echo waiting for database; sleep 2; done;',
            ]
      containers:
        - name: backend
          image: nginx

An example of a sidecar container is a GUI called Adminer for the database. The GUI has a lifecycle tightly coupled to the Postgres container (i.e., if we don't need the database anymore, we don't need the GUI). To configure a sidecar, append another container to the Deployment's spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-database
  namespace: default
spec:
  selector:
    matchLabels:
      app: database
  replicas: 1
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - name: database
          image: postgres
          envFrom:
            - configMapRef:
                name: database-access
          ports:
            - containerPort: 5432
        - name: database-admin
          image: adminer
          ports:
            - containerPort: 8080

With the sidecar in place, we can deploy and leverage the GUI to log into our database.

# assumes cluster created from 00-eksctl-configuration first
kubectl apply -f database.configmap.yaml
kubectl apply -f backend.deployment.yaml
# check that the primary container is not yet running because the init container has not completed
# STATUS shows as Init:0/1
kubectl get pods
# deploy database and service
kubectl apply -f database.deployment.yaml
kubectl apply -f database.service.yaml
# verify that init container has completed
# get database pod
kubectl get pods
# forward ports
kubectl port-forward pod/postgres-database-697695b774-xcp9p 9000:8080
# open Adminer in browser
# see screenshot for logging in
wget http://localhost:9000
# clean up
kubectl delete -f ./

17: Deployment Strategies (CKAD Topic)​

The four most common deployment strategies are rolling, blue/green, canary, and recreate. Rolling updates involve deploying new Pods in a batch while decreasing old Pods at the same rate. This is the default behavior in Kubernetes. Blue/green deployments provision an entirely new environment (green) parallel to the existing one (blue), then perform a Service selector cutover when approved for production release. Canary deployments allow developers to test a new deployment with a subset of users in parallel with the current production release. Recreating an environment involves destroying the old environment and then provisioning a new one, which may result in downtime.

For a blue/green release, let's start by creating the blue and green deployments. The following YAML for the blue deployment is nearly identical to the green. The only difference is the Docker image used.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blue-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
      role: blue
  template:
    metadata:
      labels:
        app: nginx
        role: blue
    spec:
      # the green deployment uses the green Docker image
      containers:
        - name: blue
          image: scottenriquez/blue-nginx-app
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          resources:
            limits:
              memory: '128Mi'
              cpu: '200m'

By default, the production Service should point to the blue environment.

kind: Service
apiVersion: v1
metadata:
  name: production-service
  labels:
    env: production
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - port: 9000
      targetPort: 80

To perform the release, change the selector on the Production service. Then verify that the web application contains the green release instead of the blue.

# perform cutover
# can also be done via manifest
kubectl set selector service production-service 'role=green'
# entering BusyBox container shell
kubectl run -it --rm --restart=Never busybox --image=busybox sh
# verify green in HTML
wget production-service:9000
cat index.html

Switching gears to a canary release, we start by creating stable and canary Deployments. In this code example, the two web applications are nearly identical, except that the canary has a yellow message in a <h1> tag. We control the percentage of canary Pods by splitting the number of canary and stable replicas. For this example, there is a 20% chance of using a canary Pod because there is one canary replica and four stable replicas.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: canary-deployment
spec:
  # the stable Deployment has four replicas
  replicas: 1
  selector:
    matchLabels:
      track: canary
  template:
    metadata:
      labels:
        app: nginx
        track: canary
    spec:
      # the stable deployment uses the stable Docker image
      containers:
        - name: canary-deployment
          image: scottenriquez/canary-nginx-app
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          resources:
            limits:
              memory: '128Mi'
              cpu: '200m'

With this approach, traffic will be directed to the canary pod on average 20% of the time. It may take several requests to the Service, but a canary webpage will eventually be returned.

18: Probes (CKAD Topic)​

There are two primary types of probes: readiness and liveness. Kubernetes uses liveness probes to determine when to restart a container (i.e., a health check). It uses readiness probes to determine when a container is ready to accept traffic. These two probes are independent and unaware of each other. Probes of type HTTP, TCP, gRPC, and shell commands are supported. For this example, we'll use HTTP for both and add them as endpoints to an API:

@api.get('/api/healthy')
def config():
    return {
        'healthy': True
    }

@api.get('/api/ready')
def config():
    return {
        'ready': True
    }

In the Deployment manifest, we simply map the probes to the endpoints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-api-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: probe-api
  template:
    metadata:
      labels:
        name: probe-api
    spec:
      containers:
        - name: probe-api-container
          # deployed via CDK
          # replace with your image
          image: 196736724465.dkr.ecr.us-west-2.amazonaws.com/cdk-hnb659fds-container-assets-196736724465-us-west-2:86b591781a296c7b2980608eeb67e30aaf316c732c92b6a47e536555bce0dc93
          ports:
            - containerPort: 8000
          resources:
            limits:
              cpu: 250m
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /api/healthy
              port: 8000
          readinessProbe:
            httpGet:
              path: /api/ready
              port: 8000

19: SecurityContext (CKAD Topic)​

A SecurityContext resource configures a Pod's privilege and access control settings, such as enabling or disabling Linux capabilities and running as a specific user ID. This topic is straightforward but critical for the exam.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
    - name: security-context-pod-volume
      emptyDir: {}
  containers:
    - name: security-context-container
      image: busybox:1.28
      command: ['sh', '-c', 'id']
      volumeMounts:
        - name: security-context-pod-volume
          mountPath: /data/security-context-volume
      securityContext:
        allowPrivilegeEscalation: false

20: ServiceAccounts and Role-Based Access Control (CKAD Topic)​

Like how Identity and Access Management (IAM) in AWS grants principals permissions to specific actions for specific resources, Kubernetes Roles and ServiceAccounts allow resources within the cluster to leverage the control plane to perform operations on the cluster. For this example, let's grant a Pod access to get other Pods. We start by creating a Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
  # "" indicates the core API group
  - apiGroups: ['']
    resources: ['pods']
    verbs: ['get', 'watch', 'list']

A ServiceAccount provides an identity for processes that run inside Pods. We generate one next:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubernetes.io/enforce-mountable-secrets: 'true'
  name: sa-pod-reader

Next, we bind the Role to the ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
  - kind: ServiceAccount
    name: sa-pod-reader
    apiGroup: ''
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: ''

Then, we create a Pod that leverages the ServiceAccount:

apiVersion: v1
kind: Pod
metadata:
  name: reader-pod
spec:
  serviceAccountName: sa-pod-reader
  containers:
    - name: reader-container
      image: alpine:3.12
      resources:
        limits:
          memory: '128Mi'
          cpu: '500m'
      command: ['/bin/sh']
      args: ['-c', 'sleep 3600']

Finally, we can test specific operations against the API server to validate that certain actions are allowed and others are denied.

# entering Pod shell
kubectl exec -it reader-pod -- sh
# install curl
apk --update add curl
# get Pods
# allowed by Role
curl -s --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes/api/v1/namespaces/default/pods
# get Secrets
# denied by Role
curl -s --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes/api/v1/namespaces/default/secrets

21: NetworkPolicy (CKAD Topic)​

By default, network traffic between Pods is unrestricted. In other words, any Pod can communicate with any other Pod. A NetworkPolicy is a Kubernetes resource that uses selectors to implement granular ingress and egress rules. However, a network plugin must first be installed in the cluster to leverage NetworkPolicies. For this example, we will use Calico. For EKS, this only requires two commands:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yaml
kubectl create -f - <<EOF
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: AmazonVPC
  calicoNetwork:
    bgp: Disabled
EOF

With the network plugin installed, we can define a simple NetworkPolicy based on three example Pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pod-one-and-two-network-policy
spec:
  podSelector:
    matchLabels:
      # this label is also attached to the first two Pods but not the third
      network: allow-pod-one-and-two
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              network: allow-pod-one-and-two
  egress:
    - to:
        - podSelector:
            matchLabels:
              network: allow-pod-one-and-two

After applying the manifest above, we can validate that the network traffic now behaves as expected (i.e., the first two Pods can communicate with each other without allowing traffic from the third).

# get Pod IP addresses
kubectl get pods -o wide
# enter pod-three shell
kubectl exec -it pod-three -- sh
# ping pod-one and pod-two IP address (replace with yours)
# these commands should hang
ping 192.168.2.246
ping 192.168.21.2
# returning to default shell
exit
# enter pod-one shell
kubectl exec -it pod-one -- sh
# ping pod-two IP address (replace with yours)
# this command should be successful
ping 192.168.21.2
# ping pod-three IP address (replace with yours)
# this command should hang
ping 192.168.71.123

22: ArgoCD​

ArgoCD is a declarative continuous delivery tool for Kubernetes that leverages the GitOps pattern (i.e., storing configuration files in a Git repository to serve as the single source of truth). Instead of developers constantly typing kubectl apply -f manifest.yaml, ArgoCD monitors a specified Git repository for changes to manifests. ArgoCD applications can be configured to automatically update when deltas are detected or require manual intervention. Applications can be created using a GUI or through a YAML file. To get started, we install ArgoCD on the cluster and configure port forwarding to access the UI locally.

# install CLI
brew install argocd
# install ArgoCD on the cluster
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# get initial password
argocd admin initial-password -n argocd
# forward ports to access the ArgoCD UI locally
kubectl port-forward svc/argocd-server -n argocd 8080:443

Once we've navigated to the UI in the browser, we create an ArgoCD application using the following YAML:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-webapp-dev
spec:
  destination:
    name: ''
    namespace: default
    server: https://kubernetes.default.svc
  source:
    path: helm-webapp
    # all credit to devopsjourney1 for the repository
    # https://github.com/devopsjourney1
    # https://www.youtube.com/@DevOpsJourney
    repoURL: https://github.com/scottenriquez/argocd-examples
    targetRevision: HEAD
    helm:
      valueFiles:
        - values-dev.yaml
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: false
      selfHeal: false

Based on the configuration, the ArgoCD application will automatically be updated when we commit to the specified GitHub repository. Via the UI, we can monitor the resources that have been created, sync status, commit information, etc.

23: cdk8s​

The AWS Cloud Development Kit (CDK) is an open-source software development framework that brings the capabilities of general-purpose programming languages (e.g., unit testing, adding robust logic, etc.) to infrastructure as code. In addition to being more ergonomic for those with a software engineering background, CDK also provides higher levels of abstraction through constructs and patterns. HashiCorp also created a spinoff called CDK for Terraform (CDKTF). Using a similar design, AWS created a project called Cloud Development Kit for Kubernetes (cdk8s). Rather than managing the cloud infrastructure, cdk8s only manages the resources within a Kubernetes cluster. The code compiles the TypeScript (or language of your choice) to a YAML manifest file. Below is an example:

export class MyChart extends Chart {
  constructor(scope: Construct, id: string, props: ChartProps = {}) {
    super(scope, id, props);
    new KubeDeployment(this, 'my-deployment', {
      spec: {
        replicas: 3,
        selector: { matchLabels: { app: 'frontend' } },
        template: {
          metadata: { labels: { app: 'frontend' } },
          spec: {
            containers: [
              {
                name: 'app-container',
                image: 'nginx:latest',
                ports: [{ containerPort: 80 }],
              },
            ],
          },
        },
      },
    });
  }
}

Which produces:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-my-deployment-c8e7fb18
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - image: nginx:latest
          name: app-container
          ports:
            - containerPort: 80