Debugging Archives

CircleCI CLI

Ryan Kienstra on July 30, 2022

The CircleCI CLI lets you run a job locally.

So if your build ever fails, you don’t have to:

Guess what’s wrong
Push a commit
Wait for it to fail
Guess what’s wrong again
Push another commit…

For me, this makes CircleCI® the clear leader.

Here’s how to get bash access to the job locally, using the CLI.

So you’ll be able to see what’s wrong.

Real Example

We’re going to debug a failed CI/CD build:

/bin/bash: svn: command not foundCode language: Bash (bash)

The solution isn’t clear, at least to me.

We don’t know:

What package management system does the container use?

What package should we install to run the svn command?

So you’re going to get bash access to debug this.

Start by cloning the example repo, where the failed build was:

$ git clone -b add/wp-org-svn https://github.com/kienstra/adapter-responsive-video
$ cd adapter-responsive-videoCode language: Bash (bash)

Of course, you can use your own repo instead, if there was a failed build there.

Adding Time To Debug

Add this to .circleci/config.yml, right before the step that failed:

diff --git a/.circleci/config.yml b/.circleci/config.yml
index e27995a..1c93a1d 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -9,6 +9,7 @@ jobs:
       - image: cimg/base:current
     steps:
       - checkout
+      - run: sleep 1000
       - run: svn co https://plugins.svn.wordpress.org/adapter-responsive-video --depth=empty .
 
 workflows:Code language: Diff (diff)

This will keep the job running, so you have time to debug it with bash access.

Installing the CircleCI CLI (if you haven’t yet)

Mac:

$ brew install circleciCode language: Bash (bash)

Mac and Linux:

$ curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/master/install.sh | bashCode language: JavaScript (javascript)

This CircleCI CLI will run the job in your local.

Please also ensure Docker is running on your machine.

Running the CLI

$ circleci local execute --job deployCode language: Bash (bash)

This will run the job on your machine:

You might see that pulling some images to run the job.

Getting the Job Image Name

Open a new terminal tab, and run this:

$ docker psCode language: Bash (bash)

That should show the image that’s running the job:

CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS          PORTS     NAMES
b6e09b7009ab   cimg/base:current   "/bin/sh"                13 seconds ago   Up 10 seconds             naughty_jones
b992e2d2c0a8   circleci/picard     "/opt/circleci/linux…"   16 seconds ago   Up 16 seconds             naughty_colden
5c233ff5c3b9   cimg/base:current   "/bin/sh"                6 minutes ago    Up 5 minutes              elastic_brattain
Code language: Bash (bash)

Look for an image right above circleci/picard.

If you don’t see circleci/picard, you’ll probably have to wait 10-20 seconds while the circleci local command above pulls that image.

In this case, it’s the cimg/base:current image.

Copy the container ID of that image to use it below.

In this example, it’s b6e09b7009ab.

That’s the image where your job is running, and you’re going to get bash access to it.

Bash

Then, run this in your terminal to debug the job.

$ docker exec -it b6e09b7009ab /bin/shCode language: Bash (bash)

The argument b6e09b7009ab is the container ID that you got above, from running docker ps.

You should then have bash access to the container:

CircleCI CLI bash access to the container

$ whoami
circleci
$ pwd
/home/circleci/projectCode language: Bash (bash)

Then, you can debug the container, and find out how to install the svn command:

CircleCI CLI

This faster debugging makes CircleCI the leader, in my opinion.

No more guessing and waiting.

You can fix your CI locally.

Now that you’ve debugged your config…

Here’s a way to make it faster and simpler.

How do I debug my CI/CD?

Ryan Kienstra on January 20, 2022

Here’s how I debug my CI/CD.

This is a real example of debugging CircleCI jobs on a GitHub repo.

We’ll debug it locally…

Without pushing commits to the repo and waiting.

First, we notice that CI/CD is failing:

When we click Details, the errors mention node-gyp:

npm ERR!   g++ '-DNODE_GYP_MODULE_NAME=libsass' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1'

From experience, node-gyp has failed when changing the Node version.

Looking at package-lock.json, we’d probably expect the Node version to be 14 or lower.

That’s because the lockfileVersion is 1, which usually infers Node 14 or lower:

So let’s see if the Node version in CI is 14 or lower.

We’ll open VS Code and install the extension Local CI:

It’ll start a free trial automatically, without entering a credit card.

Then, we’ll run the CI job on our local machine.

And enter node --version to see what the version is.

We can run the job without pushing a commit or clicking anything in the repo.

It turns out that the Node version is 16, where it should probably be 14, at least with the current dependencies.

So let’s see if changing the Node version to 14 makes this job pass.

We’ll test this without pushing any code to the repo, or triggering another failed build there.

First, we’ll add an orb for Node:

Orbs are like utility functions for CircleCI, though they’re written in .yml.

And add a node/install step to set Node to 14.18.1:

Then, we’ll commit that change in git.

And we’ll rerun this job locally, no need to push the commit yet:

It failed with an error:

====>> Checkout code
Error: 
Directory (/home/circleci/project) you are trying to checkout to is not empty and not a git repository

Step failed

Maybe there was a problem with the order of the checkout step.

So let’s see if moving the checkout step before the node/install step fixes it:

We’ll rerun the job locally:

It passed!

Now, let’s make sure the next job doesn’t have an obvious problem.

We’ll run the next job, py39:

The first few minutes of the job went well.

So let’s push to the repo, as it looks like CI should pass.

All of the the jobs passed:

The ideal solution might have been to upgrade the package.json dependencies so they work with Node 16 or 17.

But the idea here is to show how to debug CI/CD locally.

This debugging feedback is one of CircleCI‘s huge advantages.

No endless cycle of guessing, pushing, waiting for it to fail, guessing, pushing…

Now that you’ve seen how I debug my CI/CD, here’s a fast way to speed up your builds.

CI Builds: Commit And Wait

Ryan Kienstra on July 25, 2021

Your CI build fails…

And it’s the end of your work day.

There’s a cryptic error message.

So you make a commit with pwd && ls

And wait for it to fail again.

10 minutes later, it fails again, with those logging messages.

You guess it’s because Node is on the wrong version.

So you push a commit to update it to 15.

And wait for 10 more minutes.

It still fails, but with a Jest error.

It’s passing locally, so you’re not sure why.

You guess that the state is persisting after the tests.

So you add an afterEach() callback to clean it up.

After pushing that, you wait 10 more minutes for it to fail.

After repeating this 4 more times, the CI build finally passes.

You request a review on the PR.

And wonder if there’s a better way…

Coding Can Be Like Flying Blind

Ryan Kienstra on July 25, 2021

You have to open a PR in an unfamiliar language.

Maybe Python.

It seems easy enough, just a few changes.

So you volunteered for it in Sprint Planning.

You make a few edits and run the script locally.

But it doesn’t work.

Ah, maybe because it’s using a tuple.

So you add a print() to see what its value is, because you don’t remember how to set up debug breakpoints in Python.

It looks completely different than you thought.

It’s like flying blind.

You’re thinking of using a dictionary instead.

But how do dictionaries work in Python, again?

You make a few edits, and it fails again.

So you add more print statements.

Late that afternoon, it finally works,

After hours of guessing and struggling.

In the languages you know…

This wouldn’t be a problem.

You would know how to debug them.

But here, you’re flying blind.

Category: Debugging

CircleCI CLI

Real Example

Adding Time To Debug

Installing the CircleCI CLI (if you haven’t yet)

Running the CLI

Getting the Job Image Name

Bash

CircleCI CLI

Be the first to get CI/CD tips like this

How do I debug my CI/CD?

Be the first to get CI/CD tips like this

CI Builds: Commit And Wait

Be the first to get CI/CD tips like this

Coding Can Be Like Flying Blind

Be the first to get CI/CD tips like this