Some jq tricks

Posted on

Learned a few things today, thought I’d post them online as code snippets.

I was using Terraform to manage some infrastructure, and some resource was giving me grief - breaking the program and giving an unbeatably vague error message when I tried to do a plan involving it. Unfortunately the TF state file was over a thousand lines, and so it was hard to track down the resource in question. It was hard to isolate individual resources in such a big file.

Each domain name creates four kinds of resources from each zone, so that meant working with four separate arrays in the JSON.1 So: I needed to manipulate the 4 kinds of resources created from each zone, directly in tfstate, and remove everything except the broken part to know what’s causing the problem. Processing JSON? Sounds like a task for the sublime jq.

Setup

Given a tfstate file like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{"resources": [
  {something boring},
  {"name": "parking_dkim",
   "instances": [
      {"index_key": "lord.geek.nz", ...},
      {"index_key": "dal-corp.com", ...},
      ... lots more instances here
    ]},
  {"name": "parking_dmarc",
    "instances": [
       {"index_key": "lord.geek.nz", ...},
       {"index_key": "dal-corp.com", ...},
      ... lots more instances here
     ]}
}

Back up the original file:

1
cp prod.tfvars prod_full.tfvars

and then run the following examples like this:

1
jq -f example.jq < prod_full.tfvars > prod.tfvars

Let’s get into it.

Binary search functions vs an array of numbers

Make your own functions for a binary search by chaining these:

1
2
3
4
5
6
7
8
9
10
11
def right_half(addr):
  del(addr[0:(addr|length)/2]);
def left_half(addr):
  del(
    addr[
      ((addr|length)/2 | floor) : ]
  );

[range(1; 18)] |
right_half(.) |
left_half(.)

The example starts with the array [1..18], then takes the right half. It slices that up and takes the left half of what’s left, returning [10, 11, 12, 13].

With these functions in hand, you can apply them to do your own binary search in tfstate. Narrow down and see what’s breaking it.

Binary search on the Terraform state

Try modifying the state by removing half of it, then run Terraform again, and see if it breaks:

1
2
3
4
5
6
7
8
9
def right_half(addr):
  del(addr.instances[0:(addr.instances|length)/2]);
def left_half(addr):
  del(
    addr.instances[
      ((addr.instances|length)/2 | floor) : ]
  );

left_half(.resources[1,2,3,4])

If that works fine (no error), try the right half instead:

1
2
# declarations as above
right_half(.resources[1,2,3,4])

Say that showed the error, then you’d subdivide it:

1
2
3
# declarations as above
right_half(.resources[1,2,3,4]) |
left_half(.resources[1,2,3,4])

If that shows the error, great, add another | left_half to keep narrowing it down. If not, try | right_half again. Keep doing this until you’ve found one set of resources corresponding to one domain. Tada, you’ve done it.

Once you know what you’re looking for, or only a couple, you can try another approach to only keep specific domains’ resources and discard the rest.

Another approach: filtering elements based on sub-elements

This syntax is super useful, using the update operator with map and select.

You can filter members of the resources array by whether their index_key has one of the zones you want.

1
2
["lord.geek.nz", "dal-corp.com", "angry.nerds"] as $targets |
.resources[1,2,3,4].instances |= map(select([.index_key] | inside($targets))) |

Wrapping up

This is not the best explanation I can write (promise!) but I hope these code samples come in handy for you.

The jq manual is packed full of useful information. It’s usually a better place to start than StackOverflow.

  1. Hashicorp discourage you from treating it as JSON, but if I respected warning labels, I wouldn’t have eaten all those paint chips as a kid. Lead is an all-natural sweetener and sometimes you’ve just gotta hack some state. ↩