bit bland

Debugging wandering IPs with arping

2023-05-05T07:57:23+00:00

We have a number of on premises Kubernetes clusters on virtual hardware which utilise MetalLB for load balancing. New configuration options enables the coupling of specific IP address pools to named Kubernetes nodes, this functionality prompted an upgrade.

During the upgrade of a specific cluster our services started flapping. We noticed IPs in the range were constantly being reassigned to different nodes whilst also intermittently failing, here demonstrated by sshing into the node via the IP assigned to it by MetalLB and retrieving the hostname:

(code examples are for the fish shell)

for ip in (host vip-pool.hostname | grep address | awk '{print $4}' | sort) ; 
    echo -n "$ip   " && ssh -o ConnectTimeout=2 -o "UserKnownHostsFile=/dev/null" \
        -o LogLevel=ERROR root@$ahost "cat /etc/hostname"; 
end;
10.34.11.190   kubenode001
10.34.11.191   kubenode002
10.34.11.192   ssh: connect to host 10.34.11.192 port 22: Connection timed out
10.34.11.193   ssh: connect to host 10.34.11.193 port 22: Connection timed out
10.34.11.194   ssh: connect to host 10.34.11.194 port 22: Connection timed out
10.34.11.195   kubenode003
10.34.11.196   kubenode004
10.34.11.197   kubenode005
10.34.11.198   ssh: connect to host 10.34.11.198 port 22: Connection timed out
10.34.11.199   kubenode006

# and again

for ip in (host vip-pool.hostname | grep address | awk '{print $4}' | sort) ;
    echo -n "$ip   " && ssh -o ConnectTimeout=2 -o "UserKnownHostsFile=/dev/null" \
        -o LogLevel=ERROR root@$ahost "cat /etc/hostname";                                    
end;
10.34.11.190   ssh: connect to host 10.34.11.190 port 22: Connection timed out
10.34.11.191   ssh: connect to host 10.34.11.191 port 22: Connection timed out
10.34.11.192   kubenode001
10.34.11.193   kubenode002
10.34.11.194   ssh: connect to host 10.34.11.194 port 22: Connection timed out
10.34.11.195   kubenode003
10.34.11.196   ssh: connect to host 10.34.11.196 port 22: Connection timed out
10.34.11.197   kubenode004
10.34.11.198   ssh: connect to host 10.34.11.198 port 22: Connection timed out
10.34.11.199   ssh: connect to host 10.34.11.199 port 22: Connection timed out

It seems IP addresses are constantly being reassigned to different nodes, and this should happen, according to the docs because

. . . the leader node [has] failed for some reason, failover is [then] automatic: the failed node is detected using memberlist, at which point new nodes take over ownership of the IP addresses from the failed node.

Let’s investigate further with arping

for suffix in $(seq 190 199) ; do arping -I ens192 -c 1 10.34.11.$suffix; done;
ARPING 10.34.11.190 from 10.34.11.230 ens192
Unicast reply from 10.34.11.190 [00:50:56:AB:47:53]  0.695ms
Unicast reply from 10.34.11.190 [00:50:56:AB:62:40]  0.730ms
Unicast reply from 10.34.11.190 [00:50:56:AB:D4:6D]  0.740ms
Unicast reply from 10.34.11.190 [00:50:56:AB:89:59]  0.837ms
Unicast reply from 10.34.11.190 [00:50:56:AB:7A:9F]  0.872ms
Sent 1 probes (1 broadcast(s))
Received 5 response(s)
ARPING 10.34.11.191 from 10.34.11.230 ens192
Unicast reply from 10.34.11.191 [00:50:56:AB:62:40]  0.700ms
Unicast reply from 10.34.11.191 [00:50:56:AB:47:53]  0.745ms
Unicast reply from 10.34.11.191 [00:50:56:AB:D4:6D]  0.753ms
Unicast reply from 10.34.11.191 [00:50:56:AB:89:59]  0.827ms
Unicast reply from 10.34.11.191 [00:50:56:AB:86:6B]  0.852ms
Sent 1 probes (1 broadcast(s))
Received 5 response(s)
ARPING 10.34.11.192 from 10.34.11.230 ens192
Unicast reply from 10.34.11.192 [00:50:56:AB:47:53]  0.643ms
Unicast reply from 10.34.11.192 [00:50:56:AB:62:40]  0.666ms
Unicast reply from 10.34.11.192 [00:50:56:AB:D4:6D]  0.732ms
Unicast reply from 10.34.11.192 [00:50:56:AB:89:59]  0.800ms
Unicast reply from 10.34.11.192 [00:50:56:AB:38:AC]  0.981ms
Unicast reply from 10.34.11.192 [00:50:56:AB:A4:FC]  1.034ms 
...

Here we have responses coming from multiple interfaces for the same IP. Grepping for the above MAC addresses in the output of ip a on the cluster nodes revealed many were for unconfigured virtual interfaces, no wonder our services were flapping. By default MetalLB will attempt to broadcast on every interface, up or down. To disable the default behaviour we need to augment our L2Advertisement Custom Resource configuration to specify the interfaces we want to use, resulting in a stable service:

for ip in (host vip-pool.hostname | grep address | awk '{print $4}' | sort) ;
    echo -n "$ip   " && ssh -o ConnectTimeout=2 -o "UserKnownHostsFile=/dev/null" \
        -o LogLevel=ERROR root@$ahost "cat /etc/hostname";
end;
10.34.11.190   kubenode001
10.34.11.191   kubenode002
10.34.11.192   kubenode003
10.34.11.193   kubenode004
10.34.11.194   kubenode005
10.34.11.195   kubenode006
10.34.11.196   kubenode007
10.34.11.197   kubenode008
10.34.11.198   kubenode009
10.34.11.199   kubenode010

Simulating –dry-run and –patch in Hashicorp Vault

2022-06-27T14:32:31+00:00

Hashicorp Vault is a great resource for keeping secrets secret. However version 1 of the KV secrets engine lacked the ability to patch a particular endpoint which made modifying existing values fraught with danger. We mitigate this by utilising the ever wonderful jq alongside a little used Vault flag to perform inline modifcations before similuating writes.

Let’s write some test data and read it back again in sequentially more useful ways:

vault write /secret/test/smchm testkey=testvalue
Success! Data written to: secret/test/smchm

vault read /secret/test/smchm
Key                 Value
---                 -----
refresh_interval    768h
testkey             testvalue

vault read --format=json /secret/test/smchm | jq .
{
  "request_id": "087490c9-ccad-e225-cd17-9f88df12abd5",
  "lease_id": "",
  "lease_duration": 2764800,
  "renewable": false,
  "data": {
    "testkey": "testvalue"
  },
  "warnings": null
}

vault read --format=json /secret/test/smchm | jq .data
{
  "testkey": "testvalue"
}

Now lets patch the data with jq, the resultant patched data is not yet stored in Vault:

vault read --format=json /secret/test/smchm | jq '.data += { "testkey": "newvalue" }' | jq .data
{
  "testkey": "newvalue"
}

We can also add values:

vault read --format=json /secret/test/smchm | jq '.data += { "testkey": "newvalue", "testkey2": "testvalue2" }' | jq .data
{
  "testkey": "newvalue",
  "testkey2": "testvalue2"
}

Now we have the values we want in our JSON object we can write them back to our Vault endpoint. However first we can simulate the write by performing a dry-run using the Vault flag –output-curl-string to make sure everything is as we expect:

vault read --format=json /secret/test/smchm | jq '.data += { "testkey": "newvalue", "testkey2": "testvalue2" }' | jq .data | vault write -output-curl-string /secret/test/smchm - 
curl -X PUT -H "X-Vault-Token: $(vault print token)" -d '{"testkey":"newvalue","testkey2":"testvalue2"}' https://ourvaulthost:8200/v1/secret/test/smchm

which can be a little difficult to grok for endpoints with many values, so lets parse the output with sed and jq:

vault read --format=json /secret/test/smchm | jq '.data += { "testkey": "newvalue", "testkey2": "testvalue2" }' | jq .data | vault write -output-curl-string /secret/test/smchm - | sed 's/^.*{/{/' | sed 's/}.*/}/' | jq .
{
  "testkey": "newvalue",
  "testkey2": "testvalue2"
}

looks good, so finally lets stream our values back to Vault to patch our endpoint and validate the results:

vault read --format=json /secret/test/smchm | jq '.data += { "testkey": "newvalue", "testkey2": "testvalue2" }' | jq .data | vault write /secret/test/smchm -
Success! Data written to: secret/test/smchm

vault read --format=json /secret/test/smchm | jq .data
{
  "testkey": "newvalue",
  "testkey2": "testvalue2"
}

Of course this doesn’t prevent us writing to the wrong endpoint!

All your passwords are Troy’s

2022-06-07T07:57:23+00:00

Our institution is re-opening the age old debate about password hygiene. One of the topics under discussion is whether to introduce expiry conditions to limit password longevity. A bad idea. Instead store the hashes and periodically check Troy Hunt’s excellent database of compromised credentials via the API. The example below generates the hash from the passed argument so not something which can be used in production:

#!/bin/bash

PASSWD="${1:-password}"
SHAPREFIX="$(echo -n $PASSWD | sha1sum | cut -c1-5)"
SHASUFFIX="$(echo -n $PASSWD | sha1sum | cut -c 6- | cut -f1 -d" ")"
PWNED=$( \
  curl -Ss https://api.pwnedpasswords.com/range/$SHAPREFIX | grep -i $SHASUFFIX \
)
if (test $PWNED)
  then echo "pwned"
  else echo "not pwned"  
fi

and to use:

[smchm@home ~]$ have-i-been-pwned.sh 'passwd'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'not-in-the-database-as-it-is-too-long-but-reasonably-easy-to-remember'
not pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password1234567890'
pwned

or perhaps more interestingly:

[smchm@home ~]$ have-i-been-pwned.sh 'password'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password123'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password123123'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password123123123'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password123123123123'
pwned
[smchm@home ~]$ have-i-been-pwned.sh 'password123123123123123'
not pwned

JUnit5 test exclusions based on Spring profile names

2022-06-01T13:30:48+00:00

To prevent some JUnit5 tests from running when specific Spring Profiles are active simply implement the ExecutionCondition interface and interrogate the spring.profiles.active system property;

package com.example.project.config;

import org.junit.jupiter.api.extension.ConditionEvaluationResult;
import org.junit.jupiter.api.extension.ExecutionCondition;
import org.junit.jupiter.api.extension.ExtensionContext;

public class DisabledInProductionExtension implements ExecutionCondition {
  @Override
  public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext extensionContext) {
    final String springProfileKey = "spring.profiles.active";
    if (System.getProperties().containsKey(springProfileKey)) {
      if (System.getProperties().get(springProfileKey).toString().contains("integration-test-production")) {
        return ConditionEvaluationResult.disabled("Test disabled in production environment");
      }
      return ConditionEvaluationResult.enabled("Test enabled in non-production environment");
    }
    return ConditionEvaluationResult.enabled("No profile active, test enabled");
    }
}

And to use the Extension

package com.example.project;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import com.example.project.config.DisabledInProductionExtension;

@ExtendWith({DisabledInProductionExtension.class})
public class IntegrationTestForNonProductionEnvironment {
}

To define multiple Extensions for different profiles we can create complimentary ExecutionCondition implementations. The framework will chain the invocations until one evaluates to disabled and exclude the annotated test accordingly.

package com.example.project;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import com.example.project.config.DisabledInProductionExtension;
import com.example.project.config.DisabledInStagingExtension;

@ExtendWith({DisabledInProductionExtension.class, DisabledInStagingExtension.class})
public class IntegrationTestForDevelopmentEnvironment {
}