Scripts

These area scripts we use to interact with our deployed boxes, they’re all in the identity-devops repo.

Each script also has a --help with additional documentation.

Prerequisites

Before you can access any systems, you will need to set up AWS-vault

app-s3-secret

These examples are for the IDP app in the sandbox AWS account and the dev environment:

Viewing Secrets

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev

Recommended: grep for the keys you want to check

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev | grep foo
some_foo_key: 'true'

Editing Secrets

The adding --edit will

• Download the secrets to a tempfile
• Open your $EDITOR (defaults to vim) to edit that copy
• Show you a diff of the preview before uploading
• Clean up the tempfile after

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --edit
#
# opens vim
#
app-s3-secret: Here's a preview of your changes:
2a3
>   foobar: 'true'
app-s3-secret: Upload changes to S3? (y/n)
y

After updating, restart_passenger so that passenger is restarted and will download this updated config without needing to stand up new instances.

Looking at Changes to Secrets

The --last flags lets us look at the last N changes:

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --last 1
Comparing: 2022-09-14 03:59:18 UTC (DtE0w1CVOkRrhxCSUcmFJhPFPsoJI9So)
       to: 2022-09-01 21:01:10 UTC (CRuDO2Ii4UIu14HCSgYj5g85fNUsLAHX)
(no diff)

The --log flag lets us look at all changes like git log

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --log
Comparing: 2022-09-01 21:01:10 UTC (snK3BVbsNWMW-WhTLO-_RM_M53oI3DMB)
       to: 2022-08-30 20:06:14 UTC (FedolxH-3uevGB_WBcdliPBNx90a1tOK)
59c59
<   foo_bar: '["a","b","c"]'
---
>   foo_bar: '[]'
Comparing: 2022-08-30 20:06:14 UTC (FedolxH-3uevGB_WBcdliPBNx90a1tOK)
       to: 2022-08-30 14:56:21 UTC (6VuhS9TAH0EXtlfr0Ueo3P4QcIPhLAAF.F9Lyz)
75a76
>   abc: "123"
Comparing: 2022-08-30 14:56:21 UTC (6VuhS9TAH0EXtlfr0Ueo3P4QcIPhLAAF.F9Lyz)
       to: 2022-08-26 14:08:49 UTC (2fXwjhRjy9pyzlbKijgNbZlqoEyOLBRn)
59a60
>   def: "456"

Comparing Secrets Across Environments

The --diff flag lets us compare values across environments (only within the same AWS account)

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --diff dev,int
+-----------------------+-----------+-----------+
| key                   | dev       | int       |
+-----------------------+-----------+-----------+
| foo_bar_baz           | (null)    | 50        |

ls-servers

Lists servers in an environment as a table

aws-vault exec sandbox-power --
    ./bin/ls-servers -e dev

query-cloudwatch

In the web UI, cloudwatch results are limited to:

• 15 minutes of time
• 10,000 results

So to get around that, we have a script that can help split up the query into multiple slices of time and help combine the results, use the --slice to specify different slice durations.

aws-vault exec sandbox-power --
    ./bin/query-cloudwatch \
  --app idp --env dev --log events.log \
  --from 10d --slice 1d --query "$QUERY"

The script can output as new-line delimited JSON (--json) or as a CSV (--csv).

scp-s3

Imitates scp by copying a file in and out of S3. Use the instance ID to refer to remote hosts (see ls-servers to find them). You must be on the VPN for this script to work.

Also note that currently you must prepend the script with bundle exec in order for it to work due to a known bug.

aws-vault exec sandbox-power --
    bundle exec ./bin/scp-s3 i-abcdef1234:/tmp/file.txt ./file.txt

ssm-instance

ssm-instance opens an interactive session with a server (EC2 instance) over HTTPS using the SSM Session service. No SSH needed!

-h - Listing Documents

Shows usage plus a list of the available SSM session documents for the application environment.

aws-vault exec sandbox-power --
    ./bin/ssm-instance -h

uuid-lookup

Looks up the UUID for a user by their email address.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document uuid-lookup --any asg-dev-idp

review-pass

Activates a user that has a profile deactivated due to a pending ThreatMetrix review status. Requires the user UUID from the uuid-lookup task.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document review-pass --any asg-dev-idp

review-reject

Deactivates a user that has a pending ThreatMetrix review status with the reason “ThreatMetrix review rejected”. Requires the user UUID from the uuid-lookup task.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document review-reject --any asg-dev-idp

rails-c

Opens a Rails console (in read-only mode)

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document rails-c --any asg-dev-idp

rails-w

Opens a Rails console (in read-write mode). Be careful please.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document rails-w --any asg-dev-idp

tail-cw

Tails and streams cloudwatch logs, specifically /var/log/cloud-init-output.log. Useful for checking that a box spins up correctly, such as during a deploy.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document tail-cw --any asg-dev-idp

ssm-command

ssm-command issues a set of commands (as defined in a “command document”) on one or more servers (EC2 instances) using the SSM Command service.

HAZARD WARNING

Running commands on a fleet of servers is inherently risky. It will cut you. There are mild guardrails in ssm-command:

• By default it runs against 25% of servers at a time (adjustable with the -p or -c flag)
• It stops when any single command fails (exits with a non-zero status)
• ssm-command has a hard time dealing with new instances coming online or shutting down in an autoscaling group

-h - Listing Documents

Shows usage plus a list of the available SSM command documents for the application environment.

aws-vault exec sandbox-power --
    ./bin/ssm-command -h

passenger-restart

“Safely” restart the NGINX/Passenger service which reloads application.yml from S3.

aws-vault exec sandbox-power --
    ./bin/ssm-command -d passenger-restart -r idp -e dev

If this fails it is recommended that you perform a recycle to ensure all instances are running from the same configuration.

worker-restart

Safely restart GoodJob (idp-workers) service.

aws-vault exec sandbox-power --
    ./bin/ssm-command -d worker-restart -r worker -e dev