These area scripts we use to interact with our deployed boxes, they’re all in the identity-devops repo.

Each script also has a --help with additional documentation.


Before you can access any systems, you will need to set up AWS-vault


These examples are for the IDP app in the sandbox AWS account and the dev environment:

Viewing Secrets

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev

Recommended: grep for the keys you want to check

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev | grep foo
some_foo_key: 'true'

Editing Secrets

The adding --edit will

  • Download the secrets to a tempfile
  • Open your $EDITOR (defaults to vim) to edit that copy
  • Show you a diff of the preview before uploading
  • Clean up the tempfile after
aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --edit
# opens vim
app-s3-secret: Here's a preview of your changes:
>   foobar: 'true'
app-s3-secret: Upload changes to S3? (y/n)

After updating, restart_passenger so that passenger is restarted and will download this updated config without needing to stand up new instances.

Looking at Changes to Secrets

The --last flags lets us look at the last N changes:

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --last 1
Comparing: 2022-09-14 03:59:18 UTC (DtE0w1CVOkRrhxCSUcmFJhPFPsoJI9So)
       to: 2022-09-01 21:01:10 UTC (CRuDO2Ii4UIu14HCSgYj5g85fNUsLAHX)
(no diff)

The --log flag lets us look at all changes like git log

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --env dev --log
Comparing: 2022-09-01 21:01:10 UTC (snK3BVbsNWMW-WhTLO-_RM_M53oI3DMB)
       to: 2022-08-30 20:06:14 UTC (FedolxH-3uevGB_WBcdliPBNx90a1tOK)
<   foo_bar: '["a","b","c"]'
>   foo_bar: '[]'
Comparing: 2022-08-30 20:06:14 UTC (FedolxH-3uevGB_WBcdliPBNx90a1tOK)
       to: 2022-08-30 14:56:21 UTC (6VuhS9TAH0EXtlfr0Ueo3P4QcIPhLAAF.F9Lyz)
>   abc: "123"
Comparing: 2022-08-30 14:56:21 UTC (6VuhS9TAH0EXtlfr0Ueo3P4QcIPhLAAF.F9Lyz)
       to: 2022-08-26 14:08:49 UTC (2fXwjhRjy9pyzlbKijgNbZlqoEyOLBRn)
>   def: "456"

Comparing Secrets Across Environments

The --diff flag lets us compare values across environments (only within the same AWS account)

aws-vault exec sandbox-power -- \
  ./bin/app-s3-secret --app idp --diff dev,int
| key                   | dev       | int       |
| foo_bar_baz           | (null)    | 50        |


Lists servers in an environment as a table

aws-vault exec sandbox-power --
    ./bin/ls-servers -e dev


In the web UI, cloudwatch results are limited to:

  • 15 minutes of time
  • 10,000 results

So to get around that, we have a script that can help split up the query into multiple slices of time and help combine the results, use the --slice to specify different slice durations.

aws-vault exec sandbox-power --
    ./bin/query-cloudwatch \
  --app idp --env dev --log events.log \
  --from 10d --slice 1d --query "$QUERY"

The script can output as new-line delimited JSON (--json) or as a CSV (--csv).


Imitates scp by copying a file in and out of S3. Use the instance ID to refer to remote hosts (see ls-servers to find them). You must be on the VPN for this script to work.

Also note that currently you must prepend the script with bundle exec in order for it to work due to a known bug.

aws-vault exec sandbox-power --
    bundle exec ./bin/scp-s3 i-abcdef1234:/tmp/file.txt ./file.txt


ssm-instance opens an interactive session with a server (EC2 instance) over HTTPS using the SSM Session service. No SSH needed!

-h - Listing Documents

Shows usage plus a list of the available SSM session documents for the application environment.

aws-vault exec sandbox-power --
    ./bin/ssm-instance -h


Looks up the UUID for a user by their email address.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document uuid-lookup --any asg-dev-idp


Activates a user that has a profile deactivated due to a pending ThreatMetrix review status. Requires the user UUID from the uuid-lookup task.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document review-pass --any asg-dev-idp


Deactivates a user that has a pending ThreatMetrix review status with the reason “ThreatMetrix review rejected”. Requires the user UUID from the uuid-lookup task.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document review-reject --any asg-dev-idp


Opens a Rails console (in read-only mode)

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document rails-c --any asg-dev-idp


Opens a Rails console (in read-write mode). Be careful please.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document rails-w --any asg-dev-idp


Tails and streams cloudwatch logs, specifically /var/log/cloud-init-output.log. Useful for checking that a box spins up correctly, such as during a deploy.

aws-vault exec sandbox-power --
    ./bin/ssm-instance --document tail-cw --any asg-dev-idp


ssm-command issues a set of commands (as defined in a “command document”) on one or more servers (EC2 instances) using the SSM Command service.


Running commands on a fleet of servers is inherently risky. It will cut you. There are mild guardrails in ssm-command:

  • By default it runs against 25% of servers at a time (adjustable with the -p or -c flag)
  • It stops when any single command fails (exits with a non-zero status)
  • ssm-command has a hard time dealing with new instances coming online or shutting down in an autoscaling group

-h - Listing Documents

Shows usage plus a list of the available SSM command documents for the application environment.

aws-vault exec sandbox-power --
    ./bin/ssm-command -h


“Safely” restart the NGINX/Passenger service which reloads application.yml from S3.

aws-vault exec sandbox-power --
    ./bin/ssm-command -d passenger-restart -r idp -e dev

If this fails it is recommended that you perform a recycle to ensure all instances are running from the same configuration.


Safely restart GoodJob (idp-workers) service.

aws-vault exec sandbox-power --
    ./bin/ssm-command -d worker-restart -r worker -e dev