YMMV; as of June 2023, AWS's 1GB limit on cloud shell persistent storage is too low for real world proxy deployments, which typically require install gcloud CLI / Azure CLI to connect to sources

So use use your local machine, or a VM/container elsewhere in AWS (EC2, AWS Cloud9, etc

1. clone the repo

git clone https://github.com/Worklytics/psoxy.git

1. add the following lines to your ~/.bashrc. (AWS Cloud Shell preserves only your HOME directory across sessions, so add any commands that modify/install things outside to your .bashrc)

# install Maven (and, via dependency, java)
sudo yum -y install maven

# GCloud SDK (if using Google Workspace data sources)
# The next line updates PATH for the Google Cloud SDK.
if [ -f '/home/cloudshell-user/google-cloud-sdk/path.bash.inc' ]; then . '/home/cloudshell-user/google-cloud-sdk/path.bash.inc'; fi

# The next line enables shell command completion for gcloud.
if [ -f '/home/cloudshell-user/google-cloud-sdk/completion.bash.inc' ]; then . '/home/cloudshell-user/google-cloud-sdk/completion.bash.inc'; fi

Then source ~/.bashrc, to execute the above.

1. install Terraform

git clone https://github.com/tfutils/tfenv.git ~/.tfenv
mkdir ~/bin
ln -s ~/.tfenv/bin/* ~/bin/
tfenv install
tfenv use latest

1. if using Google Workspace data sources, install Google Cloud CLI and authenticate.

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-435.0.1-linux-x86_64.tar.gz
tar -xvf google-cloud-cli-435.0.1-linux-x86_64.tar.gz
sudo ./google-cloud-sdk/install.sh
rm google-cloud-cli-435.0.1-linux-x86_64.tar.gz

1. if using Microsoft 365 data sources, install Azure CLI and authenticate.

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

You should now be ready for the general instructions in the README.md.

Other stuff

If default NodeJS tooling doesn't work for you, legacy testing tools use python/awscurl, installed via pip. See example below:

# install AWS Curl (used for testing)
sudo yum -y install pip
pip install awscurl

These shell command examples presume Ubuntu; you may need to translate to your *nix variant. If you starting with a fairly rich environment, many of these tools may already be on your machine.

1. install dependencies

sudo apt update

1. install Java + maven (required to build the proxy binary to be deployed)

sudo apt install openjdk-11-jdk
sudo apt install maven

# check that maven version at least 3.6+ and java 11+
mvn -v

# if not, get latest direct from Apache Maven
# https://maven.apache.org/install.html

1. install Terraform

Follow Terraform's install guide (recommended) or, if you need to manage multiple Terraform versions, use tfenv:

git clone https://github.com/tfutils/tfenv.git ~/.tfenv
mkdir ~/bin
sudo apt install unzip
ln -s ~/.tfenv/bin/* ~/bin/
tfenv install
tfenv use latest

1. if you're deploying in AWS, install the AWS CLI

sudo apt install awscli

1. if you want to test an AWS deployment, install AWS Curl (which requires python 3.6+ and pip)

# check python version; please ensure it's at least 3.6+
python --version

# if not 3.6+, get latest direct for your environment from Python.org

install pip (likely included with fresh python install), then use that to install awscurl

sudo apt install pip
pip install awscurl

1. if deploying to GCP or using Google Workspace data sources, install Google Cloud CLI and authenticate.

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-384.0.0-linux-x86_64.tar.gz
tar -xvf google-cloud-cli-384.0.0-linux-x86_64.tar.gz
sudo ./google-cloud-sdk/install.sh

# if prompted for location of your .bashrc by the gcloud install script, on EC2 it's '/home/ubuntu/.bashrc'

rm google-cloud-cli-384.0.0-linux-x86_64.tar.gz

# source your .bashrc OR restart your terminal so gcloud is found on your $PATH
source ~/.bashrc

# authenticate with Google Cloud CLI
gcloud auth application-default login --no-launch-browser

1. if using Microsoft 365 data sources, install Azure CLI and authenticate.

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

You should now be ready for the general instructions in the README.md.

Modes

Psoxy can be deployed/used in 3 different modes:

• API - psoxy sits in front of a data source API. Any call that would normally be sent to the data source API is instead sent to psoxy, which parses the request, validates it / applies ACL, and adds authentication before forwarding to the host API. After the host API response, psoxy sanitizes the response as defined by its roles before returning the response to the caller. This is an http triggered flow.

• Bulk File - psoxy is triggered by files (objects) being uploaded to cloud storage buckets (eg, S3, GCS, etc). Psoxy reads the incoming file, applies one or more sanitization rules (transforms), writing the result(s) to a destination (usually in distinct bucket).

• Command-line (cli) - psoxy is invoked from the command-line, and is used to sanitize data stored in files on the local machine. This is useful for testing, or for one-off data sanitization tasks. Resulting files can be uploaded to Worklytics via the file upload of its web portal.

Prereqs

Required:

Optional:

• AWS SAM CLI () for local testing, if desired

• for direct testing of deployed AWS lambda from a terminal

Build

Maven build produces a zip file.

1. Build core library

2. From java/impl/aws/:

Run Locally

Locally, you can test function's behavior from invocation on a JSON payload (but not how the API gateway will map HTTP requests to that JSON payload):

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html

Deploy to AWS

We recommend deploying your Psoxy code into AWS using the terraform modules found in [infra/modules/](../../infra/modules/] for AWS. These modules both provision the required AWS infrastructure, as well as deploying the built binaries for Psoxy as lambdas in the target account.

You'll ultimately provision infrastructure represented in green in the following diagram:


If you use psoxy to send pseudonymized data to Worklytics and later wish to re-identify the data that you export from Worklytics on your premises, you'll need a lookup table in your data warehouse to JOIN to data exported from Worklytics.

In our various examples (see ), we provide a variable lookup_table_builder that allows you to control generation of these lookup tables. Populating this variable will generate another version of your HRIS data (aside from the one exposed to Worklytics) which you can then import back to your data warehouse.

The S3 bucket in which this table, if any, will be shown as a Terraform output. eg

Use the bucket name shown in your output to build import pipeline to your data warehouse.

Overview

You'll provision the following to host Psoxy in AWS:

• Lambda Functions

• IAM Roles and Policies

• System Manager Parameter Store Parameters

• Cloud Watch Log Groups

• S3 buckets, if using the 'bulk' mode to sanitize file data (such as CSVs)

• Cognito Pools and Identities, if connecting to Microsoft 365 (Azure AD) data sources

The diagram below provides an architecture overview of the 'REST' and 'Bulk' use-cases.

Prerequisites

1. An AWS Account in which to deploy Psoxy We strongly recommend you provision one specifically for use to host Psoxy, as this will create an implicit security boundary, reduce possible conflicts with other infra configured in the account, and simplify eventual cleanup.

   You will need the numeric AWS Account ID for this account, which you can find in the AWS Console.

   If your AWS organization enforces Service Control Policies, ensure that these are allow the AWS components required by Psoxy or exempt the AWS Account in which you will deploy Psoxy from these policies.
2. A sufficiently privileged AWS Role You must have a IAM Role within the AWS account with sufficient privileges to (AWS managed policy examples linked):

   (Yes, the use of AWS Managed Policies results in a role with many privileges; that's why we recommend you use a dedicated AWS account to host proxy which is NOT shared with any other use case)

   You will need the ARN of this role.
3. An authenticated AWS CLI in your provisioning environment. Your environment (eg, shell/etc from which you'll run terraform commands) must be authenticated as an identity that can assume that role. (see next section for tips on options for various environments you can use)

   Eg, if your Role is arn:aws:iam::123456789012:role/PsoxyProvisioningRole, the following should work:

Provisioning Environment

To provision AWS infra, you'll need the aws-cli installed and authenticated on the environment where you'll run terraform.

Here are a few options:

Your Local Machine or a VM/Container Outside AWS

2. Run aws configure in a terminal on the machine you plan to use, and configure it with the key you generated in step one.

EC2 Instance

If your organization prefers NOT to authorize the AWS CLI on individual laptops and/or outside AWS, provisioning Psoxy's required infra from an EC2 instance may be an option.

1. provision an EC2 instance (or request that your IT/dev ops team provision one for you). We recommend a micro instance with an 8GB disk, running ubuntu (not Amazon Linux; if you choose that or something else, you may need to adapt these instructions). Be sure to create a PEM key to access it via SSH (unless your AWS Organization/account provides some other ssh solution).

2. associate the Role above with your instance (see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)

Terraform State Backend

S3

You'll also need a backend location for your Terraform state (such as an S3 bucket). It can be in any AWS account, as long as the AWS role that you'll use to run Terraform has read/write access to it.

See https://developer.hashicorp.com/terraform/language/settings/backends/s3 for more details.

Local File System

Alternatively, you may use a local file system, but this is not recommended for production use - as your Terraform state may contain secrets such as API keys, depending on the sources you connect.

See https://developer.hashicorp.com/terraform/language/settings/backends/local

Bootstrap

Getting Started

Once you've fulfilled the prereqs, including having your terraform deployment environment, backend, and AWS account prepared, we suggest you use our AWS example template repo:

• https://github.com/Worklytics/psoxy-example-aws

Follow the 'Usage' instructions there to continue.

A serverless, pseudonymizing, DLP layer between Worklytics and the REST API of your data sources.

Psoxy replaces PII in your organization's data with hash tokens to enable Worklytics's analysis to be performed on anonymized data which we cannot map back to any identifiable individual.

Psoxy is a pseudonymization service that acts as a Security / Compliance layer, which you can deploy between your data sources (SaaS tool APIs, Cloud storage buckets, etc) and the tools that need to access those sources.

Psoxy ensures more secure, granular data access than direct connections between your tools will offer - and enforces access rules to fulfill your Compliance requirements.

Psoxy functions as API-level Data Loss Prevention layer (DLP), by blocking sensitive fields / values / endpoints that would otherwise be exposed when you connect a data sources API to a 3rd party service. It can ensure that data which would otherwise be exposed to a 3rd party service, due to granularity of source API models/permissions, is not accessed or transfered to the service.

Objectives:

• serverless - we strive to minimize the moving pieces required to run psoxy at scale, keeping your attack surface small and operational complexity low. Furthermore, we define infrastructure-as-code to ease setup.

• transparent - psoxy's source code is available to customers, to facilitate code review and white box penetration testing.

• simple - psoxy's functionality will focus on performing secure authentication with the 3rd party API and then perform minimal transformation on the response (pseudonymization, field redaction) to ease code review and auditing of its behavior.

Psoxy may be hosted in or .

Data Flow

A Psoxy instances reside on your premises (in the cloud) and act as an intermediary between Worklytics and the data source you wish to connect. In this role, the proxy performs the authentication necessary to connect to the data source's API and then any required transformation (such as pseudonymization or redaction) on the response.

Orchestration continues to be performed on the Worklytics side.

Source API data may include PII such as:

But Psoxy ensures Worklytics only sees:

These pseudonyms leverage SHA-256 hashing / AES encryption, with salt/keys that are known only to your organization and never transferred to Worklytics.

For data sources APIs which require keys/secrets for authentication, such values remain stored in your premises and are never accessible to Worklytics.

You authorize your Worklytics tenant to access your proxy instance(s) via the IAM platform of your cloud host.

Supported Data Sources

As of March 2023, the following sources can be connected to Worklytics via psoxy.

Note: Some sources require specific licenses to transfer data via the APIs/endpoints used by Worklytics, or impose some per API request costs for such transfers.

Google Workspace (formerly GSuite)

If you use our provided Terraform modules, specific instructions that you can pass to the Google Workspace Admin will be output for you.

NOTE: 'Google Directory' connection is required prerequisite for all other Google Workspace connectors.

NOTE: you may need to enable the various Google Workspace APIs within the GCP project in which you provision the OAuth Clients. If you use our provided terraform modules, this is done automatically.

Microsoft 365

Other Data Sources via REST APIs

These sources will typically require some kind of "Admin" within the tool to create an API key or client, grant the client access to your organization's data, and provide you with the API key/secret which you must provide as a configuration value in your proxy deployment.

The API key/secret will be used to authenticate with the source's REST API and access the data.

Other Data Sources without REST APIs

Other data sources, such as Human Resource Information System (HRIS), Badge, or Survey data can be exported to a CSV file. The "bulk" mode of the proxy can be used to pseudonymize these files by copying/uploading the original to a cloud storage bucket (GCS, S3, etc), which will trigger the proxy to sanitize the file and write the result to a 2nd storage bucket, which you then grant Worklytics access to read.

Getting Started - Customers

Host Platform and Data Sources

The prequisites and dependencies you will need for Psoxy are determined by:

1. Where you will host psoxy? eg, Amazon Web Services (AWS), or Google Cloud Platform (GCP)

2. Which data sources you will connect to? eg, Microsoft 365, Google Workspace, Zoom, etc, as defined in previous sections.

Once you've gathered that information, you can identify the required software and permissions in the next section, and the best environment from which to deploy Psoxy.

Prerequisites

At a high-level, you need 3 things:

1. a cloud host platform account to which you will deploy Psoxy (eg, AWS account or GCP project)

2. an environment on which you will run the deployment tools (usually your laptop)

3. some way to authenticate that environment with your host platform as an entity with sufficient permissions to perform the deployment. (usually an AWS IAM Role or a GCP Service Account, which your personal AWS or Google user can assume).

You, or the IAM Role / GCP Service account you use to deploy Psoxy, usually does NOT need to be authorized to access or manage your data sources directly. Data access permissions and steps to grant those vary by data source and generally require action to be taken by the data source administrator AFTER you have deployed Psoxy.

Required Software and Permissions

As of Feb 2023, Psoxy is implemented with Java 11 and built via Maven. The proxy infrastructure is provisioned and the Psoxy code deployed using Terraform, relying on Azure, Google Cloud, and/or AWS command line tools.

You will need all the following in your deployment environment (eg, your laptop):

NOTE: we will support Java versions for duration of official support windows, in particular the LTS versions. As of Nov 2023, we still support java 11 but may end this at any time. Minor versions, such as 12-16, and 18-20, which are out of official support, may work but are not routinely tested.

NOTE: Refrain to use Terraform versions 1.4.x that are < v1.4.3. We've seen bugs.

Depending on your Cloud Host / Data Sources, you will need:

For testing your psoxy instance, you will need:

NOTE: NodeJS 16 is unmaintained since Oct 2023, so we recommend newer version; but in theory should work.

Setup

1. Choose the cloud platform you'll deploy to, and follow its 'Getting Started' guide:
2. Based on that choice, pick from the example template repos below. Use your choosen option as a template to create a new GitHub repo, or if you're not using GitHub Cloud, create clone/fork of the choosen option in your source control system:

   • AWS - https://github.com/Worklytics/psoxy-example-aws

   • GCP - https://github.com/Worklytics/psoxy-example-gcp

   You will make changes to the files contained in this repo as appropriate for your use-case. These changes should be committed to a repo that is accessible to other members of your team who may need to support your Psoxy deployment in the future.

3. Pick the location from which you will deploy (provision) the psoxy instance. This location will need the software prereqs defined in the previous section. Some suggestions:

   • your local machine; if you have the prereqs installed and can authenticate it with your host platform (AWS/GCP) as a sufficiently privileged user/role, this is a simple option
4. Follow the 'Setup' steps in the READMEs of those repos, ultimately running terraform apply to deploy your Psoxy instance(s).

5. follow any TODO instructions produced by Terraform, such as:

   • provision API keys / make OAuth grants needed by each Data Connection

   • create the Data Connection from Worklytics to your psoxy instance (Terraform can provide TODO file with detailed steps for each)

6. Various test commands are provided in local files, as the output of the Terraform; you may use these examples to validate the performance of the proxy. Please review the proxy behavior and adapt the rules as needed. Customers needing assistance adapting the proxy behavior for their needs can contact support@worklytics.co

Component Status

Support

As of June 2023, the following resources provisioned by Psoxy modules support use of CMEKs:

• Lambda function environment variables

• SSM Parameters

• Cloud Watch Log Groups

• S3 Buckets

Pre-existing Key

The psoxy-example-aws example provides a project_aws_key_arn variable, that, if provided, will be set as the encryption key for these resources. A few caveats:

• The AWS principal your Terraform is running as must have permissions to encrypt/decrypt with the key (it needs to be able to read/write the lambda env, ssm params, etc)

• The key should be in the same AWS region you're deploying to.

• CloudWatch must be able to use the key, as described in

In example-dev/aws-all/kms-cmek.tf, we provide a bunch of lines that you can uncomment to use encryption on S3 and properly set key policy to support S3/CloudWatch use.

For production use, you should adapt the key policy to your environment and scope as needed to follow your security policies, such as principle of least privilege.

Provisioning a Key

More options

If you need more granular control of CMEK by resource type, review the main.tf and variables exposed by the aws-host module for some options.

Tips and tricks for using AWS as to host the proxy.

Who are you?

If above doesn't happen seem to work as expected, some ideas in the next section may help.

Your AWS Organization uses SSO via Okta or some similar provider

Options:

1. execute terraform via

2. find credentials output by your SSO helper (eg, aws-okta) then fill the AWS CLI env variables yourself:

1. if your SSO helper fills default AWS credentials file but simply doesn't set the env vars, you may be able to export the profile to AWS_PROFILE, eg

References: https://discuss.hashicorp.com/t/using-credential-created-by-aws-sso-for-terraform/23075/7

Your AWS User has MFA

Options:

Logs via Cloud Watch

via Web Console

1. Log into AWS web console

2. navigate to the AWS account that hosts your proxy instance (you may need to assume a role in that account)

3. then the region in that account in which your proxy instance is deployed. (default us-east-1)

4. then search or navigate to the AWS Lambdas feature, and find the specific one you wish to debug

5. find the tabs for Monitoring then within that, Logging, then click "go to Cloud Watch"

via CLI

Unless your AWS CLI is auth'd as a user who can review logs, first auth it for such a role.

You can do this with a new profile, or setting env variables as follows:

Then, you can do a series of commands as follows:

Errors in Terraform apply

error creating Lambda Function URL

Something like the following:

Your Terraform state is inconsistent. Run something like the following, adapted for your connector:

NOTE: you likely need to change outlook-mail if your error is with a different data source. The \ chars are needed to escape the double-quotes/brackets in your bash command.

Permissions Errors

error reading SSM Parameters

Something like the following:

Check:

• the SSM parameter exists in the AWS account

• the SSM parameter can be decrypted by the lambda's execution role (if it's encrypted with a KMS key)

Setting IS_DEVELOPMENT_MODE to "true" in the Lambda's Env Vars via the console can enable some additional logging with detailed SSM error messages that will be helpful; but note that some of these errors will be expected in certain configurations.

Our Terraform examples should provide both of the above for you, but worth double-checking.

If those are present, yet the error persists, it's possible that you have some org-level security constraint/policy preventing SSM parameters from being used / read. For example, you have a "default deny" policy set for SSM GET actions/etc. In such a case, you need to add the execute roles for each lambda as exceptions to such policies (find these under AWS --> IAM --> Roles).

beta - This is now available for customer-use, but may still change in backwards incompatible ways.

Our aws-host module provides a vpc_config variable to specify the VPC configuration for the lambdas that our Terraform modules will create, analogous to the block supported by the AWS lambda terraform resource.

Some caveats:

• API connectors on a VPC must be exposed via rather than

• VPC must be configured such that your lambda has connectivity to AWS services including S3, SSM, and CloudWatch Logs; this is typically done by adding a for each service.

• VPC must allow any API connector to connect to data source APIs via HTTPS (eg 443); usually these APIs are on the public internet, so this means egress to public internet.

• VPC must allow your API gateway to connect to your lambdas.

The requirements above MAY require you to modify your VPC configuration, and/or the security groups to support proxy deployment.

Usage - with vpc.tf

Prequisites:

• the AWS principal (user or role) you're using to run Terraform must have permissions to manage VPCs, subnets, and security groups. The AWS managed policy AmazonVPCFullAccess provides this.

• all pre-requisites for the api-gateways (see )

NOTE: if you provide vpc_config, the value you pass for use_api_gateway_v2 will be ignored; using a VPC requires API Gateway v2.

Add the following to "psoxy" module in your main.tf (or uncomment if already present):

Uncomment the relevant lines in vpc.tf in the same directory, and modify as you wish. This file pulls the default VPC/subnet/security group for your AWS account under terraform.

Alternatively, you modify vpc.tf to use a provision non-default VPC/subnet/security group, and reference those from your main.tf - subject to the caveats above.

See the following terraform resources that you'll likely need:

Troubleshooting

Check your Cloud Watch logs for the lambda. Proxy lambda will time out in INIT phase if SSM Parameter Store or your secret store implementation (AWS Secrets Manager, Vault) is not reachable.

Some potential causes of this:

• DNS failure - it's going to look up the SSM service by domain; if the DNS zone for the SSM endpoint you've provisioned is not published on the VPC, this will fail; similarly, if the endpoint wasn't configured on a subnet - then it won't have an IP to be resolved.

• if the IP is resolved, you should see failure to connect to it in the logs (timeouts); check that your security groups for lambda/subnet/endpoint allow bidirectional traffic necessary for your lambda to retrieve data from SSM via the REST API.

Switching back from using a VPC

Terraform with aws provider doesn't seem to play nice with lambdas/subnets; the subnet can't be destroyed w/o destroying the lambda, but terraform seems unaware of this and will just wait forever.

So:

1. destroy all your lambdas (terraform state list | grep aws_lambda_function; then terraform destroy --target= for each, remember '' as needed)

2. destroy the subnet terraform destroy --target=aws_subnet.main

References

• https://docs.aws.amazon.com/lambda/latest/dg/foundation-networking.html

• https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html

Some organizations require use of API Gateway. This is not the default approach for Psoxy since AWS added support for Lambda Function URLs (March 2022), which are a simpler and more direct way to expose lambdas via HTTPS.

Nonetheless, should you wish to use API Gateway we provide beta support for this. It is needed if you wish to put your Lambda functions on a VPC (See lambdas-on-vpc.md).

In particular:

• IAM policy that allows api gateway methods to be invoked by the proxy caller role is defined once, using wildcards, and exposes GET/HEAD/POST methods for all resources. While methods are further constrained by routes and the proxy rules themselves, this could be another enforcement point at the infrastructure level - at expense of N policies + attachments in your terraform plan instead of 1.

• proxy instances exposed as lambda function urls have 55s timeout, but API Gateway seems to support 30s as max - so this may cause timeouts in certain APIs

Usage

Prerequisites:

• the AWS principal (user or role) to provision API gateways. The AWS managed policy AmazonAPIGatewayAdministrator provides this.

Add the following to your terraform.tfvars file:

use_api_gateway_v2=true

Then terraform apply should create of API gateway-related resources, including policies/etc; and destroy lambda function urls (if you've previously applied with use_api_gateway=false, which is the default).

API Gateway v1 - not supported, but FWIW

If you wish to use API Gateway V1, you will not be able to use the flag above. Instead, you'll have to do something like the following:

locals {
  rest_instances = {
    for id, instance in module.psoxy-aws-google-workspace.instances :
      id => instance if instance.proxy_kind == "rest"
  }
}

resource "aws_apigateway_api" "psoxy-api" {
  name          = "psoxy-api"
  protocol_type = "HTTP"
  description   = "API to expose psoxy instances"
}

resource "aws_cloudwatch_log_group" "gateway-log" {
  name              = aws_apigateway_api.psoxy-api.name
  retention_in_days = 7
}

resource "aws_apigatewayv2_stage" "live" {
  api_id      = aws_apigateway_api.psoxy-api.id
  name        = "live" # q: what name??
  auto_deploy = true
  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.gateway-log.arn
    format          = "$context.identity.sourceIp $context.identity.caller $context.identity.user [$context.requestTime] \"$context.httpMethod $context.path $context.protocol\" $context.status $context.responseLength $context.requestId $context.extendedRequestId $context.error.messageString $context.integrationErrorMessage"
  }
}


resource "aws_apigateway_integration" "map" {
  for_each = local.rest_instances

  api_id                    = aws_apigateway_api.psoxy-api.id
  integration_type          = "AWS_PROXY"
  connection_type           = "INTERNET"

  integration_method        = "POST"
  integration_uri           = each.value.function_arn
  request_parameters        = {}
  request_templates         = {}
}


resource "aws_apigateway_route" "get_route" {
  for_each = local.rest_instances

  api_id             = aws_apigateway_api.psoxy-api.id
  route_key          = "GET /${each.key}/{proxy+}"
  authorization_type = "AWS_IAM"
  target             = "integrations/${aws_apigateway_integration.map[each.key].id}"
}

resource "aws_apigateway_route" "head_route" {
  for_each = local.rest_instances

  api_id             = aws_apigatewayv2_api.psoxy-api.id
  route_key          = "HEAD /${each.key}/{proxy+}"
  authorization_type = "AWS_IAM"
  target             = "integrations/${aws_apigateway_integration.map[each.key].id}"
}

# allow API gateway to invoke the lambda function
resource "aws_lambda_permission" "lambda_permission" {
  for_each = local.rest_instances

  statement_id  = "Allow${each.key}Invoke"
  action        = "lambda:InvokeFunction"
  function_name = each.value.function_name
  principal     = "apigateway.amazonaws.com"


  # The /*/*/ part allows invocation from any stage, method and resource path
  # within API Gateway REST API.
  source_arn = "${aws_apigateway_api.psoxy-api.execution_arn}/*/*/${each.value.function_name}/{proxy+}"
}

resource "aws_iam_policy" "invoke_api" {
  name_prefix = "PsoxyInvokeAPI"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : "execute-api:Invoke",
        "Resource" : "arn:aws:execute-api:*:${var.aws_account_id}:${aws_apigateway_api.psoxy-api.id}/*/GET/*",
      },
      {
        "Effect" : "Allow",
        "Action" : "execute-api:Invoke",
        "Resource" : "arn:aws:execute-api:*:${var.aws_account_id}:${aws_apigateway_api.psoxy-api.id}/*/HEAD/*",
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "invoke_api_policy_to_role" {
  role       = "PsoxyCaller" # Name (not ARN) of the API caller role
  policy_arn = aws_iam_policy.invoke_api.arn
}

Additionally, you'll need to set a different handler class to be invoked instead of the default (co.workltyics.psoxy.Handler, should be co.worklytics.psoxy.APIGatewayV1Handler). This can be done in Terraform or by modifying configuration via AWS Console.

Some ideas on how to support scenarios and configuration requirements beyond what our default examples show:

Encryption Keys

see encryption-keys.md

Tagging ALL infra created by your Terraform Configuration

If you're using our AWS example, it should support a default_tags variable.

You can add the following in your terrform.tfvars file to set tags on all resources created by the example configuration:

default_tags = {
  Vendor = "Worklytics"
}

If you're not using our AWS example, you can add the following to your configuration, then you will need to modify the aws provider block in your configuration to add a default_tags. Example shown below:

See: [https://registry.terraform.io/providers/hashicorp/aws/latest/docs#default_tags]

provider "aws" {
  region = var.aws_region

  assume_role {
    role_arn = var.aws_assume_role_arn
  }

  default_tags {
    Vendor  = "Worklytics"
  }

  allowed_account_ids = [
    var.aws_account_id
  ]
}

Extensibility

To support extensibility, our Terraform examples/modules output the IDs/names of the major resources they create, so that you can compose them with other Terraform resources.

Buckets

The aws-host module outputs bulk_connector_instances; a map of id => instance for each bulk connector. Each of these has two attributes that correspond to the names of its related buckets:

• sanitized_bucket_name

• input_bucket_name

So in our AWS example, you can use these to enable logging, for example, you could do something like this: (YMMV, syntax etc should be tested)

local {
  id_of_bucket_to_store_logs = "{YOUR_BUCKET_ID_HERE}"
}

resource "aws_s3_bucket_logging" "logging" {
  for_each = module.psoxy.bulk_connector_instances

  bucket = each.value.sanitized_bucket_name

  target_bucket = local.id_of_bucket_to_store_logs
  target_prefix = "psoxy/${each.key}/"
}

resource "aws_s3_bucket_logging" "logging" {
  for_each = module.psoxy.bulk_connector_instances

  bucket = each.value.input_bucket_name

  target_bucket = local.id_of_bucket_to_store_logs
  target_prefix = "psoxy/${each.key}/"
}

Analogous approaches can be used to configure versioning, replication, etc;

• aws_s3_bucket_versioning

• aws_s3_bucket_replication

Note that encryption, lifecycle, public_access_block are set by the Workltyics-provided modules, so you may have conflicts issues if you also try to set those outside.

Lambda Execution Role

beta - released from v0.4.50; YMMV, and may be subject to change.

The terraform modules we provide provision execution roles for each lambda function, and attach by default attach the appropriate AWS Managed Policy to each.

Specifically, this is AWSLambdaBasicExecutionRole, unless you're using a VPC - in which case it is AWSLambdaVPCAccessExecutionRole(https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaVPCAccessExecutionRole.html).

For organizations that don't allow use of AWS Managed Policies, you can use the aws_lambda_execution_role_policy_arn variable to pass in an alternative which will be used INSTEAD of the AWS Managed Policy.

This page provides an overview of how psoxy authenticates and confirms authorization of clients (Worklytics tenants) to access data for GCP-hosted deployments.

Authentication

As Worklytics tenants run inside GCP, they are implicitly authenticated by GCP. No secrets or keys need be exchanged between your Worklytics tenant and your Psoxy instance. GCP can verify the identity of requests from Worklytics to your instance, just as it does between any process and resource within GCP.

Authorization

Invocations of your proxy instances are authorized by the IAM policies you define in GCP. For API connectors, you grant the Cloud Function Invoker role to your Worklytics tenant's GCP service account on the Cloud Function for your instance.

For the bulk data case, you grant the Storage Object Viewer role to your Worklytics tenant's GCP service account on the sanitized output bucket for your connector.

You can obtain the identity of your Worklytics tenant's GCP service account from the Worklytics portal.

getting-started.md

Tips and tricks for using GCP as to host the proxy.

GCloud CLI client blocked by Organization policy

Some orgs have policies that block authentication of the GCloud CLI client, requiring you to contact your IT team and have it added to an approved list. Apart from that, there are several possibilities:

1. use the GCP Cloud Shell (via GCP web console). gcloud is pre-installed and pre-authorized as your Google user in the Cloud Shell.

2. use a VM in GCP Compute Engine, with the VM running as a sufficiently privileged service account. In such a scenario, gcloud will be pre-authenticated by GCP on the VM as that service account.

3. create credentials within the project itself:

   • enable IAM API and Cloud Resource Manager API within the project

   • create OAuth credentials for a 'desktop application' within the target GCP project

   • download the client-secrets.json file to your environment

   • run gcloud auth application-default login --client-id-file=/path/to/client-secrets.json

GCP rejects calls because APIs disabled on target project

Terraform relies on GCP's REST APIs for its operations. If these APIs are disabled either the target project OR the project in which the identity (service account, OAuth client) under which you're running terraform resides, you may get an error.

The solution is to enable APIs via the Cloud Console, specifically:

• IAM API

• Cloud Resource Manager API

GCP Terraform State Inconsistencies

If some resources seem to not be properly provisioned, try terraform taint or terraform state rm, to force re-creation. Use terrafrom state list | grep to search for specific resource ids.

Error 400 : One or more users named in policy do not belong to a permitted Customer

If you receive an error such as:

Error: Error applying IAM policy for cloudfunctions cloudfunction googleapi: Error 400: One or more users named in the policy do not belong to a permitted customer.

This may be due to an Organization Policy that restricts the domains that can be used in IAM policies. See https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains

You may need define an exception for the GCP project in which you're deploying the proxy, or add the domain of your Worklytics Tenant SA to the list of allowed domains.

Done with your Psoxy deployment?

Terraform makes it easy to clean up when you're through with Psoxy, of you wish to rebuild everything from scratch.

First, a few caveats:

• this will NOT undo any changes outside of Terraform, even those we instructed you to perform via TODO - files that Terraform may have generated.

• be careful with anything you created outside of Terraform and later imported into Terraform, such as GCP project / AWS account themselves. If you DON'T want to destroy these, do terraform state rm <resource> (analogue of the import) for each.

Do the following to destroy your Psoxy infra:

1. open you main.tf of your terraform confriguation; remove ALL blocks that aren't terraform, or provider. You'll be left with ~30 lines that looks like the following.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.12"
    }

    azuread = {
      version = "~> 2.0"
    }
  }

  backend "local" {
  }
}

provider "aws" {
  region = var.aws_region

  assume_role {
    role_arn = var.aws_assume_role_arn
  }
  allowed_account_ids = [
    var.aws_account_id
  ]
}

provider "azuread" {
  tenant_id = var.msft_tenant_id
}

NOTE: do not edit your terraform.tfvars file or remove any references to your AWS / Azure / GCP accounts; Terraform needs be authenticated and know where to delete stuff from!

1. run terraform apply. It'll prompt you with a plan that says "0 to create, 0 to modify" and then some huge number of things to destroy. Type 'yes' to apply it.

That's it. It should remove all the Terraform infra you created.

1. if you want to rebuild from scratch, revert your changes to main.tf (git checkout main.tf) and then terraform apply again.

Guides

• Deploying via Terraform Cloud

• Migrating Deployment to Another Host/Account

• Cleaning Up

• Testing

1. clone the repo (or a private-fork of it)

git clone https://github.com/Worklytics/psoxy.git

1. if using Microsoft 365 sources, install and authenticate Azure CLI

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

1. if deploying AWS infra, install and authenticate AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

You should now be ready for the general instructions in the README.md.

Overview

This document describes how to migrate your deployment from one cloud provider to another, or one project/account to another. It does not cover migrating between proxy versions.

Use cases:

• move from a dev account to a prod account (Account / Project Migration)

• move from a "shared" account to a "dedicated" account (Account / Project Migration)

• move from AWS --> GCP, and vice versa (Provider Migration)

Preparation

Preserving Existing Infrastructure

Some data/infrastructure MUST, or at least SHOULD be preserved during your migration. Below is an enumeration of both cases.

Data, such as configuration values, can generally be copied; you just need to make a new copy of it in the new environment managed by the new Terraform configuration.

Some infrastructure, such as API Clients, will be moved; eg, the same underlying resource will continue to exist, it will just be managed by the new Terraform configuration instead of the old one. This is the more tedious case, as you must both import this infrastructure to your new configuration and then rm (remove) it from your old configuration, rather than having it be destroyed when you teardown the old configuration. You should carefully review every terraform apply, including terraform destroy commands, to ensure that infrastructure you intend to move is not destroyed, or replaced (eg, terraform sees it as tainted, and does a destroy + create within a single apply operation).

What you MUST copy:

• SALT value. This is a secret used to generate the pseudonyms. If this is lost/destroyed, you will be unable to link any data pseudonymized with the original salt to data you process in the future.

  • NOTE: the underlying resource to preserve is actually a random_password resource, not an SSM parameter / GCP Secret - because those simply are being filled from the terraform random_password resource; if you import parameter/secret, but not the random_password, Terraform will generate a new value and overwrite the parameter/secret.

  • as of v0.4.35 examples, the terraform resource ID for this value is expected to be module.psoxy.module.psoxy.random_password.pseudonym_salt; if not, you can search for it with terraform state list | grep random_password

• value for PSEUDONYMIZE_APP_IDS. This value, if set to true will have the proxy use a rule set that pseudonymizes identifiers issued by source applications themselves in some cases where these identifiers aren't inherently PII - but the association could be considered discoverable.

• value for EMAIL_CANONICALIZATION. prior to v0.4.52, this default was in effect STRICT; so if your original deployment was built on a version prior to this, you should explicitly set this value to STRICT in your new configuration (likely email_canonicalization variable in terraform modules)

• any custom sanitization rules that you've set, either in your Terraform configuration or directly as the value of a RULES environment variable, SSM Parameter, or GCP Secret.

• historical sanitized files for any bulk connectors, if you wish to continue to have this data analyzed by Worklytics. (eg, everything from all your -sanitized buckets)

NOTE: you do NOT need to copy the ENCRYPTION_KEY value; rotation of this value should be expected by clients.

What you SHOULD move:

• API Clients. Whether generated by Terraform or not, the "API Client" for a data source must typically be authorized by a data source administrator to grant it access to the data source. As such, if you destroy the client, or lose its id, you'll need to coordinate with the administrator again to recreate it / obtain the configuration information.

  • as of v0.4.35, Google Workspace and Microsoft 365 API clients are managed directly by Terraform, so these are important to preserve.

What you SHOULD copy:

• API Client Secrets, if generated outside of Terraform. If you destroy/lose these values, you'll need to contact the data source administrator to obtain new versions.

Prior to beginning your migration, you should make a list of what existing infrastructure and/or configuration values you intend to move/copy.

Migration Plan

The following is a rough guide on the steps you need to take to migrate your deployment.

Phase 1 : Gather information from Existing Environment

1. Salt value. If using an example forked from our template repos at v0.4.35 or later, you can find the output block in your main.tf for pseudonym_salt, uncomment it, run terraform apply. You'll then be able to obtain the value with: terraform output --raw pseudonym_salt On macOS, you can copy the value to your clipboard with: terraform output --raw pseudonym_salt | pbcopy

2. Microsoft 365 API client, if any:

   • Find the resource ids: terraform state list | grep "\.azuread_application\."

   • For each, obtain it's objectId: terraform state show 'module.psoxy.module.msft-connection["azure-ad"].azuread_application.connector'

   • Prepare import command for each client for your new configuration, eg: terraform import 'module.psoxy.module.msft-connection["azure-ad"].azuread_application.connector' '<objectId>'

3. Google Workspace API clients, if any:

   • Find the resource ids: tf state list | grep 'google_service_account\.connector-sa'

   • For each, obtain its unique_id: terraform state show 'module.worklytics_connectors_google_workspace.module.google_workspace_connection["gdirectory"].google_service_account.connector-sa'

   • Prepare import command for each client for your new configuration, eg: terraform import 'module.worklytics_connectors_google_workspace.module.google_workspace_connection["gdirectory"].google_service_account.connector-sa' '<unique_id>'

Phase 2 : Create New Environment

1. Create a new Terraform configuration from scratch; run terraform init there (if you begin with one of our examples, our init script does this). Use the terraform.tfvars of your existing configuration as a guide for what variables to set, copying over any needed values.

2. Run a provisional terraform plan and review.

3. Run the imports you prepared in Phase 1, if all appear OK, run another terraform plan and review (comparing to the old one).

4. Optionally, run terraform plan -out=plan.out to create a plan file; if you send this, along with all the *.tf/*.tfvars files to Worklytics, we can review it and confirm that it is correct.

5. Run terraform apply to create the new infrastructure; re-confirm that the plan is not re-creating any API clients/etc that you intended to preserve

6. Via AWS / GCP console, or CLIs, move the values of any secrets/parameters that you intend to by directly reading the values from your old account/project, and copying them into the new account/project

Phase 3: Migrate

1. Look at the TODO 3 files/output variables for all your connectors. Make a mapping between the old values and the new values. Send this to Worklytics. It should include for each the proxy URLs, AWS Role to use, and any other values that are changing.

2. Wait for confirmation that Worklytics has migrated all your connections to the new values. This may take 1-2 days.

Phase 4: Destroy Old Environment

1. Remove references to any API Clients you migrated in Phase 1:

   • eg, terraform state rm 'module.psoxy.module.msft-connection["azure-ad"].azuread_application.connector'

2. run terraform destroy in the old configuration. Carefully review the plan before confirming.

   • if you're using Google Workspace sources, you may see destruction of google_project_service resources; if you allow these to be destroyed, these APIS will be disabled; if you are using the same GCP project in your other configuration, you should run terraform apply there again to re-enable them.

3. You may also destroy any API clients/etc that are managed outside of Terraform and which you did not migrate to the new environment.

4. You may clean up any configuration values, such as SSM Parameters / GCP Secrets to customize the proxy rules sets, that you may have created in your old host environment.

Overview

You'll provision infrastructure that ultimately looks as follows:

This includes:

• Cloud Functions

• Service Accounts

• Secret Manager Secrets, to hold pseudonymization salt, encryption keys, and data source API keys

• Cloud Storage Buckets (GCS), if using psoxy to sanitize bulk file data, such as CSVs

NOTE: if you're connecting to Google Workspace as a data source, you'll also need to provision Service Account Keys and activate Google Workspace APIs.

Prerequisites

• a Google Project

  • we recommend a dedicated GCP project for your deployment, to provide an implicit security boundary around your infrastructure as well as simplify monitoring/cleanup

• a GCP (Google) user or Service Account with permissions to provision Service Accounts, Secrets, Storage Buckets, Cloud Functions, and enable APIs within that project. eg:

  • Cloud Functions Admin - proxy instances are deployed as GCP cloud functions

  • Cloud Storage Admin - processing of bulk data (such as HRIS exports) uses GCS buckets

  • IAM Role Admin - create custom roles for the proxy, to follow principle of least privilege

  • Secret Manager Admin - your API keys and pseudonymization salt is stored in Secret Manager

  • Service Account Admin - admin Service Accounts that personify Cloud Functions or are used as Google Workspace API connections

  • Service Usage Admin - you will need to enable various GCP APIs

• the following APIs enabled in the project: (via GCP Console)

  • IAM Service Account Credentials API (iamcredentials.googleapis.com) - generally needed to support authenticating Terraform. May not be needed if you're running terraform within a GCP environment.

  • Service Usage API (serviceusage.googleapis.com)

• additional APIs enabled in the project: (using the Service Usage API above, our Terraform will attempt to enable these, but as there is sometimes a few minutes delay in activation and in some cases they are required to read your existing infra prior to apply, you may experience errors. To pre-empt those, we suggest ensuring the following are enabled:

  • Compute Engine API (compute.googleapis.com)

  • Cloud Build API (cloudbuild.googleapis.com)

  • Cloud Functions API (cloudfunctions.googleapis.com)

  • Cloud Resource Manager API (cloudresourcemanager.googleapis.com)

  • IAM API (iam.googleapis.com)

  • Secret Manager API (secretmanager.googleapis.com)

  • Storage API (storage-api.googleapis.com)

Terraform State Backend

You'll also need a secure backend location for your Terraform state (such as a GCS or S3 bucket). It need not be in the same host platform/project/account to which you are deploying the proxy, as long as the Google/AWS user you are authenticated as when running Terraform has permissions to access it.

Some options:

• GCS : https://developer.hashicorp.com/terraform/language/settings/backends/gcs

• S3 : https://developer.hashicorp.com/terraform/language/settings/backends/s3

Alternatively, you may use a local file system, but this is not recommended for production use - as your Terraform state may contain secrets such as API keys, depending on the sources you connect.

See: https://developer.hashicorp.com/terraform/language/settings/backends/local

Bootstrap

For some help in bootstraping a GCP environment, see also: infra/modules/gcp-bootstrap/README.md

The module psoxy-constants is a dependency-free module that provides lists of GCP roles, etc needed for bootstraping a GCP project in which your proxy instances will reside.

Example

The https://github.com/Worklytics/psoxy-example-gcp repo provides an example configuration for hosting proxy instances in GCP. You use that template, following it's Usage docs to get started.

Security Considerations

• the 'Service Account' approach described in the prerequisites is preferable to giving a Google user account IAM roles to administer your infrastructure directly. You can pass this Service Account's email address to Terraform by setting the gcp_terraform_sa_account_email. Your machine/environments CLI must be authenticated as GCP entity which can impersonate this Service Account, and likely create tokens as it (Service Account Token Creator role).

• using a dedicated GCP project is superior to using a shared project, as it provides an implicit security boundary around your infrastructure as well as simplifying monitoring/cleanup. The IAM roles specified in the prerequisites must be granted at the project level, so any non-Proxy infrastructure within the GCP project that hosts your proxy instances will be accessible to the user / service account who's managing the proxy infrastructure.

If you're using Terraform Cloud or Enterprise, here are a few things to keep in mind.

NOTE: this is tested only for gcp; for aws YMMV, and in particular we expect Microsoft 365 sources will not work properly, given how those are authenticated.

Getting Started

Prereqs:

• git/java/maven, as described here https://github.com/Worklytics/psoxy#required-software-and-permissions

• for testing, you'll need the CLI of your host environment (eg, AWS CLI, GCloud CLI, Azure CLI) as well as npm/NodeJS installed on your local machine

After authenticating your terraform CLI to Terraform Cloud/enterprise, you'll need to:

1. Create a Project in Terraform Cloud; and a workspace within the project.

2. Clone one of our example repos and run the ./init script to initialize your terraform.tfvars for Terraform Cloud. This will also put a bunch of useful tooling on your machine.

3. Commit the bundle that was output by the ./init script to your repo:

1. Change the terraform backend main.tf to point to your Terraform Cloud rather than be local

   • remove backend block from main.tf

   • add a cloud block within the terraform block in main.tf (obtain content from your Terraform Cloud)

2. run terraform init to migrate the initial "local" state to the remote state in Terraform Cloud

3. You'll have to authenticate your Terraform Cloud with Google / AWS / Azure, depending on the cloud you're deploying to / data sources you're using.

TODOs as Outputs

If you're using Terraform Cloud or Enterprise, our convention of writing "TODOs" to the local file system might not work for you.

To address this, we've updated most of our examples to also output todo values as Terraform outputs, todos_1, todos_2, etc.

To get them nicely on your local machine, something like the following:

Terraform API

1. get an API token from your Terraform Cloud or Enterprise instance (eg, https://developer.hashicorp.com/terraform/cloud-docs/users-teams-organizations/api-tokens).

2. set it as an env variable, as well as the host:

1. run a curl command using those values to get each todos:

Terraform CLI

If you have terraform CLI auth'd against your Terraform Cloud or Enterprise instance, then you might be able to avoid the curl-hackery above, and instead use the following:

(This approach should also work with Terraform CLI running with backend, rather than cloud)

Testing Locally

If you have run our init script locally (as suggested in 'Getting Started') then the test tool should have been installed (likely at .terraform/modules/psoxy/tools/). You will need to update everything in todos_2.md to point to this path for those test commands to work.

If you need to directly install/re-install it, something like the following should work:

Specific Data Sources

Error: Attempted to load application default credentials (Google provider authentication failure)

This is related to gcloud not being authenticated (or installed?) in the environment where you're running terraform, which the google terraform provider requires.

If you DO NOT intend to use Google Workspace as a data source, you should do the following:

• remove the google-*.tf files from your terraform configuration

• remove module/local references from your main.tf file that referred to those files; as of v0.4.53, there are 3 such references you must remove; you will get errors in terraform commands until you remove all of them. The error messages should reference the impacted line numbers.

If you DO intend to use Google Workspace as a data source, you must install and authenticate the gcloud CLI and/or modify the google provider block in google-workspace.tf with your desired authentication details. See:

Specific Host platforms:

General Tips

Verify Pre-Requisites

Our example templates include a script to check for the prerequisites for running the psoxy. You can run this prior to ./init to get feedback/suggestions on what prerequisites you may be missing and how to install them.

General Build / Packaging Failures

Our example Terraform configurations should compile and package the Java code into a JAR file, which is then deployed by Terraform to your host environment.

If, on your first terraform plan/terraform apply, you see the line such as

module.psoxy-aws-msft-365.module.psoxy-aws.module.psoxy-package.data.external.deployment_package: Reading...

And that returns really quickly, something may have gone wrong with the build. You can trigger the build directly by running:

That may give you some clues as to what went wrong.

You can also look for a file called last-build.log in the directory where your Terraform configuration resides.

If you want to go step-by-step, you can run the following commands:

Some problems we've seen:

• Maven repository access - the build process must get various dependencies from a remote Maven respository; if your laptop cannot reach Maven Central, is configured to get dependencies from some other Maven repository, etc - you might need to fix this issue. You can check your ~/.m2/settings.xml file, which might give you some insight into what Maven repository you're using. It's also where you'd configure credentials for a private Maven repository, such as Artifactory/etc - so make sure those are correct.

Upgrading Psoxy Code

If you upgrade your psoxy code, it may be worth trying terraform init --upgrade to make sure you have the latest versions of all Terraform providers on which our configuration depends.

By default, terraform locks providers to the version that was the latest when you first ran terraform init. It does not upgrade them unless you explicitly instruct it to. It will not prompt you to upgrade them unless we update the version constraints in our modules.

While we strive to ensure accurate version constraints, and use provider features consistent with these constraints, our automated tests will run with the latest version of each provider. Regretably, we don't currently have a way to test with ALL versions of each provider that satisfy the constraints, or all possible combinations of provider versions.

State Inconsistencies

Often, in response to errors, a second run of terraform apply will work.

If something was actually created in the cloud provider, but Terraform state doesn't reflect it, then try terraform import [resource] [provider-resource-id]. [resource] should be replaced with whatever the path to it is in your terraform configuration, which you can get from the terraform plan output. provider-resource-id is a little trickier, and you might need to find the format required by finding the Terraform docs for the resource type on the web.

NOTE: resources in plan with brackets/quotes will need these escaped with a backslash for use in bash commands.

eg

Unsupported Terraform versions

Errors such as the following on terraform plan?

The solution is to downgrade your Terraform version to one that's supported by our modules (>= 1.3.x, <= 1.7.x as of March 2024).

If you're running Terraform in cloud/CI environment, including Terraform Cloud, GitHub Actions, etc, you can likely explicitly set the desired Terraform version in your workspace settings / terraform setup action.

By default, the Terraform examples provided by Worklytics install a NodeJS-based tool for testing your proxy deployments.

Full documentation of the test tool is available . And the code is located in the tools directory of the .

Testing Pre-requisites

Wherever you run this test tool from, your AWS or GCloud CLI must be authenticated as an entity with permissions to invoke the Lambda functions / Cloud functions that you deployed for Psoxy.

If you're testing the bulk cases, the entity must be able to read/write to the cloud storage buckets created for each of those bulk examples.

Testing Locally when Terraform ran remotely (eg, Terraform Cloud, GitHub Actions, etc)

If you're running the Terraform examples in a different location from where you wish to run tests, then you can install the tool alone:

1. Clone the Psoxy repo to your local machine:

1. From within that clone, install the test tool:

1. Get specific test commands for your deployment

   • If you set the todos_as_outputs variable to true, your Terraform apply run should contain todo2 output variable with testing instructions.

   • If you set todos_as_local_files variable to true, your Terraform apply run should contain local files named TODO 2 ... with testing instructions.

   In both cases, you will need to replace the test tool path included there with the path to your installation.

2. Example commands of the primary testing tool: "Psoxy Test Calls"

Testing Deployments made without Terraform

If you used and approach other than Terraform, or did not directly use our Terraform examples, you may not have the testing examples or the test tool installed on your machine.

In such a case, you can install the test tool manually by following steps 1+2 above, and then can review the documentation on how to use it from your machine.

Psoxy supports specifying sanitization rule sets to use to sanitize data from an API. These can be configured by encoding a rule set in YAML and setting a parameter in your instance's configuration. See an example of rules for Zoom: .

If such a parameter is not set, a proxy instances selects default rules based on source kind, from the corresponding .

You can configure custom rule sets for a given instance via Terraform, by adding an entry to the custom_api_connector_rules map in your terraform.tfvars file.

eg,

API Connector Rules Syntax

<ruleset> ::= "endpoints:" <endpoint-list> <endpoint-list> ::= <endpoint> | <endpoint> <endpoint-list>

A ruleset is a list of API endpoints that are permitted to be invoked through the proxy. Requests which do not match a endpoint in this list will be rejected with a 403 response.

Endpoint Specification

<endpoint> ::= <path-template> <allowed-methods> <path-parameter-schemas> <query-parameter-schemas> <response-schema> <transforms>

<path-template> ::= "- pathTemplate: " <string> Each endpoint is specified by a path template, based on OpenAPI Spec v3.0.0 Path Template syntax. Variable path segments are enclosed in curly braces ({}) and are matched by any value that does not contain an / character.

See: https://swagger.io/docs/specification/paths-and-operations/

Allowed Methods

<allowed-methods> ::= "- allowedMethods: " <method-list> <method-list> ::= <method> | <method> <method-list>

<method> ::= "GET" | "POST" | "PUT" | "PATCH" | "DELETE" | "HEAD"

If provided, only HTTP methods included in this list will be permitted for the endpoint. Given semantics of RESTful APIs, this allows an additional point to enforce "read-only" data access, in addition to OAuth scopes/etc.

Path / Query Parameter Schemas

<path-parameter-schemas> ::= "- pathParameterSchemas: " <parameter-schema> <query-parameter-schemas> ::= "- queryParameterSchemas: " <parameter-schema>

alpha - a parameter schema to use to validate path/query parameter values; if validation fails, proxy will return 403 forbidden response. Given the use-case of validating URL / query parameters, only a small subset of JSON Schema is supported.

As of 0.4.38, this is considered an alpha feature which may change in backwards-incompatible ways.

Currently, the supports JSON Schema features are:

• type :

  • string value must be a JSON string

  • integer value must be a JSON integer

  • number value must be a JSON number

• format :

  • reversible-pseudonym : value MUST be a reversible pseudonym generated by the proxy

• pattern : a regex pattern to match against the value

• enum : a list of values to match against the value

null/empty is valid for all types; you can use a pattern to restrict this further.

Response Schema

<response-schema> ::= "responseSchema: " <json-schema-filter>

<transforms> ::= "transforms:" <transform-list> <transform-list> ::= <transform> | <transform> <transform-list>

For each Endpoint, rules specify a list of transforms to apply to the response content.

Transform Specification

<transform> ::= "- " <transform-type> <json-paths> [<encoding>]

Supported Transform Types:

<transform-type> ::= "!<pseudonymizeEmailHeader>" | "!<pseudonymize>" | "!<redact>" | "!<redactRegexMatches>" | "!<tokenize>" | "<!filterTokenByRegex>" | "!<redactExceptSubstringsMatchingRegexes"

NOTE: these are implementations of com.avaulta.gateway.rules.transforms.Transform class in the psoxy codebase.

Pseudonymize

!<pseudonymize> - transforms matching values by normalizing them (triming whitespace; if appear to be emails, treating them as case-insensitive, etc) and computing a SHA-256 hash of the normalized value. Relies on SALT value configured in your proxy environment to ensure the SHA-256 is deterministic across time and between sources. In the case of emails, the domain portion is preserved, although the hash is still based on the entire normalized value (avoids hash of alice@acme.com matching hash of alice@beta.com).

Options:

• includeReversible (default: false): If true, an encrypted form of the original value will be included in the result. This value, if passed back to the proxy in a URL, will be decrypted back to the original value before the request is forward to the data source. This is useful for identifying values that are needed as parameters for subsequent API requests. This relies on symmetric encryption using the ENCRYPTION_KEY secret stored in the proxy; if ENCRYPTION_KEY is rotated, any 'reversible' value previously generated will no longer be able to be decrypted by the proxy.

• encoding (default: JSON): The encoding to use when serializing the pseudonym to a string.

  • JSON - a JSON object structure, with explicit fields

  • URL_SAFE_TOKEN - a string format that aims to be concise, URL-safe, and format-preserving for email case.

Pseudonymize Email Header

!<pseudonymizeEmailHeader> - transforms matching values by parsing the value as an email header, in accordance with RFC 2822 and some typical conventions, and generating a pseudonym based only on the normalized email address itself (ignoring name, etc that may appear) . In particular:

• deals with CSV lists (multiple emails in a single header)

• handles the name <email> format, in effect redacting the name and replacing with a pseudonym based only on normalized email

Redact

!<redact> - removes the matching values from the response.

Some extensions of redaction are also supported:

• !<redactExceptSubstringsMatchingRegexes> - removes the matching values from the response except value matches one of the specified regex options. (Use case: preserving portions of event titles if match variants of 'Focus Time', 'No Meetings', etc)

• !<redactRegexMatches> - redact content IF it matches one of the regexs included as an option.

By using a negation in the JSON Path for the transformation, !<redact> can be used to implement default-deny style rules, where all fields are redacted except those explicitly listed in the JSON Path expression. This can also redact object-valued fields, conditionally based on object properties as shown below.

Eg, the following redacts all headers that have a name value other than those explicitly listed below:

Tokenize

!<tokenize> - replaces matching values it with a reversible token, which proxy can reverse to the original value using ENCRYPTION_KEY secret stored in the proxy in subsequent requests.

Use case are values that may be sensitive, but are opaque. For example, page tokens in Microsoft Graph API do not have a defined structure, but in practice contain PII.

Options:

• regex a capturing regex to use to extract portion of value that needs to be tokenized.

Filter Tokens by Regex

!<filterTokenByRegex> - tokenizes matching string values by a delimiter, if provided; and matches result against a list of filters, removing any content that doesn't match at least one of the filters. (Use case: preserving Zoom URLs in meeting descriptions, while removing the rest of the description)

Options:

• delimiter - used to split the value into tokens; if not provided, the entire value is treated as a single token.

• filters - in effect, combined via OR; tokens matching ANY of the filters is preserved in the value.

Response Schema Specification

A "response schema" is a "JSON Schema Filter" structure, specifying how response (which must be JSON) should be filtered. Using this, you can implement a "default deny" approach to sanitizing API fields in a manner that may be more convenient than using JSON paths with conditional negations (a redact transform with a JSON path that matches all but an explicit list of named fields is the other approach to implementing 'default deny' style rules).

See: https://json-schema.org/implementations.html#schema-generators

If a responseSchema attribute is specified for an endpoint, the response content will be filtered (rather than validated) against that schema. Eg, fields NOT specified in the schema, or not of expected type, will be removed from the response.

• type - one of :

  • object a JSON object

  • array a JSON array

  • string a JSON string

  • number a JSON number, either integer or decimal.

  • integer a JSON integer (not a decimal)

  • boolean a JSON boolean

• properties - for type == object, a map of field names to schema to filter field's value against (eg, another JsonSchemaFilter for the field itself)

• items - for type == array, a schema to filter each item in the array against (again, a JsonSchemaFilter)

• format - for type == string, a format to expect for the string value. As of v0.4.38, this is not enforced by the proxy.

• $ref - a reference to a schema specified in the definitions property of the root schema.

• definitions - a map of schema names to schemas of type JsonSchemaFilter; only supported at root schema of endpoint.

Example:

The following is for a User from the GitHub API, via graphql. See: https://docs.github.com/en/graphql/guides/forming-calls-with-graphql#the-graphql-endpoint

The apply (java, maven, etc).

With those, you can can run locally via IntelliJ, using run configs (located in .idea/runConfigurations):

• package install core builds the core JAR, on which implementations depend

• gcp - run gmail builds and runs a local instance for GMail

Or from command line:

By default, that serves the function from http://localhost:8080.

GMail Example

1.) run terraform init and terraform apply from infra/dev-personal to provision environment

Local

2.) run locally via IntelliJ run config

3.) execute the following to verify your proxy is working OK

Health check (verifies that your client can reach and invoke the proxy at all; and that is has sensible config)

Using a message id you grab from that:

Cloud

1.) deploy to GCP using Terraform (see infra/). Follow steps in any TODO files it generates.

2.) Set your env vars: (these should be in a TODO file generated by terraform in prev step

3.) grant yourself access (probably not needed if you have primitive role in project, like Owner or Editor)

alternatively, you can add Terraform resource for this to your Terraform config, and apply it again:

Either way, if this function is for prod use, please remove these grants after you're finished testing.

4.) invocation examples

Overview

Psoxy can be used to sanitize bulk files (eg, CSV, NDSJON, etc), writing the result to another bucket.

You can automate a data pipeline to push files to an -input bucket, which will trigger a Psoxy instance (GCP Cloud Function or AWS Lambda), which will read the file, sanitize it, and write the result to a corresponding -sanitized bucket.

You should limit the size of files processed by proxy to 200k rows or less, to ensure processing of any single file finishes within the run time limitations of the host platform (AWS, GCP). There is some flexibility here based on the complexity of your rules and file schema, but we've found 200k to be a conservative target.

Compression

To improve performance and reduce storage costs, you should compress (gzip) the files you write to the -input bucket. Psoxy will decompress gzip files before processing and then compress the result before writing to the -sanitized bucket. Ensure that you set Content-Encoding: gzip on all files in your -input bucket to enable this behavior. Note that if you are uploading files via the web UI in GCP/AWS, it is not possible to set this metadata in the initial upload - so you cannot use compression in such a scenario.

Sanitization Rules

The 'bulk' mode of Psoxy supports either column-oriented or record-oriented file formats.

Column-Oriented Formats (ColumnarRules)

To cater to column-oriented file formats (Eg .csv, .tsv), Psoxy supports a ColumnarRules format for encoding your sanitization rules. This rules format provides simple/concise configuration for these cases, where more complex processing of repeated values / complex field types is required.

If your use-case is record oriented (eg, NDJSON, etc), with nested or repeated fields, then you will likely need RecordRules as an alternative.

Pseudonymization

The core function of the Proxy is to pseudonymize PII in your data. To pseudonymize a column, add it to columnsToPseudonymize.

To avoid inadvertent data leakage, if a column specified to be pseudonymized is not present in the input data, the Proxy will fail with an error. This is to avoid simple column name typos resulting in data leakage.

Additional Transformations

To ease integration, the 'bulk' mode also supports a few additional common transformations that may be useful. These provide an alternative to using a separate ETL tool to transform your data, or modifying your existing data export pipelines.

Redaction

To redact a column, add it to columnsToRedact. By default, all columns present in the input data will be included in the output data, unless explicitly redacted.

Inclusion

Alternatively to redacting columns, you can specify columnsToInclude. If specified, only columns explicitly included will be included in the output data.

Renaming Columns

To rename a column, add it to columnsToRename, which is a map from original name --> desired name. Renames are applied before pseudonymization.

This feature supports simple adaptation of existing data pipelines for use in Worklytics.

See Also

Record-Oriented Formats (RecordRules)

As of Oct 2023, this is a beta feature

RecordRules parses files as records, presuming the specified format. It performs transforms in order on each record to sanitize your data, and serializes the result back to the specified format.

eg.

Each transform is a map from transform type --> to a JSONPath to which the transform should be applied. The JSONPath is evaluated from the root of each record in the file.

The above example rules applies two transforms. First, it redacts $.summary - the summary field at the root at of the record object. Second, it pseudonymizes $.email - the email field at the root of the record object.

transforms itself is an ordered-list of transforms. The transforms should be applied in order.

CSV format is also supported, but in effect is converted to a simple JSON object before rules are applied; so JSON paths in transforms should all be single-level; eg, $.email to refer to the email column in the CSV.

See Also

Mixing File Formats

As of Oct 2023, this feature is in beta and may change in backwards incompatible ways

You can process multiple file formats through a single proxy instance using MultiTypeBulkDataRules.

These rules are structured with a field fileRules, which is a map from parameterized path template within the "input" bucket to one of the above rule types (RecordRules,ColumnarRules) to be applied to files matching that path template.

Path templates are evaluated against the incoming file (object) path in order, and the first match is applied to the file. If no templates match the incoming file, it will not be processed.

Configuration

Worklytics' provided Terraform modules include default rules for expected formats for hris, survey, and badge data connectors.

If your input data does not match the expected formats, you can customize the rules in one of the following ways.

NOTE: The configuration approaches described below utilized Terraform variables as provided by our gcp and aws template examples. Other examples may not support these variables; please consult the variables.tf at the root of your configuration. If you are directly using Worklytics' Terraform modules, you can consult the variables.tf in the module directory to see if these variables are exposed.

Custom Rules of Predefined Bulk Connector (preferred)

You can override the rules used by the predefined bulk connectors (eg hris, survey, badge) by filling the custom_bulk_connector_rules variable in your Terraform configuration.

This variable is a map from connector ID --> rules, with the rules encoded in HCL format (rather than YAML as shown above). An illustrative example:

This approach ONLY supports ColumnarRules

Custom Bulk Connector

Rather than enabling one of the predefined bulk connectors providing in the worklytics-connector-specs Terraform module, you can specify a custom connector from scratch, including your own rules.

This approach is less convenient than the previous one, as TODO documentation and deep-links for connecting your data to Worklytics will not be generated.

To create a Custom Bulk Connector, use the custom_bulk_connectors variable in your Terraform configuration, for example:

The above example is for ColumnarRules.

Direct Configuration of RULES

You can directly modify the RULES environment variable on the Psoxy instance, by directly editing your instance's environment via your hosting provider's console or CLI. In this case, the rules should be encoded in YAML format, such as:

Alternatively, you can remove the environment variable from your instance, and instead configure a RULES value in the "namespace" of your instance, in the AWS Parameter Store or GCP Secret Manager (as appropriate for your hosting provider).

This approach is useful for testing, but note that if you later run terraform apply again, any changes you make to the environment variable may be overwritten by Terraform.

JSON Filter is inspired by , but with the goal to filter documents rather than validate. As such, the basic idea is that data nodes that do not match the filter schema are removed, rather than the whole document failing validation.

The goal of JsonFilter is that only data elements specified in the filter pass through.

Some differences:

• required properties are ignored. While in JSON schema, an object that was missing a "required" property is invalid, objects missing "required" properties in a filter will be preserved.

• { } , eg, a schema without a type, is interpreted as any valid leaf node (eg, unconstrained leaf; everything that's not 'array' or 'object') - rather than any valid JSON.

Compatibility goals:

• a valid JSON Schema is convertible to valid JSON filter (with JSON schema features not supported by JSON filter ignored)

Motivations

{ } as "any valid leaf node"

1. compactness, esp when encoding filter as YAML. can put { } instead of { "type": "string" }

2. flexibility; for filtering use-case, often you just care about which properties are/aren't passed, rather than 'string' vs 'number' vs 'integer'

3. "any valid JSON" is more common use case in validation than in filtering.