Setting up ControlShift Labs Integration

A mailer, member database, and so much more, for digital activism.

Setting up ControlShift Labs Integration

Identity can be setup to pull data from ControlShift Labs (CSL) platform.

Pushing Controlshift data into Identity

The first and most fundamental step in setting up a CSL <-> Identity integration is getting data about who has signed petitions, subscribed to emails, etc into Identity. There are multiple ways to do this, and two most commonly used methods are listed below. However, both involve some common setup steps in Identity.

Common Configuration in Identity

  1. Create an APIUser in Identity for Controlshift in the rails console. This will ensure that only valid data from Controlshift is processed by Identity - and that malicious 3rd parties cannot send bad data into Identity! Once you have created the APIUser, print out the Api token (we’ll need this later!)
csl = APIUser.create_with_permission!('csl-webhook', name: "controlshift", active: true)
csl.token

  1. To import CSL Signatures (into the Identity MemberAction table), set the PROCESS_CBY_ACTIONS Identity ENV VAR to true. Note, that when actions are recorded, Member records for the user who signed the petition are also created automatically.

  2. Optional: To import CSL Members (into the Identity Member table), set the PROCESS_CBY_MEMBERS Identity ENV VAR to true. Note, that this is not required, since users signing petitions will have Member records created anyway - and having this option enabled has caused issues for some orgs, such as empty names for members. Therefore a number of organisations currently leave this option disabled.

  3. PROCESS_CBY_ACTIONS means actions (CSL signatures) from the incremental CSL webhook are processed by ID. If you also want to process nightly full exports of all historic actions (CSL signatures) you also need to set the Identity ENV VAR PROCESS_CBY_FULL_ACTIONS to true, as well as ticking the ‘nightly csv exports’ box in the CSL Admin UI (this is covered later as well). If you have more than 400k signatures in CSL and you’re running on heroku, it is recommended to leave the PROCESS_CBY_FULL_ACTIONS ENV VAR turned off, because processing this many events every night will take a long time.

  4. The LOG_ALL_CONTROLSHIFT_WEBHOOKS Identity ENV VAR can be set to true for extra logging of the webhooks Identity processes.

  5. If you want to, set CONTROLSHIFT_API_RATE_LIMIT (defaults to 1000 per interval).

CSL webhooks direct to Identity

This is the simplest and most commonly used method to get CSL data into Identity.

How CSL webhooks direct to ID works

Identity listens for CSL webbooks notifying it of CSL data that has been exported to CSV files. It then queues background jobs to download these CSV files and import them. The detailed workflow is as follows:

More information on the schema of the exported tables can be found in the CSL developer docs.

Setting up CSL webhooks direct to ID

  1. Login to your CSL instance as an Admin, and navigate to Settings, then Integrations

Settings -> Integrations -> Webhooks screen

  1. Click on the NEW WEBHOOK ENDPOINT button, and enter the following URL https://<instance-url>/api/ctrlshift_webhook?api_token=<api-token>, replacing <instance-url> with the URL of your Identity instance and <api-token> with the APIUser token you generated in ID earlier.

New Webhook Endpoint screen

  1. Click Save, and you should see that the webhook endpoint has been saved and Enabled. If ControlShift can’t access your Identity instance or the webhook, instead of this you’ll see an error explaining why.

Saved Webhook screen

  1. Tick the ‘incremental csv exports’ and ‘nightly csv exports’ boxes according to your organizations needs. Incremental exports are frequent CSV exports, recommended if you need any realtime analytics of CSL signatures. Nightly is a once-per-day bulk export of all data. Usually organisations enable both of these.

  2. TODO: Document how to setup HMAC signing of CSL payloads

CSL webhooks to Identity via an Amazon Web Services Queue

This option may be useful if you are experiencing heavy load on your Identity instance, or if you are worried about downtime on your Identity instance leading to loss of events from CSL.

However, this method is significantly more complex, and involves utilising multiple services in Amazon Web Services.

With this method, CSL webhook events are pushed onto an Amazon Web Services queue first, and the events from the AWS queue are then pulled off by a process running in Identity.

How CSL webhooks via AWS SQS works

CSL is configured to push events to an AWS endpoint, which then stores the event in a SQS queue. A process in Identity then reads events from this queue and processes them in the same way as the direct CSL -> Identity method.

Setting up CSL webhooks via AWS SQS

  1. Create an APIUser in Identity for Controlshift in the rails console. This will ensure that only valid data from Controlshift is processed by Identity - and that malicious 3rd parties cannot send bad data into Identity via the CSL endpoint! Once you have created the APIUser, print out the Api token (we’ll need this later!)
csl = APIUser.create_with_permission!('csl-webhook', name: "controlshift", active: true)
csl.token

  1. Create an AWS SQS queue to queue CSL webhook events for processing by Identity. You should refer to AWS documentation on exactly how to create this, since the AWS UI may change over time.

  2. Create a REST-based AWS API Gateway. You should refer to AWS documentation on exactly how to create this, since the AWS UI may change over time. But you want a REST API endpoint which CSL can POST to, and which pushes data to an AWS Lambda.

AWS API Gateway

  1. Ensure Query parameters, HTTP headers, etc are passed through to the Lambda function. The AWS documentation on this is not great. This article explains how to pass HTTP Header through, but it doesn’t deal with URL Query parameters. So the following explains how to set this up, correct at the time of writing: When configuring the Integration Request for the AWS API Gateway, select an Integration Type of ‘Lambda Function’ and expand the ‘Mapping Templates’ section. For Request Body Passthrough, select ‘When there are no templates defined (recommended)’. Add a mapping template with Content-Type of ‘application/json’, and then in the Generate Template drop-down box, select ‘Method Request Passthrough’. This will generate code in the text box which will take HTTP headers, URL query parameters, and other context of the HTTP request, and pass this through to the Lambda function. There are also comments in the sqs.rake file which detail the message format required in case eg. the code generated by AWS changes.

  2. Setup your Lambda function to push to your SQS queue. The sample code below does this, and works with a Node.js 12.x runtime:

var AWS = require('aws-sdk');
var sqs = new AWS.SQS({region : '<YOUR_AWS_SQS_REGION>'});
var http = require('https');

var postToSQS = function(data, callback) {
    sqs.sendMessage({
        MessageBody: JSON.stringify(data),
        QueueUrl: '<YOUR_SQS_QUEUE_URL>'
    }, function(err, data) {
        callback(err, data);
    });
};

exports.handler = function(event, context) {
    postToSQS(event,  function(err, data) {
        if (err) {
          console.log('error:', "Error posting to SQS: " + err);
          context.done('error', "Error sending action data");
        } else {
          console.log('Success', 'Action posted');
          context.done(null, '');
        }
    });
};

  1. You may wish to limit access to the AWS API Gateway to only the specific CSL IP Addresses. This AWS documentation explains how to do so. Note, restrictions are applied to your entire API Gateway, not to just a single endpoint. If you use the same API Gateway for other endpoints (eg. Speakout -> AWS -> Identity data) then you need to ensure the source IPs which use those endpoints are also allowed!

  2. Login to your CSL instance as an Admin, and navigate to Settings, then Integrations

  3. Click on the NEW WEBHOOK ENDPOINT button, and enter your AWS API Gateway URL, adding an api_token URL parameter, eg: https://<aws-api-gateway-url>?api_token=<api-token>, replacing <aws-api-gateway-url> with the URL of your AWS API Gateway, and <api-token> with the APIUser token you generated in ID earlier.

  4. Click Save, and you should see that the webhook endpoint has been saved and Enabled. If ControlShift can’t access your AWS API Gateway, you’ll see an error explaining why.

  5. Tick the ‘incremental csv exports’ and ‘nightly csv exports’ boxes according to your organizations needs. Incremental exports are frequent CSV exports, recommended if you need any realtime analytics of CSL signatures. Nightly is a once-per-day bulk export of all data. Usually organisations enable both of these.

  6. TODO: Document how to setup HMAC signing of CSL payloads

  7. In your Identity environment, set the AWS_REGION, AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY ENV VARs if not set already. The AWS IAM user for Identity must also have access to read from the SQS queue you setup earlier.

  8. In your Identity environment, set the SQS_WEBHOOKS_QUEUE ENV VAR to the name of your SQS queue - just the name, not the full URL!

  9. Finally, ensure that your Identity installation has a process running the sqs:process_webhooks rake task (see the sqs.rake file). The Identity Procfile includes a line to run this, so if you deploy to Heroku then it is likely this is already running, but you should check (eg. in Heroku, ensure the worker_ctrlshift_sqs process has 1 dyno running)

Configuring Consents

If you want a nice (and rigorous) connection between CSL opt-ins / consents and Identity, you can follow these steps to configure your consent texts and subscriptions. Check out the separate docs on user consents for more background.

  1. Enable Settings.gdpr.disable_auto_subscribe by adding it your org’s settings .yml file. This guarantees that the default behaviour (subscribing members based on CSL’s join_organisation column without creating consents) will not be used.
  2. Configure your consent text in Identity and CSL.
    1. In CSL, this means adding (or editing) an entry at https://<YOUR.CSL.URL>/org/email_opt_in_types/advanced. Currently reached via Settings > Email Opt-In Types > Manage Email Opt-In Types. Go ahead and set an External ID (this is just a name, pick anything you like, for example email_opt_in_v1). Grab the ID of your opt in to use later, you can see this in the URL when you click through like org/email_opt_in_types/advanced/123 would be opt-in ID 123.
    2. In Identity, this means creating a few new records. Below is an example you could run in a rails console (feel free to use SQL instead) but… DO NOT JUST COPY & PASTE THIS CODE SNIPPET. Take a moment to read and understand what you’re running, and make sure you replace everything in caps. 
       consent_text = ConsentText.create!(
   public_id: 'EMAIL_OPT_IN_V1',
   consent_short_text: "I'D LIKE TO BE EMAILED ABOUT THIS CAMPAIGN! I HOPE YOUR CONSENT TEXT IS LESS SHOUTY THAN THIS!",
   full_legal_text_link: 'https://YOUR.ORGANISATIONS/PRIVACY_POLICY'
 )
 controlshift_consent = ControlshiftConsent.create(controlshift_consent_type:"email_opt_in_type", controlshift_version_id:<PUT_YOUR_CSL_OPTIN_ID_HERE>, controlshift_consent_external_id:"EMAIL_OPT_IN_V1")

 ControlshiftConsentMapping.create(controlshift_consent_id:controlshift_consent.id,consent_text_id:consent_text.id,method:"CSL-CHECKBOX",opt_in_level:"explicit_opt_in",opt_in_option:"CONSENT TEXT AGAIN HERE",opt_out_level:"none_given",opt_out_option:"OPPOSITE OF CONSENT TEXT")
      1. public_id and controlshift_consent_external_id do not need to be the same, but your life will be less confusing and lead to greater enlightenment if they are.
      2. method is just a string, try to put something descriptive in there
      3. opt_in_level and opt_out_level must be one of no_change, none_given, implicit, explicit_not_opt_out or explicit_opt_in. For more discussion of their meaning check docs on user consents.
      4. opt_in_option and opt_out_option are designed for a consent involving radio buttons, in which case they should correspond to the labels for the Yes and No options. If you’re using a checkbox, it’s best to put a copy of your consent short text in opt_in_option and the opposite in opt_out_option.
  3. It’s probably easiest to only follow this setup for your currently active opt-in in CSL, but just be aware if you do nightly full table exports, your older opt-ins will be sent and Identity will be unhappy if it doesn’t have a mapping to recognise them.
  4. You’ll want to follow the same process for any other opt ins / consents you have configured in CSL. For example you may have a “Data processing consent label” configured if you operate in the EU. Go to https://<YOUR.CSL.URL>/org/consent_content_versions to set an external ID and find the CSL ID (hover over the Add/Edit button in the External ID column). Then follow the same process to make the corresponding records in Identity, but use a controlshift_consent_type of consent_content_version instead of email_opt_in_type.
  5. Once that’s configured, it’s time to set the env variable PROCESS_CONTROLSHIFT_CONSENTS to true in Identity.
  6. At this stage, it’s worth checking to see if your configuration is set up correctly. If you have incremental exports turned on, you can do this by signing a petition, waiting a minute, and then checking to see if a new MemberActionConsent exists.
  7. Now that you have consents being stored, you would probably like to see email subscriptions updated to match. Here comes another code snippet you should absolutely READ BEFORE COPYING… 
     PostConsentMethod.create(consent_text_id:YOUR_CONSENT_TEXT_ID,min_consent_level:4,max_consent_level:4,method:'email_subscribe')
 PostConsentMethod.create(consent_text_id:YOUR_CONSENT_TEXT_ID,min_consent_level:1,max_consent_level:1,method:'email_unsubscribe')
    1. So, first of all, make sure you pick the correct YOUR_CONSENT_TEXT_ID
    2. The concept of consent_level here relates to the opt_in_level and opt_out_level we set above. Remember the levels are no_change, none_given, implicit, explicit_not_opt_out or explicit_opt_in. In this case, after a consent is stored with a certain level (based on CSL saying checkbox ticked/unticked or which radio button was selected), we can run a certain method.
    3. For simple use cases, email_subscribe and email_unsubscribe are probably all you need, but if you want to do something more advanced feel free to take a look and add methods to member_action_consent.rb:post_consent_methods.
  8. Optionally - if you have a more complex CSL consent setup, you might want to subscribe to other subscriptions in response to some consent being given. You can do this with the post consent methods subscribe_by_id and unsubscribe_by_id. They work similarly to the code above, but just add a subscription_id to express which subscription you want to use: 
     PostConsentMethod.create(consent_text_id:YOUR_CONSENT_TEXT_ID,min_consent_level:4,max_consent_level:4,method:'subscribe_by_id', subscription_id: YOUR_SUBSCRIPTION_ID)
 PostConsentMethod.create(consent_text_id:YOUR_CONSENT_TEXT_ID,min_consent_level:1,max_consent_level:1,method:'unsubscribe_by_id', subscription_id: YOUR_SUBSCRIPTION_ID)
  9. If you want to unsubscribe members from CSL campaigns when they unsubscribe from emails in identity, enable options.unsub_csl_when_unsub_email, either by modifying your organisation’s settings file, or setting the env var UNSUB_CSL_WHEN_UNSUB_EMAIL to true

Managing Subscriptions: Alternatives to configuring consents

In some circumstances, whether GDPR does not apply to your organisation or you have your own specific legal advice about not needing separate consent records, you may not wish to configure consents between CSL and Identity, but you still want signatures in CSL to create members subscribed to email in Identity.

Disclaimer: This section is intended as advice about how identity should behave. As it is a core functionality for most organisations, with legal requirements in some jurisdictions, it is recommended that you understand and verify this setup for yourself. The best way to do this is to test it yourself - sign a petition and check if you’re subscribed in identity. The second best way is to have a techie in your organisation review Identity’s code.

Subscribe based on CSL’s ‘join_organisation’ column

  1. Disable Settings.gdpr.disable_auto_subscribe. It’s actually false/disabled by default but to avoid being impacting by the default changing in the future, it’s safest to add it to your org’s settings .yml file.
  2. All set! Now members in Identity will be subscribed to email when CSL sends actions with the join_organisation column set to ‘true’ or ‘t’ and they previously had no subscription, or their previous unsubscribe time was earlier than the action being recorded.

Subscribe all new members

NOTE: This is a very aggressive configuration. Unless you are astonishingly confident that every new member record (not just the one’s coming from CSL) should be subscribed to every channel (email, SMS, push notifications, phone, facebook), then you should consider a different configuration.

  1. Enable Settings.options.allow_subscribe_via_upsert_member - you need to add it to your org’s settings .yml file, it is false by default. This setting allows subscriptions to be modified in two ways - using defaults (with the setting below) or by passing a subscriptions hash when uploading a member CSV.
  2. Enable Settings.options.default_member_opt_in_subscriptions - you can add this to your org’s settings yml file, or simply add a DEFAULT_MEMBER_OPT_IN_SUBSCRIPTIONS environment variable. This setting has no effect unless the previous one is enabled. When both are enabled, all new members will be subscribed to all default subscriptions (email, SMS, push notifications, phone, facebook)

Setting up the Identity integration in CSL (AKA pre-filling forms AKA soft login)

If you’d like to pre-fill member details (name and email address) on CSL petition pages, you should contact CSL support and ask them to add the “id” integration to your account. They will ask for your identity host (just the domain where your identity is hosted, no https://) and API token. You can re-use the same API token from earlier in this guide, but you’ll need to grant extra permissions like this:

csl.role.permissions << Permission.find_by_slug('members-api')

After the integration is added, if you’re using consents, you can also configure CSL to avoid presenting the consent question to members who have already consented. Just follow the instructions in CSL and add the correct public ID(s).

In order for this feature to work, a member needs to click a link in an identity email, which briefly redirects them via identity and sets a cookie with their identity GUID, then CSL uses this GUID cookie to make a request to identity for the member’s details. It’s worth double checking that identity is rewriting your CSL links, you need to either have CSL running on a subdomain of Settings.app.base_domain or you can enable Settings.options.add_tracking_to_all_links to add tracking to all links. (This is true by default but that may change in future!)

User Anonymizing / Ghosting [New / Experimental]

If you would like the Anonymize Member Data button to also delete user data in CSL, you’ll need to configure Identity as an OAuth client for CSL.

  1. In CSL go to Settings > Integrations > Rest API Apps > New Application
  2. Pick a name (whatever you like, Identity would be fine)
  3. Use the suggested redirect URI of urn:ietf:wg:oauth:2.0:oob
  4. Click Authorize, you don’t need to save the authorization code.
  5. Now you’re ready to set some env variables in Identity. Set CONTROLSHIFT_API_CLIENT_ID to the Application Id, CONTROLSHIFT_API_CLIENT_SECRET to the Secret and CONTROLSHIFT_API_URL to your base CSL URL like https://my.csl.com.
  6. All done! Go ahead and try anonymizing a test user and see if their CSL data is deleted.
    1. Minor note - there’s some delay between users being created in CSL and the OAuth API returning them. So if you create a test user to anonymize, you’ll want to wait about an hour after creating them before ghosting them from Identity.

Petition Categories

Available in identity version 20.04 and later If you use petition categories in CSL and want to use that data in identity, you can import by enabling the setting features.pull_issue_categories_from_controlshift in your org’s settings yaml file. If you’re enabling it for the first time and want to pull old category data, it’s worth running ControlshiftReviseIssueCategoriesWorker.perform_async(false) once in a rails console to pull all the categories for your current petitions. It’s also synced nightly so if you change a petition’s categories, that will be reflected within 24 hours.

Once you’ve got your category data pulling into identity, you can access it via the “Issue Categories include” (has-taken-issue-category) search filter. This allows you to quickly and easily gather all the people who have signed at least one petition marked with that category.

If you’re interested in summary information about your members’ category interests, you could take a look at these SQL queries as a starting point.

Categories for speakout

If you use speakout as well as CSL, you can replicate your category setup over there. You’ll need a speakout version of v0.18.0 or later. Configure these settings in Speakout:

To get you setup with an initial list of categories matching what you have in identity (so maybe pull from CSL first ;-)), you can run bundle exec rake categories:pull_categories_from_id in Speakout. Now you’re all set and ready to configure categories in speakout campaigns. To make sure that info makes its way back to identity, make sure you enable features.pull_issue_categories_from_speakout in your org’s identity settings yaml file.

Troubleshooting / Gotchas

I have a lot of signatures to import.