A mailer, member database, and so much more, for digital activism.
Setup docs are written for Mac OSX and Debian-based Linux distributions like Ubuntu. If you have any trouble with these instructions please let us know on Slack or file an issue here.
You will need Ruby, Postgres, Redis, NodeJS, and Git. We recommend using rbenv and nvm for Ruby and Node version management, respectively. If you already have these installed, you can skip this section if you already have these installed. If not, continue on…
Before installing any dependencies you should update your package manager with brew update or apt update.
brew install postgresql on Mac/OSX
sudo apt install postgresql on Linux
libpq-dev:
brew install libpq-dev on Mac/OSX machine
sudo apt install libpq-dev on Linux machine
rbenv install 3.1.2gem install bundlerbrew install nvm or for linux,
see the nvm docs and try the curl method to install.
nvm install nodebrew install node or
sudo apt install nodejs – nodejs setup docs.Note: If any of these steps gives you trouble, make sure you check below for common setup issues, email or ask on Slack.
stable branch:
git clone git@github.com:the-open/identity.git
cd identity
git checkout stable
bundle install
.env.development.sample to .env.development
1.1. Check the value for DATABASE_URL – the default works with Postgres.app, but there’s another value (commented-out) that will work for Linux usersbundle exec rails db:setupbundle exec rails dev:generate_fake_data
bundle exec rails mailer:dev:generate_fake_data./start.sh (which is just a one-line script that runs Foreman)Lots of tasks like sending emails, processing CSVs, and sending blasts, require Sidekiq to work properly, and this is often a source of trouble when getting your Identity instance up and running. See here for some tips on troubleshooting Sidekiq issues.
If you’re getting lots of ActiveRecord::ConnectionTimeoutErrors - this is caused by the ActiveRecord connection pool being saturated. There seems to be an issue where connections are not always checked in. Two practical steps you can take: Set your pool size to be slightly greater than your Sidekiq concurrency. Set your reaping_frequency to be quite frequent - say 10 seconds.
Errors like
'split': invalid byte sequence in US-ASCII (ArgumentError)
may occur when running the server with ./start.sh or Foreman. This may be fixed by running the following:
RUBYOPT="-KU -E utf-8:utf-8"
Sometimes libpq-dev fails to install and you need to google it or ask someone on Slack.
Sometimes you’ll need to explicitly tell Rails that you’re in “development mode” by setting the RACK_ENV, like this: export RACK_ENV=development. This is how the application knows to pick up vars in .env.development.
Identity consolidates all environment variables and settings into a single Settings variable for use in application code. If at any point you’d like to check what Settings are being used, enter the rails console with bundle exec rails c inspect the Settings object directly. Note that any change to Settings requires a server restart to take effect.
Have a read through the main settings file, which shows how all the application settings link up with all the environment variables.
Create a file config/settings/your-org.yml with the contents:
app: domain: ‘localhost:3000’ home_url: ‘http://your-org.net’ org_title: ‘Your Campaigning Organisation’
Some settings are now being cached in the AppSetting model, so they can be changed while the application is running. For now, this means those settings still default to the values defined in .yml files, but their ‘live’ state can only be changed from the app admin screen.
Optional: Check out some of the other organisations’ settings in config/settings to see how to set your time zone, currency symbol, and much more!
In .env.development, file set ORG_NAME=your-org to pick up settings from your-org.yml
Put secrets like API keys and passwords in .env.development, and non-secret settings that you’ll commit to the repository in your-org.yml.
Heroku users: Because the .env.development file contains secrets, it’s never version controlled, so it won’t be deployed to Heroku. Instead, use Heroku config variables.
In this section, we’ll refer to settings by the setting name, rather than the environment variable. For example you might set Settings.app.locale by setting ENV['APP_LOCALE'] but for the purposes of this readme we’ll refer to this as app.locale. You can review settings.yml to see which env variables these correspond to.
By this point, you should already have app.org_name and app.database_url populated via the ENV variables ORG_NAME and DATABASE_URL. These are key!
There is also database connection setting called redshift_url – if you’re not using Redshift (e.g. testing in development), you can leave this blank and it will fall back to the main database_url.
If you have a non-standard Postgres setup, you may need to enter your own Database URL. Remember the format looks like this:
DATABASE_URL=postgres://identity_dbuser:password@localhost/identity_development
See here for more on Postgres connection strings.
When you’re in dev mode locally, you can skip user athentication altogether with skip_auth=true. In production, this won’t work, but you can at least bypass two-factor authentication use_2fa=false. (See Section V, “Production Readiness” for more on configuring 2FA.)
You can test the app with email.backend=fake, but to send samples or blasts, you’ll need to change this to either ses, sendgrid or mailjet. If you already have a Sendgrid API Key, you can save your email backend to “sendgrid” and set your SENDGRID_APIKEY and emails should work. (See Section V: Production Readiness, for more on Sendgrid, SES and Mailjet setup.)
aws.access_key_id, aws.secret_access_key, and aws.region are required for any of the other AWS services to work.aws.s3_bucket_name is required to use images in emails, or use CSVs for datasets, member- and action-imports.aws.s3_endpoint is required to use a Amazon S3 compatible object storage server (e.g. minio)aws.s3_region and aws.ses_region will fall back to aws.region if left empty.You’ll need an AWS S3 bucket to upload images or import files; choose the bucket you want, or create a new one through the AWS console, and then set its permissions so we can upload files to it and those files can be viewed by people receiving our emails.
Log into your AWS Console, go to S3, and create a new bucket. After creating an S3 bucket, change permissions -> CORS configuration rules to the following (but replace your.identity.net with your actual Identity URL):
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>https://your.identity.net</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<MaxAgeSeconds>500</MaxAgeSeconds>
<ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>https://your.identity.net</AllowedOrigin>
<AllowedMethod>HEAD</AllowedMethod>
</CORSRule>
</CORSConfiguration>
It’s also worth adding some lifecycle rules for expiring your S3 objects, you can find documentation about the types of objects Identity creates and how to handle them in the S3 doc
Most groups start off testing or using Identity on Heroku, but there are also docs to support Docker and Vagrant builds. Several organisations run via Docker in production.
If you’ve already completed parts I, II, and III, you should have a working version of Identity running locally with your own organisation’s settings file. In this section we’ll talk about how to deploy this kind of staging app for you and your campaigners to test out. (Step V covers “Production Readiness”.)
You’ll need a Heroku account and the Heroku CLI installed.
Modify the heroku.yml file and change the env variables. You can also edit the addon settings to use larger databases.
Create a new Heroku app. For most cases, the proper command will be
heroku create identity-{orgname}-staging --region eu
Make sure you’ve committed your org settings file to a git branch, and then push it to the master branch of your heroku remote.
git checkout -b deploy/{yourorg}
git add config/settings/{yourorg}.yml
git commit -m "Add settings for my new organisation"
git push heroku deploy/{yourorg}:master
In the Heroku dashboard, make sure you have web, clock and sidekiq dynos running. (Imports and test emails rely on these processes.)
Further reading: This helpful article about managing local development copies, staging and production apps on Heroku.
Two-Factor Auth: In production you’ll need to set up Authy for two-factor authentication. If you don’t already have an Authy API key, create an account on authy.com and register a new “app” for your Identity deployment.
Once have your keys, set them under Heroku settings.
AUTHY_APIKEY={ add your key here }
USE_2FA=true
Email Service Provider: Identity currently supports 3 ESPs for sending bulk email. (SMTP is not really supported.) Amazon SES is the most cost-effective service and has the best integration. Sendgrid is the easiest to test on Heroku. Some use Mailjet for superior Security Privacy assurances or stricter intepretation of GDPR obligations.
Sendgrid:
heroku addons:create sendgrid:starter
Retrieve your new Sendgrid user name and password from the configuration variables SENDGRID_USER and SENDGRID_PASSWORD and log in to sendgrid.com with those credentials.
Go to API Keys and, retrieve it and set it as SENDGRID_APIKEY
Amazon SES: Click here for Amazon SES setup.
Mailjet: TODO: docs on Mailjet setup
Redis “noeviction”: Check our Redis docs for extra info, but our recommended setup includes two redis databases:
| config env variable | Recommended Size | Recommended Eviction Policy |
|---|---|---|
| SIDEKIQ_REDIS_URL | (max recipients in 24 hours / 2,000)MB plus 10MB buffer | noeviction |
| REDIS_URL | 30MB | allkeys-lru |
If you are running on Heroku, it’s reccommended by the Sidekiq docs that you set the MALLOC_ARENA_MAX=2 ENV variable to greatly reduce memory usage, which has also been observed by a few orgs. This is set by default for new Heroku applications.
Database Connection Security: If you are connecting to a database using a public endpoint, it’s critical to use SSL to ensure data is encrypted in transit and you are not vulnerable to man in the middle attacks. Here’s a Guide on setting up SSL for RDS databases
NOTE: The default database sslmode is verify-full. This both enforces SSL, and validates the certificate of the database server to ensure it is not being impersonated (a MITM attack). Initially you may see deploy errors like:
ActiveRecord::NoDatabaseError: root certificate file "/app/.postgresql/root.crt" does not exist
Either provide the file or change sslmode to disable server certificate verification.
If your database is provided via Heroku Postgres
It is not currently possible to use verify-full with Heroku databases. So you should use sslmode=require, either in the database connection string, or by setting DATABASE_SSL_MODE to require. You need to do this for DATABASE_URL.
Other setups
You will need to make sure that the root CA certificate for the database you’re connecting to is available and change your database configuration for DATABASE_URL to connect to it. To do this add the parameter sslrootcert=path/to/cert to your database connection strings.
If you are using RDS, the CA certificate you need is already included in the repo at db/rds-combined-ca-bundle.pem.
Not recommended: If you’re unable to supply a root CA certificate then you can change the SSL mode to require using ?sslmode=require. This will ensure data is encrypted in transit, but cannot protect against a MITM attack.
Secret Key Base: To secure your logins, you’ll want to set a new SECRET_KEY_BASE; generate one with bundle exec rails secret and set it as an ENV variable.
URL Signing Key: To secure your redirection URLs (generated for click tracking in emails), set a new URL_SIGNING_KEY; generate one with bundle exec rails secret and set it as an ENV variable.
HTTPS Encryption: Make sure Settings.app.ssl is true, and set up an SSL certificate so all your campaigner logins are secure and encrypted.
GDPR Consent Settings: If you’re in the European Union you’ll need to set up member consents to comply with the GDPR.
Send SMS: If you’re using Twilio to send SMS messages, see here for how to create and find your Twilio API keys. We also support Plivo and Nexmo. See here for info about Nexmo.
Amazon Redshift: If you can afford it, and you have more than a couple hundred thousand members, you may want to set up Redshift. It improves performance for things like complex list cutting and optimising the order of mailings for better open rates.
Error Logging/Alerts: Set up an Airbrake/Bugsnag account for error reporting. Set the env variables expected (look at settings.yml for variable names)
Controlshift Labs Integration: If your organisation uses ControlShift Labs for distributed petitions, you may want to set up a CSL integration.
Freshdesk: If you use Freshdesk, try out this lightweight Freshdesk integration.
Fun Gifs: Add a list of loading gifs to use with the env variable
LOADING_GIFS=https://path.to.identity.app/images/loading.gif can be used as default
Read about using multiple mailing back-ends to more precisely manage email reputation.
Learn about the fancy machine-learning de-duplication service.
If you would like to use Amazon SQS for improved performance parsing incoming payloads (especially signatures from CSL) set your SQS should be in AWS region matching Settings.aws.region.
TODO: More info on SQS setup.
Setting up SNS to get bounce/spam notifications
AWS_SNS_CONFIRM_SUBSCRIPTIONS=truespam and bounceAWS_SNS_CONFIRM_SUBSCRIPTIONS to disable callback mechanism (identity will connect to amazon url passed in confirmation JSON)TODO: More on SNS, what it’s used for.