Resources for Developers

Elasticsearch

Elasticsearch is the official addon provided by Scalingo. Details on the available plans can be found here. This addon gives your app instant access to an Elasticsearch database.

Adding Elasticsearch to your app

You can add the Elasticsearch addon through the Dashboard or through the command-line interface. The capacity of your database is elastic, you will be able to upgrade it later.

Through the Dashboard

  1. Go to your app on the Scalingo dashboard
  2. Click on Addons tab
  3. Select the addon you want to add
  4. In the dialog select the database plan you need
  5. Validate your choice

Through the command-line interface

$ scalingo -a my-app addons-add elasticsearch 1g

-----> Addon elasticsearch has been provisionned
       ID: my-app-3030
       Modified variables: [ELASTICSEARCH_URL SCALINGO_ELASTICSEARCH_URL]
       Message from addon provider: Database successfully created

This command will provision the application my-app with a 1g Elasticsearch database plan.

To find out what other plans are available:

$ scalingo addons-plans elasticsearch

Getting your connection URI

Once the addon is provisioned, 2 environment variables are added to your app: SCALINGO_ELASTICSEARCH_URL and ELASTICSEARCH_URL. ELASTICSEARCH_URL is an alias to SCALINGO_ELASTICSEARCH_URL. To find out how to use it in your code please refer to Application environment.

In most cases, you can pass the variable directly to the client library you are using in your code. But sometimes the library requires a specific URI format, you’ll need to add a little bit of code to suit the library.

You can get environment variables from the dashboard or the command line interface.

From the Dashboard

  1. Go to your app on Scalingo Dashboard
  2. Click on Environment tab
  3. SCALINGO_ELASTICSEARCH_URL is displayed

From the command line interface

$ scalingo -a my-app env | grep ELASTIC

ELASTICSEARCH_URL=$SCALINGO_ELASTICSEARCH_URL
SCALINGO_ELASTICSEARCH_URL=http://my-app-3030:MpIxXstskccB3Ab2iwKH@my-app-3030.elasticsearch.dbs.appsdeck.eu:30995

Remote access your database

The difference between this data storage engine and the others is that Elasticsearch communicates over HTTP, so you can use any tool built for that. You don’t have to use an official client. The most common command line tool is curl. The following example uses it.

Use scalingo run

scalingo --app <application name> run curl http://<user>:<password>@<host>:<port>

Example

If my application’s name is ‘my-app’ and it has the environment variable SCALINGO_ELASTICSEARCH_URL = "http://my-app-123:H_grwjqBteMMrVye442Zw6@my-app-123.elasticsearch.dbs.scalingo.com:30000/my-app-123"

You can run:

$ scalingo --app my-app run curl $SCALINGO_ELASTICSEARCH_URL
{
  "status" : 200,
  "name" : "the Tomorrow Man Zarrko",
  "version" : {
    "number" : "1.3.4",
    "build_hash" : "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
    "build_timestamp" : "2014-09-30T09:07:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}

Force TLS connections

Our Elasticsearch image support TLS to encrypt all of its network traffic between the client and the server.

Elasticsearch cannot listen to connections with and without TLS. If you want to encrypt communications with your Elasticsearch databases, you need to force all connections to use TLS. Forcing TLS connections is as simple as heading to the database dashboard and clicking on the toggle button:

Note that you must have configured your application to use TLS when connecting to the database.

curl --insecure -X GET <URL>

With URL starting with https://.

The --insecure option is mandatory as the generated certificates for your databases are self-signed. If you want the certificate to be trust-able, you need to download our certification authority certificate and specify it to the MongoDB CLI tool.

Some existing databases may not have yet TLS support. To activate TLS, you need to restart the database. Any action leading to the restart will activate TLS (e.g. plan update, upgrade of the database).

Download the CA certificate

The certificate of our certification authority is available on the database dashboard.

After downloading it, you can specify its path:

curl --cacert=/path/to/ca.pem -X GET "<URL>"

Changing plans

You can upgrade or downgrade your database plan whenever you need it. This operation happens instantly thanks to Docker containers and no manual input is required. When you change the plan, your database will be stopped then simply restarted on a new host with new parameters of the chosen plan. During the operation the connection is dropped between your app and the database. Finally, after the operation is successful, the related app will be restarted.

From the Dashboard

  1. Go to your app on Scalingo dashboard
  2. Click on Addons tab
  3. Select the addon you want to change
  4. In the dialog select the plan you want to upgrade/downgrade to
  5. Validate your choice

From the command-line interface

To upgrade or downgrade your addon the sub-command is the same: addons-upgrade.

$ scalingo -a my-app addons-upgrade my-app-3030 2g

In this example, my-app-3030 is the ID of the addon, and 2g is the plan we want to upgrade to.

To find out the addon ID:

$ scalingo -a my-app addons

+------------------------+-------------+------+
|          ADDON         |      ID     | PLAN |
+------------------------+-------------+------+
| Elasticsearch | my-app-3030 |   1g |
+------------------------+-------------+------+

Database dashboard

The Elasticsearch dashboard is the central place for administrative tasks such as:

  • Monitor database and system stats
  • Upgrade the database engine version
  • Activate database specific features
  • Manage database users
  • Manage backups

Database Upgrade

When the database vendor releases a new version of your database engine, we will try to provide it as soon as possible. You will have the choice to upgrade your database with just one click through your database dashboard.

This operation is similar to changing your database plan. Your database will be stopped and restarted with new database environment. Thanks to Docker containers this happens seamlessly and quickly without manual action. When this operation finishes, your application will be restarted.

Container Stats

CPU usage Current CPU usage.
Memory usage Display the current, hightest and free memory. Highest is the maximum memory recorded since database restarted.
Swap usage Display the current, hightest and free swap. Highest is the maximum swap recorded since database restarted.

Database Stats

Database on disk size Effective physical space used.

Backups

If your database is in a paid plan (i.e. It’s not “free plan”), we’ll make automated backups of your database on a daily basis, at around 1 AM Central European Time (CET or UTC+0100). However, we don’t keep those backups forever. The following rules are applied for backups storing:

  • Everyday for the past 30 days (1 month)
  • One per week during the 2nd and 3rd months
  • One per month during the 4th, 5th and 6th months

Download automated backups

Automated backups are listed in the database specific dashboard.

  1. Go to your app on Scalingo Dashboard
  2. Click on Addons tab
  3. Click Link to dashboard which will take you to the Elasticsearch dashboard
  4. Click on Backups tab
  5. Download the backup you want

Restoring a backup

Download the backup file

Go to your database dashboard, in the Backups tab and download the last dump you want to restore locally.

The archive is a .tar.gz containing a dump of your database. Backups are done using Elasticsearch Knapsack for Elasticsearch 1.x or 2.x.

Knapsack is a plugin to handle dump and restore of Elasticsearch data, you need to install it locally, please follow instructions on the README of the project.

Since Elasticsearch 5.x, we use the snapshot/restore feature of Elasticsearch.

Restoring Knapsack Backup

Knapsack expects to find a file named _all.tar.gz in the directory path.logs of Elasticsearch. This parameter can be set in your Elasticsearch configuration file, otherwise it will be equivalent to /

Rename the downloaded file to _all.tar.gz and move it to the path.logs file of your installation:

$ mv 20160101010000-db-1234.tar.gz /_all.tar.gz

Then run the following curl command to trigger Knapsack importation:

$ curl -X POST http://localhost:9200/_import -d ''
{"running":true,"state":{"mode":"import","started":"2016-04-27T09:38:41.202Z","path":"file:///_all.tar.gz","node_name":"Miguel Espinosa"}}

That’s it the importation has started, you can follow it by running a request against the state endpoint:

└> curl '172.17.0.2:9200/_import/state' -d ''
{"count":2,"states":[{"mode":"import","started":"2016-04-27T09:38:41.202Z","path":"file:///_all.tar.gz","node_name":"Miguel Espinosa"},{"mode":"import","started":"2016-04-27T09:40:11.889Z","path":"file:///_all.tar.gz","node_name":"Miguel Espinosa"}]}%

Once ended, you’ll have all your data locally.


mode_edit Suggest edits