Ingesting Logs with Logstash

Logstash is a data processing pipeline, taking a multitude of source data as inputs. It is able to transform, enrich, and aggregate data on the fly before forwarding it to the chosen destination (usually an OpenSearch® database).

In a logging stack, Logstash acts a logs ingestion solution. Its role is to receive logs from the logs collector, transform them and send them to the appropriate destination.

Deploying

Using the Command Line

We maintain a buildpack and a repository called logstash-scalingo on GitHub to help you deploy Logstash on Scalingo. Here are the few extra steps to follow:

Clone our repository:

git clone https://github.com/Scalingo/logstash-scalingo
cd logstash-scalingo

Create the application on Scalingo:
```
scalingo create my-logstash
```

(optional) Scale the container to an L size:

scalingo --app my-logstash scale web:1:L

(optional) Provision a Scalingo for OpenSearch® addon:

scalingo --app my-logstash addons-add opensearch opensearch-starter-1024

Edit the logstash.conf file to change the index name of the OpenSearch® output. The goal is to make it fit semantically to the data being ingested
Create a few environment variables to protect your Logstash instance via HTTP basic auth (not everyone should be able to send data to your instance!):
```
scalingo --app my-logstash env-set USER=logstash-username
scalingo --app my-logstash env-set PASSWORD=logstash-password
```
Everything’s ready, deploy to Scalingo:
```
git push scalingo master
```

Using the Terraform Provider

Start by forking our Logstash repository

Place the following block in your Terraform file to create the app:

resource "scalingo_app" "my-logstash" {
  name        = "my-logstash"
  force_https = true

  environment = {
    USER     = "<logstash-username>"
    PASSWORD = "<logstash-password>"
  }
}

Link the app to your forked repository:

data "scalingo_scm_integration" "github" {
  scm_type = "github"
}

resource "scalingo_scm_repo_link" "default" {
  auth_integration_uuid = data.scalingo_scm_integration.github.id
  app                   = scalingo_app.my-logstash.id
  source                = "https://github.com/<username>/logstash-scalingo"
  branch                = "master"
}

Attach a Scalingo for OpenSearch® addon to your app:

resource "scalingo_addon" "my-logstash-opensearch" {
  app         = scalingo_app.my-logstash.id
  provider_id = "opensearch"
  plan        = "opensearch-starter-1024"
}

(optional) Instruct the platform to run the web process type in a single L container:

resource "scalingo_container_type" "web" {
  app    = scalingo_app.my-logstash.id
  name   = "web"
  size   = "L"
  amount = 1
}

Edit the logstash.conf file to change the index name of the OpenSearch® output. The goal is to make it fit semantically to the data being ingested
Once Terraform is done, your Logstash instance is provisioned and ready to be deployed. The deployment itself requires an extra manual step:
1. Head to your dashboard
2. Click on your Logstash application
3. Click on the Deploy tab
4. Click on Manual deployment in the left menu
5. Click the Trigger deployment button
6. After a few seconds, your Logstash instance is finally up and running!

Testing

Once your Logstash is up and running, you can try to send some data to it:

curl -X POST 'https://<logstash-username>:<logstash-password>@my-logstash.osc-fr1.scalingo.io?name=whatever' --data 'Hello World!'
ok

Check the indices that are stored in the OpenSearch® database.
For example, from a one-off container:

> curl $SCALINGO_OPENSEARCH_URL/_cat/indices

yellow open logs-2025.05.16 _0XNpJKzQc2kjhTyxf4DnQ 5 1 1 0 6.6kb 6.6kb

Logstash has created the logs-2025.05.16 index which can now be requested:

> curl $SCALINGO_ELASTICSEARCH_URL/logs-2025.05.16/_search | json_pp
{
  "_shards" : {
    // [...]
  },
  // [...]
  "hits" : {
    "total" : 1,
    "max_score" : 1,
    "hits" : [
      {
        "_id" : "lXeS7JYB_cS0_LxmtlLs",
        "_index" : "logs-2025.05.16",
        "_score" : 1,
        "_source" : {
          "@timestamp" : "2025-05-16T05:22:40.105007720Z",
          "@version" : "1",
          "host" : {
            "ip" : "10.0.0.164"
          },
          "http" : {
            "method" : "POST",
            "request" : {
              "body" : {
                "bytes" : "12"
              },
              "mime_type" : "application/x-www-form-urlencoded"
            },
            "version" : "HTTP/1.1"
          },
          "message" : "Hello World!",
          "tags" : [],
          "url" : {
            "domain" : "my-logstash.osc-fr1.scalingo.io",
            "params" : {
              "name" : "whatever"
            },
            "path" : "/?name=whatever"
          },
          "user_agent" : {
            "original" : "curl/8.5.0"
          }
        }
      }
    ]
  }
}

The result of the above search contains a document having a field url.params.name set to whatever and a field message set to Hello World!.

Updating

By default, Scalingo deploys a version of Logstash that is compatible with the OpenSearch® instances we provide.

Consequently, updating Logstash consists in triggering a new deployment of your instance.

Using the Command Line

In your Logstash repository, create an empty commit and push it to Scalingo:

git commit --allow-empty --message="Update Logstash"
git push scalingo master

Using the Terraform Provider

Head to your dashboard
Click on your Logstash application
Click on the Deploy tab
Click on Manual deployment in the left menu
Click the Trigger deployment button
After a few seconds, your updated Logstash instance is ready!

Interfacing with Log Drains

Log drains are a Scalingo feature that act as a log collector. Once activated, a log drain automatically sends every log entries generated by an application to the configured destinations.

To make a log drain forward logs to a Logstash instance, use the elk type of log drain:

scalingo --app my-app log-drains-add --type elk --url <logstash_url>

With logstash_url being the URL of your Lostash app.

The elk log drain sends the log entries over HTTP, with a content-type header set to plain/text.
The application name (appname) and the container name (hostname) are passed as query parameters and the log entry itself is in the request body.

The configuration file (logstash.conf) provided in our logstash-scalingo repository is a basic example, known to work with our log drain. Feel free to adjust it to your needs.

Customizing

Configuring

The repository we provide help deploy your Logstash instance comes with a directory named config. All the files contained in this directory are copied in the Logstash configuration directory at runtime, allowing you to precisely customize your instance.

By default, this directory is empty, which means a default deployment relies on Logstash defaults.

For example, if you’d want to modify the logging behavior of Logstash, you could edit the config/log4j2.yml file.

Environment

The following environment variable(s) can be leveraged to customize your deployment:

USER
Logstash administrator account name.
mandatory
PASSWORD
Logstash administrator password.
mandatory
LOGSTASH_PLUGINS
Comma separated list of plugins to install.
Defaults to not being set.
LOGSTASH_VERSION
Version of Logstash to deploy.
Defaults to 9.0.1

Last update: 16 May 2025 Suggest edits