Addressing

Each Private Network is always assigned an IPv4 /22 CIDR block with a fixed prefix of 10.240.0.0/22.
This allows to run a maximum of 1022 containers at the same time in a given Private Network, with IP addresses ranging from 10.240.0.1 to 10.240.3.254 included.

Put in other words, each container in a Private Network is provisioned with a network interface whose IP address falls in the 10.240.0.0/22 range. This interface is dedicated to communications within the Private Network:

• Each container in a same Private Network can directly reach the other ones using this networking interface.
• It’s not possible to reach a resource located outside of the Private Network using this interface.

Private Domain Names

In order to reference the containers in a Private Network, and to ease the interactions between them, they are each assigned a private domain name.

They are composed of 3 to 5 parts:

• the container number (optional),
• the container type (optional),
• the application ID,
• the Private Network ID,
• a common internal network identifier (private-network.internal.).

Here are some examples with an application ap-a71da13f-7c70-4c00-a644-eee8558d8053 and a Private Network pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c:

• ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal.
  DNS A records to the web containers. web containers are the default containers contacted when using the application domain name (optional prefixes excluded).

• web.ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal.
  DNS A records to the web containers.

• 1.web.ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal.
  DNS A record to the first web container.

• worker.ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal.
  DNS A records to the worker containers.

• 1.worker.ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal.
  DNS A record to the first worker container.

These domain names can be inferred by knowing the application ID, the Private Network ID and the application formation. The application and Private Network IDs are injected in the application environment at runtime.

You can also list them using the Scalingo CLI:

scalingo --app my-app private-networks-domain-names

Routing and Load-Balancing

When multiple containers are available for the same process type (see horizontal scaling for further details), requests must be routed to one of them. This routing process, called load-balancing, depends on the DNS client.

For example, a worker container can be targeted using the domain name worker.ap-a71da13f-7c70-4c00-a644-eee8558d8053.pn-ad0fd6a1-d05e-40ea-bf63-c4f8a75a9d8c.private-network.internal..

When resolving this domain name, the DNS client gets a set of A records (one for each worker container available). Multiple strategies can be considered to decide which record to use, from a basic round-robin to resource-based load-balancing. Picking a strategy highly depends on your use-case, needs, and on your code or software abilities. In any case, the decision is left to the discretion of the DNS client used by your code or framework.

Time To Live

DNS records in a Private Network have a Time To Live (TTL) of 30 seconds.

When restarting an app, or when scaling in, and if the DNS client used by your applications honors this TTL value by keeping the records in its cache, internal traffic might be sent to container(s) that no longer exist (the ones that have just been stopped). This may lead to errors or short unavailabilities.

Security

Network Isolation

The isolation of each Private Network is ensured by leveraging a technology called VXLAN. When a new Private Network is created, a unique (per region) VXLAN Network Identifier (VNI) is generated and assigned to the Private Network. This unique VNI ensures that network frames from one VXLAN cannot mix with those of another. By design, a VXLAN can’t “see” the other ones, it doesn’t even know others exist. Even though they travel across the same physical network, each VNI acts as an isolated fortress, hence protecting the resources on your Private Network from the neighbors.

Traffic Encryption

Thanks to WireGuard, all traffic transiting inside a Private Network is secured using state-of-the-art encryption algorithm, primitives and protocols.

In practice, each container is provisioned with a dedicated WireGuard interface for the Private Network. This interface uses a pair of asymmetric keys to encrypt and decrypt the traffic that flows through it.

WireGuard configuration is fully managed by Scalingo:

• The cryptographic keys are provisioned as soon as the container boots up.
• The public key is automatically distributed to other members of the Private Network.
• Each container is configured with the list of allowed WireGuard peers within the same Private Network, so traffic is only exchanged with authorized containers.

For all details about WireGuard’s technical internals, please see WireGuard’s technical whitepaper.

This encryption layer is complementary to VXLAN:

• VXLAN ensures network isolation and connectivity for the Private Network.
• WireGuard ensures confidentiality and integrity for the data exchanged inside that network.

Technical Considerations

When deploying a web application inside a Private Network, and reaching it using its private domain name, some Scalingo features are no longer available:

• There is no way to specify the address range or subnet mask to use in a Private Network.
• There is no way to modify the DNS records TTL of private domain names.
• The router logs no longer display an entry for each request. This is because these requests no longer go through our front routers. Hence we no longer have the ability to track these requests and log them.
• The traffic transiting internally between containers inside a Private Network is not monitored. This means that the router metrics (e.g. Requests per Minute or Response Time) are only available for containers that are publicly reachable (typically, web containers). As a consequence, Autoscalers based on router metrics can only be setup on publicly available containers.