> For the complete documentation index, see [llms.txt](https://newdocs.keeper.io/en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://newdocs.keeper.io/en/keeperpam/privileged-access-manager/getting-started/gateways/health-checks.md). # Health Checks ## Overview This document describes the health check functionality implemented for the KeeperPAM Gateway. Health checks provide essential monitoring capabilities that solve several common operational challenges. Health Checks provide the following benefits: * Allow load balancers to automatically remove unhealthy instances from rotation and add them back when they recover. * Integrate with monitoring systems (Prometheus, Nagios, Datadog, etc.) to provide automated alerting and dashboards showing gateway health across your infrastructure. * Enable automated monitoring scripts and orchestration tools to detect failures and trigger recovery procedures without human intervention. {% hint style="warning" %} On native installs (Windows, Linux), the health check service is **disabled by default**. You must activate it as documented in the next sections. {% endhint %} *** ## Basic Configuration The below configuration enables a basic health check service on the binary and Docker installation methods. More [advanced configuration](#advanced-healthcheck-configuration-examples) is also available as documented below. ### Activating the Health Check You can enable the health check service, add the following CLI property / environment variable on your gateway: {% tabs %} {% tab title="ENV" %} ```bash KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: 'true' ``` See [how to configure gateway environment variables](/en/keeperpam/privileged-access-manager/getting-started/gateways/gateway-environment-configuration.md#environment-variables). In a Docker Compose file, you would also need to configure the healthcheck: {% code overflow="wrap" %} ```yml keeper-gateway: ... environment: ... KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: 'true' healthcheck: test: - CMD - /usr/local/bin/keeper-gateway - health-check interval: 30s timeout: 10s retries: 3 start_period: 60s ... restart: unless-stopped ``` {% endcode %} {% endtab %} {% tab title="CLI" %} ```bash --health-check ``` See [how to configure gateway CLI arguments](/en/keeperpam/privileged-access-manager/getting-started/gateways/gateway-environment-configuration.md#cli-arguments). {% endtab %} {% endtabs %} *** ### Consulting the Health Check Consult your gateway's health check once it has been activated. #### CLI On native Windows and Linux installations, you can consult the health check with the gateway CLI: ```bash gateway health-check >> OK: Gateway is running and connected ``` If you get an error like "Could not connect to health check server", it means you haven't enabled the health check properly. If you see "Exception No such command 'keeper-gateway.exe'", you're using the wrong command syntax. Always use "gateway" as the command name. #### Docker On Docker installations, you can use Docker's `inspect` method to retrieve the health check. If the container name is `keeper-gateway`, a one-line bash command to find the service status can be found like this: ```bash docker inspect --format='{{.State.Health.Status}}' keeper-gateway >> healthy ``` If you don't know the container name, this script will give it to you: {% code overflow="wrap" %} ```bash docker ps --filter "status=running" --format "{{.Names}} {{.Image}}" | grep keeper-gateway | awk '{print $1}' ``` {% endcode %} Here's an example of checking the health with a bash command: ```bash $ docker inspect --format='{{.State.Health.Status}}' my-gateway-container >> healthy ``` The below complete bash script can be added to your watchdog services to check service status and automatically restart the container if it's unhealthy. Replace `/path/to/` with the proper path. {% code title="watchdog.sh" overflow="wrap" %} ```bash #!/bin/bash LOGFILE="/path/to/keeper-watchdog.log" # Find running container matching keeper-gateway CONTAINER_NAME=$(docker ps --filter "status=running" --format "{{.Names}} {{.Image}}" | grep keeper-gateway | awk '{print $1}') if [ -z "$CONTAINER_NAME" ]; then echo "$(date): No running keeper-gateway container found." >> "$LOGFILE" else # Get health status HEALTH=$(docker inspect --format='{{.State.Health.Status}}' "$CONTAINER_NAME") if [ "$HEALTH" != "healthy" ]; then echo "$(date): $CONTAINER_NAME is $HEALTH. Restarting..." >> "$LOGFILE" docker restart "$CONTAINER_NAME" else echo "$(date): $CONTAINER_NAME is healthy." >> "$LOGFILE" fi fi # Trim log file to last 100 lines tail -n 100 "$LOGFILE" > "$LOGFILE.tmp" && mv "$LOGFILE.tmp" "$LOGFILE" ``` {% endcode %} To schedule this health check on a Linux system, it can be added to the cron ``` crontab -e ``` Add to the crontab to watch every minute... ``` * * * * * /path/to/watchdog.sh ``` *** ## Advanced Configuration The below section provides detailed configuration for customization of the health checks in different environments. #### Activating the Health Check {% tabs %} {% tab title="ENV" %}
ConfigurationStart Command
Basic HTTPKEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
HTTP with AuthKEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mytoken
HTTPS (SSL)KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true
KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem
KEEPER_GATEWAY_SSL_KEY: /path/key.pem
HTTPS with AuthKEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true
KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem
KEEPER_GATEWAY_SSL_KEY: /path/key.pem
KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mytoken
Custom Port

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true

KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8443

Custom Host

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true

KEEPER_GATEWAY_HEALTH_CHECK_HOST: 0.0.0.0

Production Setup

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true
KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem
KEEPER_GATEWAY_SSL_KEY: /path/key.pem

KEEPER_GATEWAY_HEALTH_CHECK_HOST: 0.0.0.0

KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8443
KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: $(cat /etc/secrets/token)

{% endtab %} {% tab title="CLI" %}
ConfigurationStart Command
Basic HTTP

gateway start \

--health-check

HTTP with Auth

gateway start \

--health-check \

--health-check-auth-token mytoken

HTTPS (SSL)

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem

HTTPS with Auth

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem \

--health-check-auth-token mytoken

Custom Port

gateway start \

--health-check \

--health-check-port 8443

Custom Host

gateway start \

--health-check \

--health-check-host 0.0.0.0

Production Setup

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem \

--health-check-host 0.0.0.0 \

--health-check-port 8443 \

--health-check-auth-token $(cat /etc/secrets/token)

{% endtab %} {% endtabs %} You can consult our [Gateway Environment Variables](/en/keeperpam/privileged-access-manager/getting-started/gateways/gateway-environment-variables.md) documentation to find more about default definitiions for these properties. #### Consulting the Health Check
ConfigurationCLI Health CheckCurl Health Check
Basic HTTPgateway health-checkcurl https://127.0.0.1:8099/health
HTTP with Auth

gateway health-check \

--token mytoken

curl https://127.0.0.1:8099/health \

-H "Authorization: Bearer mytoken"

HTTPS (SSL)

gateway health-check \

--ssl

curl https://127.0.0.1:8099/health \
-k
HTTPS with Auth

gateway health-check \

--ssl \

--token mytoken

curl https://127.0.0.1:8099/health \
-k

-H "Authorization: Bearer mytoken"

Custom Port

gateway health-check \

--port 8443

curl http://127.0.0.1:8443/health
Custom Host

gateway health-check \

--host 0.0.0.0

curl http://0.0.0.0:8099/health
Production Setup

gateway health-check \

--host 0.0.0.0 \

--port 8443 \

--ssl \

--token $(cat /etc/secrets/token)

curl -k https://0.0.0.0:8443/health \

-H "Authorization: Bearer $(cat /etc/secrets/token)"

#### Output Format Examples
Output FormatCLI CommandDescription
Simple Status

gateway health-check \

--token mytoken

Returns:

OK: Gateway is running and connected

CRITICAL: ...

Detailed Info

gateway health-check \

--token mytoken \

--info

Returns:

Key=value pairs suitable for monitoring scripts

JSON Format

gateway health-check \

--token mytoken \

--json

Returns:

Full JSON response matching HTTP endpoint

*** ### HTTP Health Check You can use the health check features detailed above to enable a HTTP health check for your gateway. #### Usage When enabled, the HTTP health check endpoint will be available at: ``` http://localhost:8099/health ``` Or with SSL: ``` https://localhost:8099/health ``` #### Response Format The endpoint returns: * HTTP 200 if the gateway is healthy * HTTP 503 if the gateway is not healthy * JSON response with details **Sample Healthy Gateway** {% code expandable="true" %} ```json { "status": "healthy", "message": "Gateway is running and connected", "details": { "timestamp": 1784637354, "version": 2, "hostname": "localhost.localdomain", "instance_id": "IMJUOZ", "gateway_version": "1.8.6", "resources": { "memory": { "available_mb": 10200, "total_mb": 10939, "used_percent": 6.76, "headroom_percent": 93.24, "container_limit_mb": null }, "connections": { "total": 0, "by_type": {} }, "pressure": { "under_pressure": false, "can_accept_rbi": true } }, "rotations": { "active": 0, "pool_active": 0, "pool_limit": 1, "pool_load_percentage": 0, "can_accept": true, "suggested_delay_ms": 0, "total": 0, "failed": 0, "failure_rate_percent": null, "failure_rate_10min_percent": null, "failure_rate_10min_count": 0, "failure_rate_jobs_percent": null, "failure_rate_jobs_count": 0 }, "discoveries": { "active": 0, "pool_active": 0, "pool_limit": 1, "pool_load_percentage": 0, "can_accept": true, "suggested_delay_ms": 0, "total": 0, "failed": 0, "failure_rate_percent": null, "failure_rate_10min_percent": null, "failure_rate_10min_count": 0, "failure_rate_jobs_percent": null, "failure_rate_jobs_count": 0 }, "tubes": { "can_accept_more": true, "active_creates": 0, "max_concurrent": 100, "queue_depth": 0, "avg_create_time_ms": 0, "total_creates": 0, "total_failures": 0, "load_percentage": 0, "suggested_delay_ms": 0 }, "overall_capacity": { "can_accept_work": true, "suggested_delay_ms": 0, "bottleneck": "none" }, "connection_status": "connected", "websocket": { "uptime_seconds": 6014, "uptime_human": "1h 40m 14s", "last_ping_received_seconds_ago": 7, "latency_ms": 92, "last_ping_sent_timestamp": 1784637340, "last_pong_received_timestamp": 1784637340, "reconnect_count": 0 } } } ``` {% endcode %} **Sample Unhealthy Gateway:** {% code expandable="true" %} ```json { "status": "unhealthy", "message": "Gateway is not properly connected (status: disconnected)", "details": { "timestamp": 1784637802, "version": 2, "hostname": "localhost.localdomain", "instance_id": "HTLGDB", "gateway_version": "1.8.6", "resources": { "memory": { "available_mb": 10341, "total_mb": 10939, "used_percent": 5.47, "headroom_percent": 94.53, "container_limit_mb": null }, "connections": { "total": 0, "by_type": {} }, "pressure": { "under_pressure": false, "can_accept_rbi": true } }, "rotations": { "active": 0, "pool_active": 0, "pool_limit": 1, "pool_load_percentage": 0, "can_accept": true, "suggested_delay_ms": 0, "total": 0, "failed": 0, "failure_rate_percent": null, "failure_rate_10min_percent": null, "failure_rate_10min_count": 0, "failure_rate_jobs_percent": null, "failure_rate_jobs_count": 0 }, "discoveries": { "active": 0, "pool_active": 0, "pool_limit": 1, "pool_load_percentage": 0, "can_accept": true, "suggested_delay_ms": 0, "total": 0, "failed": 0, "failure_rate_percent": null, "failure_rate_10min_percent": null, "failure_rate_10min_count": 0, "failure_rate_jobs_percent": null, "failure_rate_jobs_count": 0 }, "tubes": { "can_accept_more": true, "active_creates": 0, "max_concurrent": 100, "queue_depth": 0, "avg_create_time_ms": 0, "total_creates": 0, "total_failures": 0, "load_percentage": 0, "suggested_delay_ms": 0 }, "overall_capacity": { "can_accept_work": true, "suggested_delay_ms": 0, "bottleneck": "none" }, "connection_status": "disconnected" } } ``` {% endcode %} Note that some metrics like `latency_ms`, `last_ping_sent_timestamp`, and `last_pong_received_timestamp` may not always be present in the response. The availability of these metrics depends on the current state of the WebSocket connection and the timing of ping/pong messages. #### Status Update Delays The health check reflects the current state of the WebSocket connection, but there may be a delay in status updates. **Delayed Status Updates** When connectivity is lost, it may take up to 2 minutes for the health check to report an "unhealthy" status, as the gateway attempts to reconnect. Similarly, when connectivity is restored, it may take up to 2 minutes for the health check to reflect a "healthy" status. This latency is intentional and allows the gateway to attempt reconnection without immediately reporting failures for transient connectivity issues. #### Security The HTTP health check includes the following security features: 1. **Authentication**: When `KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN` is set, requests must include the token in the Authorization header: ``` Authorization: Bearer ``` 2. **SSL/TLS**: When SSL is enabled, all communication is encrypted. You must provide a valid certificate and private key. 3. **Localhost binding**: The server binds to localhost only by default, not exposing the endpoint over the network. 4. **Security Headers**: The health check server adds the following security headers to responses: * X-Content-Type-Options: nosniff * X-Frame-Options: DENY * Content-Security-Policy: default-src 'none' 5. **Rate Limiting**: Automatic rate limiting is applied to non-localhost connections (60 requests per minute per IP). 6. **Information Protection**: When the server is bound to a non-localhost address, sensitive information is automatically redacted from the response. 7. **Forced SSL**: SSL is automatically enforced when binding to non-localhost interfaces. #### TLS Compatibility The health check server is configured to support a wide range of clients by: * Using secure TLS defaults (TLS 1.2+ minimum) for maximum security * Supporting modern cipher suites for strong encryption * Automatically handling protocol negotiation for HTTP and HTTPS For clients that support modern TLS versions, use standard curl commands: ``` curl -k -H "Authorization: Bearer your_token" https://localhost:8099/health ``` *** ### Docker-Specific Configuration Requirements When running Keeper Gateway inside Docker, special configuration may be required to expose the health check to the host or external systems: **Binding to 0.0.0.0** * The health check server must bind to `0.0.0.0` to be reachable outside the container. * `127.0.0.1` restricts access to within the container only. **SSL Enforcement** * When using `0.0.0.0`, Keeper Gateway forces SSL to protect health check data. * You must provide a valid certificate and key or the server will not start. **Authentication Requirement** * If binding to `0.0.0.0`, you must also specify an `AUTH_TOKEN` to secure the endpoint. **Docker Compose Example** ``` services: keeper-gateway: image: keeper/gateway:latest ports: - "8099:8099" volumes: - ./certs:/certs:ro environment: KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true KEEPER_GATEWAY_HEALTH_CHECK_HOST: "0.0.0.0" KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8099 KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /certs/healthcheck.crt KEEPER_GATEWAY_HEALTH_CHECK_SSL_KEY: /certs/healthcheck.key KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mysecrettoken ``` **Generate a Self-Signed Certificate** ``` mkdir -p certs openssl req -x509 -nodes -days 365 \ -newkey rsa:2048 \ -keyout certs/healthcheck.key \ -out certs/healthcheck.crt \ -subj "/CN=localhost" ``` **Test the Endpoint from the Host** ``` curl -k -H "Authorization: Bearer mysecrettoken" https://localhost:8099/health ``` *** ### Example Linux Configuration ``` # Enable HTTP health check export KEEPER_GATEWAY_HEALTH_CHECK_ENABLED=true export KEEPER_GATEWAY_HEALTH_CHECK_PORT=8099 export KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN=mysecrettoken # Start the gateway gateway start ``` Or using command line arguments: ``` gateway start --health-check --health-check-port 8099 --health-check-auth-token mysecrettoken ``` *** ### Self-Signed SSL Certificates For testing or internal use, you can generate self-signed certificates to enable SSL/TLS encryption: ``` # Generate a private key openssl genrsa -out healthcheck.key 2048 # Generate a certificate signing request (CSR) openssl req -new -key healthcheck.key -out healthcheck.csr -subj "/CN=localhost" # Generate a self-signed certificate (valid for 365 days) openssl x509 -req -days 365 -in healthcheck.csr -signkey healthcheck.key -out healthcheck.crt # Set the environment variables export KEEPER_GATEWAY_HEALTH_CHECK_ENABLED=true export KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL=true export KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT=/path/to/healthcheck.crt export KEEPER_GATEWAY_HEALTH_CHECK_SSL_KEY=/path/to/healthcheck.key export KEEPER_GATEWAY_HEALTH_CHECK_PORT=8443 # Typical HTTPS port export KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN=mysecrettoken # Start the gateway gateway start ``` Or using command line arguments: ``` gateway --health-check --health-check-port 8443 --health-check-ssl --health-check-ssl-cert /path/to/healthcheck.crt --health-check-ssl-key /path/to/healthcheck.key --health-check-auth-token mysecrettoken start ``` When using self-signed certificates, your HTTP client will need to be configured to trust the certificate or ignore SSL verification (not recommended for production). *** ### Monitoring Integration This endpoint can be used with monitoring systems like: * Prometheus with blackbox exporter * Nagios/Icinga * Zabbix * Datadog * AWS CloudWatch * Any monitoring system that can perform HTTP checks ## Troubleshooting ***
IssueTest CommandExpected Result
Check if server is runningcurl http://127.0.0.1:8099/healthConnection success or "Connection refused"
Test SSL connectivitycurl -k https://127.0.0.1:8099/healthSSL handshake success or SSL error
Test authenticationcurl -k -H "Authorization: Bearer wrongtoken" https://127.0.0.1:8099/health{"error": "Invalid authentication token"}
Check server bindingcurl http://0.0.0.0:8099/healthSuccess if bound to 0.0.0.0, failure if bound to 127.0.0.1
#### Error Messages and Troubleshooting The CLI health check provides detailed error messages to help diagnose issues: **Authentication Errors (HTTP 401)** ``` CRITICAL: Authentication failed when connecting to https://127.0.0.1:8099/health ERROR: Invalid or missing authentication token. Possible fixes: 1. Check if auth token is required: curl -k https://127.0.0.1:8099/health 2. Provide the correct auth token: gateway health-check --ssl --token YOUR_TOKEN 3. Check gateway startup logs for the configured token ``` **Connection Errors** ``` CRITICAL: Could not connect to health check server at http://127.0.0.1:8099/health ERROR: Connection failed. Possible causes: 1. Health check server is not running 2. Wrong host/port combination 3. Network connectivity issues 4. SSL/non-SSL mismatch Troubleshooting steps: 1. Verify gateway is running with health check enabled: gateway start --health-check 2. Check if server is using SSL: gateway health-check --ssl 3. Verify host and port: Current: 127.0.0.1:8099 4. Test with curl: curl http://127.0.0.1:8099/health ``` **SSL Certificate Errors** ``` CRITICAL: SSL error connecting to health check server at https://127.0.0.1:8099/health ERROR: SSL certificate validation failed. Possible causes: - Self-signed certificate (try curl with -k flag) - Invalid certificate path - Certificate expired ``` *** ## Specifications The Gateway health check is implemented using [Bottle](https://bottlepy.org/), a lightweight WSGI micro web-framework for Python. Bottle was chosen for the following advantages: * Minimal dependency (single file, \~60KB in size) * Enhanced security over the built-in Python HTTP server * Proper request routing and handling * Better error management * Thread safety * Production-ready with minimal overhead --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://newdocs.keeper.io/en/keeperpam/privileged-access-manager/getting-started/gateways/health-checks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.