-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[CONTP-1179] docs(agent_secrets): Add autodiscovery to secrets management docs #33624
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Mathew-Estafanous
wants to merge
3
commits into
master
Choose a base branch
from
mathew.estafanous/secrets-autodiscovery-refresh
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+71
−38
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -31,7 +31,7 @@ | |||||
|
|
||||||
| **Note**: This option is not available for FIPS-enabled Agents at this time. | ||||||
|
|
||||||
| Starting in Agent version `7.70`, the Datadog Agent natively supports several secret management solutions. Two new settings have been introduced to `datadog.yaml`: `secret_backend_type` and `secret_backend_config`. | ||||||
| Starting in Agent version `7.70`, the Datadog Agent natively supports several secret management solutions. Two new settings have been introduced to `datadog.yaml`: `secret_backend_type` and `secret_backend_config`. | ||||||
|
|
||||||
| `secret_backend_type` is used to specify which secret management solution to use, and `secret_backend_config` holds additional configuration relevant to that solution. | ||||||
|
|
||||||
|
|
@@ -44,7 +44,7 @@ | |||||
| ``` | ||||||
| More specific setup instructions depend on the backend type used. Refer to the appropriate section bellow for further information: | ||||||
| More specific setup instructions depend on the backend type used. Refer to the appropriate section bellow for further information: | ||||||
| {{% collapse-content title="AWS Secrets" level="h4" expanded=false id="id-for-secrets" %}} | ||||||
|
|
@@ -80,7 +80,7 @@ | |||||
| DD_SECRET_BACKEND_CONFIG='{"aws_session":{"aws_region":"<AWS_REGION>"}}' | ||||||
| ``` | ||||||
|
|
||||||
| After configuring the Agent to use AWS Secrets, you can reference any secrets in your configurations with `ENC[secretId;secretKey]`. | ||||||
| After configuring the Agent to use AWS Secrets, you can reference any secrets in your configurations with `ENC[secretId;secretKey]`. | ||||||
|
|
||||||
| The ENC notation is composed of: | ||||||
| * `secretId`: either the secret "friendly name" (for example, `/DatadogAgent/Production`) or the ARN (for example, `arn:aws:secretsmanager:us-east-1:123456789012:secret:/DatadogAgent/Production-FOga1K`). | ||||||
|
|
@@ -117,7 +117,7 @@ | |||||
| [1000]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html | ||||||
| [1001]: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html | ||||||
| [1002]: https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html | ||||||
| [1003]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html | ||||||
| [1003]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html | ||||||
| [1004]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html | ||||||
| [1005]: https://docs.aws.amazon.com/managedservices/latest/userguide/defaults-instance-profile.html | ||||||
| [1006]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html | ||||||
|
|
@@ -130,16 +130,16 @@ | |||||
|
|
||||||
| ##### Integration check | ||||||
|
|
||||||
| ```sh | ||||||
| datadog: | ||||||
| ```sh | ||||||
| datadog: | ||||||
| confd: | ||||||
| # This is an example | ||||||
| <INTEGRATION_NAME>.yaml: |- | ||||||
| ad_identifiers: | ||||||
| - <SHORT_IMAGE> | ||||||
| instances: | ||||||
| - [...] | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| env: | ||||||
| - name: DD_SECRET_BACKEND_TYPE | ||||||
| value: "aws.secrets" | ||||||
|
|
@@ -159,7 +159,7 @@ | |||||
|
|
||||||
| ##### Cluster check: without cluster check runners enabled | ||||||
| ```sh | ||||||
| datadog: | ||||||
| datadog: | ||||||
| env: | ||||||
| - name: DD_SECRET_BACKEND_TYPE | ||||||
| value: "aws.secrets" | ||||||
|
|
@@ -170,19 +170,19 @@ | |||||
| # IAM role ARN required to grant the Agent permissions to access the AWS secret | ||||||
| serviceAccountAnnotations: | ||||||
| eks.amazonaws.com/role-arn: <IAM_ROLE_ARN> | ||||||
| clusterAgent: | ||||||
| clusterAgent: | ||||||
| confd: | ||||||
| # This is an example | ||||||
| <INTEGRATION_NAME>.yaml: |- | ||||||
| cluster_check: true | ||||||
| instances: | ||||||
| - [...] | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| ``` | ||||||
|
|
||||||
| ##### Cluster check: with cluster check runners enabled | ||||||
| ```sh | ||||||
| datadog: | ||||||
| datadog: | ||||||
| env: | ||||||
| - name: DD_SECRET_BACKEND_TYPE | ||||||
| value: "aws.secrets" | ||||||
|
|
@@ -195,8 +195,8 @@ | |||||
| cluster_check: true | ||||||
| instances: | ||||||
| - [...] | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| clusterChecksRunner: | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| clusterChecksRunner: | ||||||
| enabled: true | ||||||
| env: | ||||||
| - name: DD_SECRET_BACKEND_TYPE | ||||||
|
|
@@ -272,15 +272,15 @@ | |||||
| # IAM role ARN required to grant the Agent permissions to access the AWS secret | ||||||
| serviceAccountAnnotations: | ||||||
| eks.amazonaws.com/role-arn: <IAM_ROLE_ARN> | ||||||
| clusterAgent: | ||||||
| clusterAgent: | ||||||
| extraConfd: | ||||||
| configDataMap: | ||||||
| # This is an example | ||||||
| <INTEGRATION_NAME>.yaml: |- | ||||||
| cluster_check: true | ||||||
| instances: | ||||||
| - [...] | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| ``` | ||||||
|
|
||||||
| <br> | ||||||
|
|
@@ -309,23 +309,23 @@ | |||||
| # IAM role ARN required to grant the Agent permissions to access the AWS secret | ||||||
| serviceAccountAnnotations: | ||||||
| eks.amazonaws.com/role-arn: <IAM_ROLE_ARN> | ||||||
| clusterAgent: | ||||||
| clusterAgent: | ||||||
| extraConfd: | ||||||
| configDataMap: | ||||||
| # This is an example | ||||||
| <INTEGRATION_NAME>.yaml: |- | ||||||
| cluster_check: true | ||||||
| instances: | ||||||
| - [...] | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| password: "ENC[secretId;secretKey]" | ||||||
| ``` | ||||||
|
|
||||||
| {{% /tab %}} | ||||||
| {{< /tabs >}} | ||||||
|
|
||||||
|
|
||||||
| {{% /collapse-content %}} | ||||||
| {{% /collapse-content %}} | ||||||
|
|
||||||
| {{% collapse-content title="AWS SSM" level="h4" expanded=false id="id-for-ssm" %}} | ||||||
| The following AWS services are supported: | ||||||
|
|
@@ -367,12 +367,12 @@ | |||||
| [1000]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html | ||||||
| [1001]: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html | ||||||
| [1002]: https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html | ||||||
| [1003]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html | ||||||
| [1003]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html | ||||||
| [1004]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html | ||||||
| [1005]: https://docs.aws.amazon.com/managedservices/latest/userguide/defaults-instance-profile.html | ||||||
| [1006]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html | ||||||
|
|
||||||
| {{% /collapse-content %}} | ||||||
| {{% /collapse-content %}} | ||||||
|
|
||||||
|
|
||||||
| {{% collapse-content title="Azure Keyvault Backend" level="h4" expanded=false id="id-for-azure" %}} | ||||||
|
|
@@ -413,7 +413,7 @@ | |||||
|
|
||||||
| [2000]: https://docs.microsoft.com/en-us/Azure/key-vault/secrets/quick-create-portal | ||||||
|
|
||||||
| {{% /collapse-content %}} | ||||||
| {{% /collapse-content %}} | ||||||
|
|
||||||
|
|
||||||
| {{% collapse-content title="HashiCorp Vault Backend" level="h4" expanded=false id="id-for-hashicorp" %}} | ||||||
|
|
@@ -425,7 +425,7 @@ | |||||
| | `hashicorp.vault` | [HashiCorp Vault (Secrets Engine Versions 1 and 2)][3000] | | ||||||
|
|
||||||
| ##### How to set up HashiCorp Vault | ||||||
| 1. Run your HashiCorp Vault. See the [official HashiCorp Vault documentation][3001] for more information. | ||||||
| 1. Run your HashiCorp Vault. See the [official HashiCorp Vault documentation][3001] for more information. | ||||||
| 2. Write a policy that gives the permission to pull secrets from your vault. Create a `*.hcl` file, and include the following permission if using Secrets Engine Version 1: | ||||||
| ``` | ||||||
| path "<your mount path>/<additional subpath>" { | ||||||
|
|
@@ -448,7 +448,7 @@ | |||||
| ``` | ||||||
| 3. Run `vault policy write <policy_name> <path_to_*.hcl_file>` | ||||||
|
|
||||||
| 4. Choose the method of authenticating to your vault. If using the AWS instance profile method, run `vault auth enable aws`. | ||||||
| 4. Choose the method of authenticating to your vault. If using the AWS instance profile method, run `vault auth enable aws`. | ||||||
|
|
||||||
| ##### AWS instance profile instructions | ||||||
|
|
||||||
|
|
@@ -468,7 +468,7 @@ | |||||
|
|
||||||
| ```yaml | ||||||
| # datadog.yaml | ||||||
| api_key: "ENC[/Datadog/Production;apikey]" | ||||||
| api_key: "ENC[/Datadog/Production;apikey]" | ||||||
| secret_backend_type: hashicorp.vault | ||||||
| secret_backend_config: | ||||||
|
|
@@ -484,7 +484,7 @@ | |||||
| [3002]: https://developer.hashicorp.com/vault/docs/auth/aws#aws-auth-method | ||||||
| [3003]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html | ||||||
| [3004]: https://developer.hashicorp.com/vault/docs/auth/aws#iam-authentication-inferences | ||||||
| {{% /collapse-content %}} | ||||||
| {{% /collapse-content %}} | ||||||
|
|
||||||
| {{% collapse-content title="JSON or YAML File Secret Backends" level="h4" expanded=false id="id-for-json-yaml" %}} | ||||||
|
|
||||||
|
|
@@ -556,7 +556,7 @@ | |||||
| [4001]: https://en.wikipedia.org/wiki/JSON | ||||||
| [4002]: https://en.wikipedia.org/wiki/YAML | ||||||
|
|
||||||
| {{% /collapse-content %}} | ||||||
| {{% /collapse-content %}} | ||||||
|
|
||||||
|
|
||||||
| ### Option 2: Using the built-in Script for Kubernetes and Docker | ||||||
|
|
@@ -833,35 +833,68 @@ | |||||
|
|
||||||
| **Note**: Your executable shares the same environment variables as the Agent. | ||||||
|
|
||||||
| ## Refreshing API/APP keys at runtime | ||||||
| ## Refreshing secrets at runtime | ||||||
|
|
||||||
| Starting in Agent v7.67, configure the Agent to refresh resolved secrets without requiring a restart. | ||||||
|
|
||||||
| Set a refresh interval: | ||||||
| ```yaml | ||||||
| secret_refresh_interval: 3600 # refresh every hour | ||||||
| ``` | ||||||
|
|
||||||
| Or, trigger a refresh manually: | ||||||
| ```shell | ||||||
| datadog-agent secret refresh | ||||||
| ``` | ||||||
|
|
||||||
| Starting in Agent version v7.67, you can configure the Agent to refresh its API and APP keys at regular intervals without requiring a restart. This relies on the API key and APP key being pulled as secrets. | ||||||
| ### API/APP key refresh | ||||||
| API/APP keys pulled as secrets support runtime refresh. | ||||||
|
|
||||||
| To enable this, set `secret_refresh_interval` (in seconds) in your `datadog.yaml` file: | ||||||
| You can enable this by setting `secret_refresh_interval` (in seconds) in `datadog.yaml`: | ||||||
| ```yaml | ||||||
| api_key: ENC[<secret_handle>] | ||||||
| secret_refresh_interval: 3600 # refresh every hour | ||||||
| ``` | ||||||
|
|
||||||
| By default the Agent randomly spreads its first refresh within the specified `secret_refresh_interval` window. This | ||||||
| means that it resolves the API key at startup, then refreshes it within the first interval and every interval after that. | ||||||
| This avoids having a fleet of Agents refreshing their API/APP key at the same time. | ||||||
| By default, the Agent randomizes the initial refresh within the `secret_refresh_interval` window to prevent a fleet of | ||||||
| Agents from refreshing simultaneously. The key is resolved at startup, then refreshed once within the first interval | ||||||
| and every interval thereafter. | ||||||
|
|
||||||
| To prevent downtime, only invalidate the previous API key and APP key when your entire fleet of Agents has | ||||||
| pulled the updated keys from your secret management solution. You can track usage of your API keys in the [Fleet | ||||||
| Management](https://app.datadoghq.com/fleet) page. | ||||||
| To prevent downtime, invalidate old keys only after your entire fleet has pulled the updated keys. You can track key | ||||||
| usage on the [Fleet Management](https://app.datadoghq.com/fleet) page. | ||||||
|
|
||||||
| You can disable this behavior by setting: | ||||||
| ```yaml | ||||||
| secret_refresh_scatter: false | ||||||
| ``` | ||||||
|
|
||||||
| To refresh manually, use: | ||||||
| ``` | ||||||
| datadog-agent secret refresh | ||||||
| ### Autodiscovery check secrets refresh | ||||||
| Starting in Agent v7.76, scheduled [Autodiscovery][1] checks can refresh secrets at runtime if the template uses the `ENC[]` syntax. | ||||||
|
|
||||||
| ```yaml | ||||||
| labels: | ||||||
| tags.datadoghq.com/redis.env: "prod" | ||||||
| tags.datadoghq.com/redis.service: "my-redis" | ||||||
| tags.datadoghq.com/redis.version: "6.0.3" | ||||||
| annotations: | ||||||
| ad.datadoghq.com/redis.checks: | | ||||||
| { | ||||||
| "redisdb": { | ||||||
| "init_config": {}, | ||||||
| "instances": [ | ||||||
| { | ||||||
| "host": "%%host%%", | ||||||
| "port":"6379", | ||||||
| "password":"ENC[<secret_handle>]" | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| Now the Agent can trigger secrets refresh at either the interval set in `secret_refresh_interval` or manually with `datadog-agent secret refresh`. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Does this keep the meaning? |
||||||
|
|
||||||
| ### Enabling DDOT collector refresh | ||||||
| If you are using [DDOT collector][6] and want to enable API/APP refresh you must add the following additional configuration to your `datadog.yaml` file: | ||||||
| ``` | ||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.