We live in a world where applications are used to do everything, be it stock trading or booking a salon, but behind the scenes, the connectivity is done using secrets. Secrets such as database passwords, API keys, tokens, etc., must be managed appropriately to avoid any breach.
The need for managing secrets is critical for any organization. Secrets can be leaked in many ways, including through version control systems (never hardcode any secrets in your code!), private messages, email and other communication channels. If secrets are leaked, it can lead to a loss of trust, credibility, and even business (you can learn why here). In some cases, leaked secrets can also lead to legal action. That's why it's so important to have a plan for managing secrets.
In this post, we will discuss 4 different ways to manage secrets in Python efficiently.
Pre-requisites
Before getting started, below are a few things to keep in mind to avoid any issues later.
Python & pip installed on your machine
Understanding of Python & CLI
Python IDE such as PyCharm or VS Code
Basic understanding of Cloud
💡 We will be using MacOS for this device. Please use the commands as per your OS.
4 Ways to Manage Secrets in Python
In this section, we will discuss 4 different ways to manage your secrets in Python.
1. From a file
Using an .env file
The .env file is a file used to store environment variables in Python. Environment variables are variables set outside of the Python code and are used to configure the Python code. The .env file is typically used to store secret keys and passwords.
We will use the python-dotenv package for accessing the content of the .env file. To get started, first install the package using the following command.
$ pip install python-dotenv
Create a .env file for testing purposes and paste the following secrets:
API_KEY=test-key
API_SECRET=test-secret
Of course, this file should not be committed to your git repo! Otherwise, it would be versioned and readable even after you delete it.
Add this line to your .gitignore file:
.env
Already committed the file? No worries, follow the instructions here.
Exposing secrets in a git repository is bad but mistakes happen. This is a complete guide and cheatsheet to rewrite and remove files from git history
Once done, create a main.py file and paste the below-mentioned code snippet. In this code, we are using the load_dotenv() function to load content from the .env file.
from dotenv import load_dotenv
import os
load_dotenv()
api_key = os.getenv("API_KEY")
api_secret = os.getenv("API_SECRET")
print("API_KEY: ", api_key)
print("API_SECRET: ", api_secret)
We can also use dotenv_values() function, which converts the secrets into a dictionary. Secrets can be accessed by using the following code snippet:
from dotenv import dotenv_values
secrets = dotenv_values(".env")
def main():
print(secrets["API_KEY"])
print(secrets["API_SECRET"])
if __name__ == "__main__":
main()
When working on a large project, you may find that you need multiple .env files. For example, you may have a .env file for your local development environment and a .env.dev file for your cloud development production environment. The following code snippet can be helpful if you have multiple env files.
from dotenv import dotenv_values
secrets = dotenv_values(".env")
local_secrets = dotenv_values(".env.dev")
def main():
print(secrets["API_KEY"])
print(local_secrets["API_SECRET"])
if __name__ == "__main__":
main()
Using a JSON file
To keep the secrets more organized, you can also use a JSON file. Let's create a secrets.json file and paste the following secrets into it.
{
"db": {
"host": "localhost",
"port": 27017,
"name": "test"
},
"server": {
"host": "localhost",
"port": 3000
}
}
💡 The same as above applies, do not commit this file!
Now that we have the JSON file ready, let’s write a function to access secrets from the file:
# main.py
import json
def get_value_from_json(json_file, key, sub_key):
try:
with open(json_file) as f:
data = json.load(f)
return data[key][sub_key]
except Exception as e:
print("Error: ", e)
print(get_value_from_json("secrets.json", "db", "host")) # prints localhost
2. Using Environment Variables
Environment variables are variables that are set by the operating system or a specific user and are used by programs to determine various settings.
We can use these variables to store our secrets and then access them in our program. You can use the following syntax to create an environment variable in macOS or a Linux machine.
$ export variable_name=value
$ export API_KEY_TEST=dummykey
On a Windows machine, you use GUI to add environment variables or use the following command to add a variable.
$ setx [variable_name] “[value]”
You can use the os package to access the os environment variable. Below mentioned is the sample code:
import os
# Get the secret key from the environment
secret_key = os.environ.get('api_key_test')
print(secret_key) // prints dummykey
Secrets at the command line should be handled with special care too! Have a look at the best practices.
Secrets at the Command Line [cheat sheet included] Developers need to prevent credentials from being exposed while working on the command line. Learn how you might be at risk and what tools and methods to help you work more safely.
3. Use a Cloud Secrets Manager
Most cloud service providers offer an inbuilt secrets manager that can be used to create and use secrets across cloud infrastructure. Following the secret managers offered by cloud providers:
Secret Manager (By Google Cloud)
AWS Secrets Manager is widely used across the industry. Let’s write a function to create and access a secret in AWS using Boto3.
import boto3
def fetch_secret_from_aws(secret_name):
try:
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name='us-east-1')
get_secret_value_response = client.get_secret_value(SecretId=secret_name)
return get_secret_value_response['SecretString']
except Exception as e:
print(e)
return None
def create_secret_in_aws(secret_name, secret_value):
try:
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name='us-east-1')
client.create_secret(Name=secret_name, SecretString=secret_value)
return True
except Exception as e:
print(e)
return False
4. Using a KMS
A KMS is a key management system that is used to manage cryptographic keys. It is typically used in organizations in order to centrally manage and secure keys. A KMS can be used to generate, store, and distribute keys. It can also be used to revoke keys and to monitor key usage.
KMS is a convenient way to centrally manage the keys used by your applications and services and helps to ensure that only authorized users have access to them.
Hashicorp Vault is one of the best open-source KMS available in the market that offers a number of features and benefits, including the ability to manage secrets and keys across multiple environments, strong security controls, and good scalability.
Let’s write a function to read and write secrets to a specific path in the vault.
💡 Note: Please ensure you have hvac (Python client for Vault) installed and have a Hashicorp Vault setup.
import hvac
def read_secret_from_vault(secret_path, token, secret_name):
try:
client = hvac.Client(
url='http://localhost:8200',
token=token,
)
read_response = client.secrets.kv.read_secret_version(path=secret_path)
return read_response['data']['data'][secret_name]
except Exception as e:
print(e)
return None
def write_secret_to_vault(secret_path, token, secret_name, secret_value):
try:
client = hvac.Client(
url='http://localhost:8200',
token=token,
)
create_response = client.secrets.kv.v2.create_or_update_secret(
path=secret_path,
secret={secret_name: secret_value},
)
return create_response
except Exception as e:
print(e)
return None
Wrapping it up
Managing secrets is an essential part of application development. When developers hardcode secrets in cleartext into their applications, it creates a potential security breach. An attacker can use those secrets to access sensitive data if those secrets are discovered.
Another alternative to the proposed methods here is to check secrets into source code and share them with your team encrypted. This can be a very flexible solution if you learn how to leverage a tool such as Mozilla SOPS. To learn more about SOPS and get started using it, read this tutorial.
Did you know that GitGuardian detects more than 16000 secrets hardcoded into GitHub commits daily? When a secret is detected, GitGuardian notifies the repository owner so they can take action to protect their data.
If you want a better view of how many secrets could be hidden inside your repositories, sign up (for free) or book a demo to the GitGuardian Platform to scan every single commit and take preventive action now!