Update Production
Helm Production Updates
Make sure you have the latest helm chart version by doing
git clone https://github.com/diffgram/diffgram-helm
Then change the diffgramVersion
on the values.yaml
file to the new version released. And redeploy chart with updated values.
Breaking Changes
Breaking changes tracking.
DB Upgrades / Deadlock Prevention
We use Alembic to manage upgrades of the database schema. Some of these upgrade may include new indexes and constraints. We strongly suggest that you reduce the database access to a minimum to prevent deadlocks on an index creation as it can cause unexpected CPU spikes and degrade the entire system's performance.
Helm Timeouts
When using helm upgrade
you may need to add the --timeout
flag if the index or migration to add will require significant time due to the large amount of data to index. You can upgrade with the timeout flag in the following manner:
helm upgrade diffgram . --timeout 5000
Index Upgrades
Use a HA (High Availability) database system for uptime during maintenance, or run during a low usage period or schedule a dedicated maintenance window to prevent deadlocks when upgrading DB schema.
Production Overview
This is usually automated, e.g. through a github action
- Initialize tracking on your preferred deployment tracking pipeline
- Migrate the database using the provided almebic migrations
- Pull the latest images
- Run tests
- Migrate traffic
Updating the SDK
pip install diffgram --upgrade
Warnings
Use Extreme Caution When Updating Kubernetes Cluster Version
Updated about 2 years ago