Overview

The most popular way to get started is to use docker. This page is focused on getting a basic installation up. For production we recommend K8s, see Production.

Getting Ready for Quickstart

  1. Install Docker and docker-compose if not already installed.
  2. Setup your Storage service.

At the moment there are 4 options Storage Providers List.
Tangent: If you would like a default local storage with no config please upvote this ticket.

Quickstart Development Install

The installer will test your storage connection so we strongly recommend setting up your storage prior to starting the installer.

Call python install.py and follow the prompts! It handles configuring your environment file, and calls the needed docker commands.

In summary you can copy and paste this and follow the prompts

git clone https://github.com/diffgram/diffgram.git
cd diffgram
pip install -r requirements.txt
python install.py

View the Install

Go to http://localhost:8085

View the docker UI to see service status.

Screenshot Examples

Install.py example

Example of Docker up and running in Windows

Example of up and and running in standard terminal

Mental Prep

Depending on multiple factors your install time will vary. In the absolute best case if you have your storage and docker already setup, and there are no surprises, it is fast. If you need to setup new hardware, new storage, credentials etc. it will take more time.

Diffgram is a system designed for scale. This means there is a little bit more configuration. It is quite normal for the install and initial setup to be the most difficult part.

A few things that take time:

  • Storage configuration and permissions
  • Setting up hardware resources (e.g. if running on cloud)
  • Admin account and project setup
  • Setting up stuff within Diffgram itself (e.g. connections, schema etc)
  • Ecosystem, like SDK etc.

We are always looking to improve the install process and welcome ideas!

Debugging and Testing Docker Installs

Stopped Containers and startup checks

Diffgram has a built in system startup checker and self tests. During development we run tests on the software itself. This is different. These startup checks are designed to help verify that for your specific configuration everything is inline. Sometimes if a startup check fails it will prevent a container from starting. Check for stopped containers.

Env vars

Docker may not refresh env variables. Please be aware of this when testing, for example you may change the .env file but not see any change. You may need to recreate containers or follow the most current best practice for updating env variables.

Debugging All Cases

  • Does it work on diffgram.com?
  • Does it work locally? (e.g. if running on cloud)

If you run into an issue to best help solve it please contain as much info as possible, e.g.

  • Complete error logs, sometimes a screenshot can show more context then copying the specific log
  • Env file
  • Diffgram version
  • Information from SuperAdmin/Install Info page
  • Screenshots/Video
  • Hardware context, where is it running?

Config

Please note that we provide the various config mechanisms e.g. docker-compose, helm chart, etc. with the intent that in most cases that config will work well out of the box.

If there are any changes required, then it is quite possible it will "break" the config. While we can do our best to help please keep that mind. For example if you use custom ingress rules, custom TLS, use some parts on docker-compose and some not etc. We welcome ideas to build into the default config templates.

If getting help please be sure to mention anything that is non-standard, and if possible provide the complete customized new config.

Self-Guided Enterprise Tour

Installation and initial setup of any new technology, including Diffgram, is sometimes the most challenging part. Depending on your needs please expect to run into at least a handful of issues. Usually we can turn around fixes to these type of config / expectations issues same-day or next-day please reach out on slack when you encounter them.

Install Video

Note: After video was filmed we added the pip install -r requirements.txt step. This is required to validate the connection during the installation process. See above for complete code.

Warnings

🚧

The docker compose dev setup is not for Production.

Because the Database may be deleted if the image is deleted! And it does not include a production server like Gunicorn. See Kubernetes setup for Production installation.

Known Install Issues

You may need to restart services.

Docker Compose Known Issues

Windows Specific

📘

Requires Elevated Permissions

Use Run as Administrator. This is becuase of aws boto3 not diffgram itself.

Production Install

Open Installation - Production

📘

Production Data Protection

Please give care during installation to consider hardware resources, configuration, and read the production installation guide above before starting. Misconfigured setups can lead to data loss.

Database

After the installation you will see a folder called postgres-data was created. This is where all the SQL data from the postgres database will be. Make sure to avoid deleting this folder, unless you want to completely delete the data inside diffgram.

Manual Install with Docker Compose.

Use our pre-written docker-compose file and paste it in your environment.

🚧

Additional keys may be required

See the settings.py for a complete list of env vars.

Also, make sure you have the .env file with the appropriate values for each of the environment variables. Here is the example file with a description of each one:

GCP_SERVICE_ACCOUNT_FILE_PATH=/path/to/your/gcp/service/account.json or /dev/null if not used
CLOUD_STORAGE_BUCKET=<GCP STORAGE BUCKET NAME| Only use if GCP is your selected storage provider>
ML__CLOUD_STORAGE_BUCKET=<GCP STORAGE BUCKET NAME FOR ML TASKS| Only use if GCP is your selected storage provider>
SAME_HOST=False <If running in containarized enviroment leave False>
DIFFGRAM_STATIC_STORAGE_PROVIDER=< one of 'gcp, 'aws', 'azure'>
USER_PASSWORDS_SECRET=<any secure secret for hashing >
SECRET_KEY=<any secure secret for hashing >
DIFFGRAM_AWS_ACCESS_KEY_ID=<aws access key ID| only relevant if using aws>
DIFFGRAM_AWS_ACCESS_KEY_SECRET=<aws access key secret| only relevant if using aws>
DIFFGRAM_S3_BUCKET_NAME=<aws s3 bucket name| only relevant if using aws>
ML__DIFFGRAM_S3_BUCKET_NAME=<aws s3 ml bucket name| only relevant if using aws>
DIFFGRAM_AZURE_CONNECTION_STRING=<azure connection string| only relevant if using azure>
DIFFGRAM_AZURE_CONTAINER_NAME=<azure container name| only relevant if using azure>
ML__DIFFGRAM_AZURE_CONTAINER_NAME=<azure container name| only relevant if using azure>
EMAIL_VALIDATION=False
WALRUS_SERVICE_URL_BASE=http://walrus:8080/

Once you have all the enviroment files setup correctly. Run:

docker-compose up
# Or for headless mode
docker-compose up -d

You should see al the containers spin up and be able to access the Web UI at http://localhost:8085

After the installation you will see a folder called postgres-data was created. This is where all the SQL data from the postgres database will be. Make sure to avoid deleting this folder, unless you want to completely delete the data inside diffgram.

Backend storage vs Connections

The "backend" storage that's set at installation time is different from "connections". The backend storage is where the system stores and read BLOB data. It is assumed to be singular and static. (Excluding ML bucket.). Where as the Connections is assumed to be many and easily editable.

In the case where you already have data in a designated bucket, then it should be used as a connection, and a new bucket created as the backend.

Contributor Install

See New Engineer Welcome

Bare metal installation

You can also spin up each of the services and use your own dispatcher.

Need help?

Join our Community.


Did this page help you?