DocumentationAPI ReferencePlaygroundCommunity

Backend Services Dev Environment Setup

Setup the Walrus and Default Services for Development

Suggest Edits

We are going to assume that your working directory is named ./working-dir. Please rename this path whenever you find it your working dir.

1. Clone the Repository

cd ./working-dir
git clone https://github.com/diffgram/diffgram.git

2. Create a Virtual Environment

Create the virtual environment and activate it.

python3 -m venv venv3 
source venv3/bin/activate

3. Install the requirements for both services.

At the time of writing, we have 2 services in Diffgram. The walrus service, and the default service. The walrus service is in charge of all the long running operations like media processing and the default service is the service for most of the backend API calls. We will install the dependencies for both below

pip install -r diffgram/default/requirements.txt 
pip install -r diffgram/walrus/requirements.txt

4. Set the Configuration file

In order for Diffgram to run correctly, we need to setup a set of values to the variables inside the settings.py file in the code. This can be done by either creating the enviornment variables, or, the recommended way for developers of diffgram, to use the secrets.py file. This is a file that is not tracked on the repo where you can programatically set all the values for your settings.py variables. You can use the below example as a starting point.

touch diffgram/shared/settings/secrets.py

Now open secrets.py with your favorite code editor and paste this sample.

# OPENCORE - REMOVE
import os
import base64

running_locally = True

os.environ['DIFFGRAM_SYSTEM_MODE'] = DIFFGRAM_SYSTEM_MODE = "testing_e2e"  # "sandbox, production, staging, testing, testing_e2e
os.environ['DIFFGRAM_STATIC_STORAGE_PROVIDER'] = 'azure'

if DIFFGRAM_SYSTEM_MODE in ["testing_e2e", "testing"]:
    db_username = 'postgres'
    db_pass = "postgres"
    connector = "postgresql+psycopg2"
    db_name = "diffgram_e2e_tests"
    if DIFFGRAM_SYSTEM_MODE == 'testing':
        db_name = 'diffgram_testing'
    os.environ['EMAIL_DOMAIN_NAME'] = "localhost"
    # db_username = 'postgres'
    # db_pass = "postgres"
    # connector = "postgresql+psycopg2"
    # db_name = "diffgram"
    db_host = "localhost"
    os.environ['DATABASE_URL'] = connector + "://" + db_username + ":" + db_pass + "@" + db_host + "/" + db_name
    os.environ['EMAIL_VALIDATION'] = 'False'

    os.environ['USER_PASSWORDS_SECRET'] = 'secret'
    os.environ['DB_SECRET'] = 'secret'
    os.environ['SECRET_KEY'] = 'secret'
    os.environ['DIFFGRAM_AWS_ACCESS_KEY_ID'] = ''
    os.environ['DIFFGRAM_AWS_ACCESS_KEY_SECRET'] = ''
    os.environ['DIFFGRAM_S3_BUCKET_NAME'] = ''
    os.environ['ML__DIFFGRAM_S3_BUCKET_NAME'] = ''
    os.environ['WALRUS_SERVICE_URL_BASE'] = "http://127.0.0.1:8085/"
    os.environ['DIFFGRAM_AZURE_CONNECTION_STRING'] = 'please set your connection string here'
    os.environ['DIFFGRAM_AZURE_CONTAINER_NAME'] = 'testdev'
    os.environ['ML__DIFFGRAM_AZURE_CONTAINER_NAME'] = 'testdev'
    os.environ['PROCESS_MEDIA_REMOTE_QUEUE_ON'] = 'True'
    os.environ['PROCESS_MEDIA_TRY_BLOCK_ON'] = 'False'
    os.environ['PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY'] = 'True'
    os.environ['SERVICE_ACCOUNT_FULL_PATH'] = ''
    os.environ['GOOGLE_PROJECT_NAME'] = ''
    os.environ['CLOUD_STORAGE_BUCKET'] = ''
    os.environ['ML__CLOUD_STORAGE_BUCKET'] = ''
    os.environ['DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE'] = 'True'

# Shared keys
os.environ["RABBITMQ_DEFAULT_USER"]="admin"
os.environ["RABBITMQ_DEFAULT_PASS"]="admin"
os.environ['RABBITMQ_HOST']="localhost"
os.environ["RABBITMQ_PORT"]="5672"
os.environ['MAILGUN_KEY'] = ""
os.environ['INTER_SERVICE_SECRET'] = "secretsecret2872872kjashdkasjhsdas******aw"
os.environ['DIFFGRAM_INSTALL_FINGERPRINT'] = 'dev_pablo'
os.environ['DIFFGRAM_VERSION_TAG'] = 'development'
os.environ['DIFFGRAM_HOST_OS'] = 'dev'
os.environ['PROCESS_MEDIA_NUM_FRAME_THREADS'] = '8'

We are assuming you have a local postgres installed on you machine with username postgres and password postrgres. You can modify secrets.py` according to your needs. This is just an initial sample to get started and may not contain all the needed configurations values for full funcionality of diffgram.x

5. Start The Default Service

cd ./working-dir
export PYTHONPATH=/path/to/working-dir/diffgram/ 
cd diffgram/default
python main.py

When you run the server you should see an error like this:

Traceback (most recent call last):
  File "main.py", line 36, in <module>
    do_routes_importing()
  File "/home/pablo/testdiffgram/diffgram/default/routes_init.py", line 2, in do_routes_importing
    from methods.attribute.attribute_template_group import new_attribute_template_group_factory_api
  File "/home/pablo/testdiffgram/diffgram/default/methods/attribute/attribute_template_group.py", line 1, in <module>
    from methods.regular.regular_api import *
  File "/home/pablo/testdiffgram/diffgram/default/methods/regular/regular_api.py", line 34, in <module>
    from shared.database.user import User
  File "/home/pablo/testdiffgram/diffgram/shared/database/user.py", line 2, in <module>
    from shared.database.common import *
  File "/home/pablo/testdiffgram/diffgram/shared/database/common.py", line 41, in <module>
    from shared import data_tools_core
  File "/home/pablo/testdiffgram/diffgram/shared/data_tools_core.py", line 31, in <module>
    data_tools = Data_tools().data_tools
  File "/home/pablo/testdiffgram/diffgram/shared/utils/singleton.py", line 7, in __call__
    cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
  File "/home/pablo/testdiffgram/diffgram/shared/data_tools_core.py", line 28, in __init__
    self.data_tools = DataToolsAzure()
  File "/home/pablo/testdiffgram/diffgram/shared/data_tools_core_azure.py", line 25, in __init__
    self.azure_service_client = BlobServiceClient.from_connection_string(settings.DIFFGRAM_AZURE_CONNECTION_STRING)
  File "/home/pablo/testdiffgram/venv3/lib/python3.7/site-packages/azure/storage/blob/_blob_service_client.py", line 170, in from_connection_string
    account_url, secondary, credential = parse_connection_str(conn_str, credential, 'blob')
  File "/home/pablo/testdiffgram/venv3/lib/python3.7/site-packages/azure/storage/blob/_shared/base_client.py", line 361, in parse_connection_str
    raise ValueError("Connection string is either blank or malformed.")

This is because we still require to set the DIFFGRAM_AZURE_CONNECTION_STRING to connect to an azure blob storage service. Please follow this link:

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/connection-strings/storage

To learn more about how to generate one.

Once you generate you connection string run the above commands again and you should see something like this:

[2021-11-02 09:01:41,552] INFO system_startup_checker.py Line 13: [default_service] Performing System Checks...
[2021-11-02 09:01:41,788] INFO system_events.py Line 94: Checking for version upgrades [default_service]
Startup in 1.3195436000823975
 * Serving Flask app "Diffgram" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
 * Restarting with stat
DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
PROCESS_MEDIA_TRY_BLOCK_ON False
PROCESS_MEDIA_REMOTE_QUEUE_ON True
PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY True
DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  testing_e2e
DATABASE_URL DATABASE_URL  True
[2021-11-02 09:01:42,770] INFO system_startup_checker.py Line 13: [default_service] Performing System Checks...
[2021-11-02 09:01:43,004] INFO system_events.py Line 94: Checking for version upgrades [default_service]

You have succesfully started the default service.

Other Storage Providers

You can also use aws or gcp as your static storage provider. Just change the DIFFGRAM_STATIC_STORAGE_PROVIDER to either aws or gcp and fill the following variables:

For aws:

  • DIFFGRAM_AWS_ACCESS_KEY_ID
  • DIFFGRAM_AWS_ACCESS_KEY_SECRET
  • DIFFGRAM_S3_BUCKET_REGION
  • DIFFGRAM_S3_BUCKET_NAME

For Gcp:

  • GOOGLE_PROJECT_NAME: The google project name
  • CLOUD_STORAGE_BUCKET: The bucket name on google storage.
  • SERVICE_ACCOUNT_FULL_PATH: The local path to your service account json file.

6. Start the Walrus Service

In A similar way you can now start the walrus service

cd ./working-dir
export PYTHONPATH=/path/to/working-dir/diffgram/ 
cd diffgram/walrus
python main.py

Updated 9 months ago