DocumentationAPI ReferencePlaygroundCommunity

Makefile Automation Option

Optional way to setup virtual envs and run Diffgram services locally

Suggest Edits

📘

Beta

The Makefile is in beta and may not work for your case. At the moment we recommend following the guide in new engineer welcome, but you can use the make file for reference.

Introduction

As part of the overall New Engineer Welcome you setup the OS, virtual envs and dependences and then run the services "Running the Services" section. The beta makefile is one option to make certain aspects of this easier and more automatic.

Make File Considerations

  • OS Setup not included
  • .env file setup not included
  • Assumptions on OS being used
  • Assumptions on how services are being run, e.g. what is in docker or not.
  • Assumptions on starting (optional) Minio
  • Other risks related to limitations of this beta automation

📘

Contributions welcome

We welcome contributions on improving Makefile and Makefile docs

Services recap

A full development environment consists of 7 services running:

  1. Default Service
  2. Walrus Service
  3. Eventhandlers
  4. Frontend Service
  5. RabbitMQ
  6. MinIO / Cloud Storage Account
  7. Postgres

In this guide you will learn how to set up each service individually.

Pre-Requisites

  1. Can run Makefiles on your local machine. This will help automate most of the processes. Otherwise you might need to inspect the makefile and run the command manually.
  2. Have python3 installed on your local machine. Version Used is : 3.11.x
  3. Have docker compose installed https://docs.docker.com/compose/install/
  1. Have yarn https://classic.yarnpkg.com/lang/en/docs/install/#debian-stable installed and Node version 16.

Getting Started

To get started, clone the Diffgram repository

git clone https://github.com/diffgram/diffgram.git
cd diffgram

Once you clone the repository and go to the main folder you can start setting up dependencies and services.

If you want to cleanup the environment to start from scratch run:

make clean

1. Setup Dev Environment

To setup a your dev environment variables and secrets, as well as your virtualenv and python dependencies run:

make setup-env-backend

Error: [Errno 13] Permission denied:

If you get a permissions denied error make sure you have permissions over the directory you are creating the virtualenv. You can grant access by running the chown command on the directory for the diffgram repo.

2. Initialize Background Services

You need an instance of postgres, rabbitmq and minio to be able to run diffgram. You can install those services manually or you can use the below utility command to spin them up from our default docker compose file.

make run-bg-services

The following sections will assume that you are running each command on a different terminal. We will prepend the prefix (T1, T2,T3,T4) to indicate which terminal you should be working on

3. [T1] Run the Default Service

make run-default

4. [T2] Run the walrus service

make run-walrus

5.[T3] Run the eventhandlers service

make run-eventhandlers

6. [T4] Install Frontend Dependencies

make frontend-deps

7. [T4] Run Frontend Service

make frontend-run

8. [T5] Run Dispatcher

make run-dispatcher

Troubleshooting

Errors While Installing Pip Dependencies

Make sure that you have installed Python 3.11 on your system and that your pip3 and python3 executables are available on your shell. If the names of the executable are pip and python without the 3 you can change the following variables on the Makefile to:

PYTHONEXEC:=python
PIPEXEC:=pip

Updated 7 months ago