diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 409d8d3..df71631 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,24 +8,35 @@ To contribute to this project, please follow the [fork and pull request](https:/ ## Development Guide ### Prerequiste -Python 3.10 and docker + linux. The code breaks on python 3.8 and below, and 3.9 is untested. +Python 3.10, [Docker](https://www.docker.com/), and Linux (or WSL). The code breaks on python 3.8 and below, and 3.9 is untested. +If you are working using WSL, follow this guide to [configure Docker](https://docs.docker.com/desktop/wsl/). And for first time WSL users, I HIGHLY recommend cloning MeMaS directly into your WSL file system. ### Development Setup -Run `source setup-env.sh`, this will install all the needed development tools, as well as setup the needed environment variables. +Run `source setup-env.sh`, this will install all the needed development tools, as well as setup the needed environment variables. -### Running Docker +**NOTE that this command needs to be ran for each new shell instance, since it sets up environment variables.** +### Using Docker In the top level of this repo, run `docker compose up`, and it will spin up 1 es nodes, 1 scylla nodes, 1 milvus node, and a few more. This is a very basic development setup. -May need to run `sysctl -w vm.max_map_count=262144`. +To stop docker execution, run Control+C in the terminal you are running `docker compose up`, or run `docker compose down`. -### Initializing the MeMaS server -Due to the dependencies, the first time running MeMaS, we need to use a special command to initialize and configure the dependencies. +If you want to clean your local docker images, run +```bash +docker compose down --volumes +``` + +FYI you may need to run `sysctl -w vm.max_map_count=262144` if you get an error when trying to start elasticsearch. + +### First time initializing the MeMaS server +**NOTE: Only run this phase when you are working with a clean set of docker dependencies, aka a fresh start or after `docker compose down --volumes`.** + +Due to the service dependencies, the first time running MeMaS, we need to use a special command to initialize and configure the dependencies. After `source setup-env.sh` and `docker compose up`, wait till the services are fully started. Then run -``` +```bash flask --app 'memas.app:create_app(config_filename="memas-config.yml", first_init=True)' run ``` @@ -35,13 +46,13 @@ This will run for a while then exit. Upon exit, your MeMaS is properly setup. After `source setup-env.sh` and `docker compose up`, wait till the services are fully started. Then run -``` +```bash flask --app 'memas.app:create_app(config_filename="memas-config.yml")' run ``` to start the memas server To run the app with wsgi server, run -``` +```bash gunicorn -w 1 -k eventlet 'memas.app:create_app(config_filename="memas-config.yml")' ``` note `-w` sets the number of worker threads. @@ -50,6 +61,8 @@ note `-w` sets the number of worker threads. After `source setup-env.sh` and `docker compose up`, wait till the services are fully started. Then run -``` +```bash python3 -m pytest integration-tests -``` \ No newline at end of file +``` + +**NOTE: MeMaS server is not needed for this** \ No newline at end of file diff --git a/README.md b/README.md index 55f4e75..1a94f93 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,9 @@ - Fine grain and real time control over even individual chatbots ## Get Started +### Examples +Examples are posted in https://github.com/memas-ai/memas-examples. + ### Setting up the server TODO, right now follow [CONTRIBUTING.md](CONTRIBUTING.md) ### Client and SDK diff --git a/docker-compose.yml b/docker-compose.yml index cb748da..dde57bd 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -7,6 +7,8 @@ services: command: --smp=2 ports: - "9042:9042" + volumes: + - scylla_data:/var/lib/scylla etcd: container_name: milvus-etcd @@ -17,7 +19,7 @@ services: - ETCD_QUOTA_BACKEND_BYTES=4294967296 - ETCD_SNAPSHOT_COUNT=50000 volumes: - - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd + - etcd_data:/etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd minio: @@ -27,8 +29,8 @@ services: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin volumes: - - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data - command: minio server /minio_data + - minio_data:/data + command: minio server /data healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s @@ -43,7 +45,7 @@ services: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 volumes: - - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus + - milvus_data:/var/lib/milvus ports: - "19530:19530" - "9091:9091" @@ -72,6 +74,18 @@ volumes: esdata01: driver: local + etcd_data: + driver: local + + minio_data: + driver: local + + milvus_data: + driver: local + + scylla_data: + driver: local + networks: default: name: milvus_dev \ No newline at end of file