Contributing to Gramex

Set up Gramex

Gramex can be developed on Python >= 3.7 on Windows or Linux. To set up the development environment:

  1. Download and install Anaconda 5.0 or later
  2. Install databases. Install PostgreSQL and MySQL. On Linux, this works:
sudo apt-get install -y git make sqlite3 postgresql postgresql-contrib libpq-dev python-dev
DEBIAN_FRONTEND=noninteractive apt-get -y -q install mysql-server

Clone and install the master branch.

git clone git@github.com:gramener/gramex.git
cd gramex
git checkout master     # Optional
pip install -e .
gramex setup --all

Test Gramex

Gramex uses nosetests for unit tests. The tests are in 2 folders:

Run make test-setup for the first time. Then you can run nosetests.

You must install Gramex Enterprise to run tests.

pip install gramexenterprise    # Install Gramex Enterprise
gramex license accept           # Accept the Gramex license

The tests take long. To test a subset, use nosetests tests.<module>:<ClassName>.<method>. For example:

make test-setup                             # Run once to install dependencies
nosetests testlib                           # Only test the libraries
nosetests testlib.test_data                 # Only run testlib/test_data.py
nosetests testlib.test_data:TestFilter      # Only run the TestFilter class
nosetests testlib.test_data:TestFilter.test_get_engine      # Run a single method

You can also see the code coverage and how long each test takes

NOSE_WITH_COVERAGE=1 nosetests      # Show code coverage. Store line-by-line results in htmlcov/
NOSE_WITH_TIMER=1 nosetests         # Show time taken for each test

Update Gramex Community Edition

In the gramex folder, create a branch for local development.

git checkout -b <branch-name>

Make your changes and check for build errors.

flake8                      # if you changed any .py files
eslint gramex/apps          # if you changed any .js files
python setup.py nosetests   # if you changed any functionality

On Windows, you may need to enable Powershell scripts.

The tests take a long time. To test a subset, use nosetests tests.<module>:<ClassName>.<method>.

Commit your changes and push your branch:

git add .
git commit -m"Your detailed description of your changes."
git push --set-upstream origin <branch-name>

Submit a pull request to the master branch. If possible:

Release Gramex Community Edition

Check build errors. Test the master branch locally on Python >= 3.7:

make test

Set up apps and & upgrade npm packages.

find gramex/apps/ -maxdepth 2 -name package.json -print0 | xargs -0 -L1 yarn upgrade

Update the following and commit to master branch:

Commit and push the master branch of both repos to the server. Ensure pipeline passes.:

git commit -m"DOC: Add v1.x.x release changes"
git push                    # Push the master branch

Merge master branch with release on both repos:

git checkout release
git merge master
git tag -a v1.x.x -m"One-line summary of features"
git push --follow-tags
git push gitlab release      # To deploy into Gramener servers. See one-time setup below
git checkout master          # Switch back to master

Note: git push gitlab release requires this one-time setup:

git remote add gitlab git@code.gramener.com:cto/gramex.git        # For Gramex
git remote add gitlab git@code.gramener.com:cto/gramex-guide.git  # For Guide

Then run these deployment steps:

# Deploy docs and code coverage on https://gramener.com/gramex-update/
make push-docs                # Deploy pydoc
make push-coverage            # Deploy coverage tests

# Deploy on pypi: https://pypi.python.org/pypi/gramex
make push-pypi                # Log in as gramener

# Deploy on conda: https://anaconda.org/gramener/gramex
# Run this on Windows AND Linux
make conda                    # Follow instructions to upload. Log in as gramener
anaconda upload /opt/conda/conda-bld/linux-64/gramex-*.tar.bz2

# Deploy on docker: https://hub.docker.com/r/gramener/gramex/
make push-docker              # Log in as sanand0 / pratapvardhan

Note: to run make conda on Linux, create a new Docker instance via docker run -it continuumio/miniconda3 /bin/bash and run:

apt-get update                                    # Update packages
apt-get install -y make gcc                       # make and gcc are the sole dependencies
git clone https://github.com/gramener/gramex/     # Clone Gramex
cd gramex
conda install -y conda-build anaconda             # Required for build
pip install -e .                                  # Test gramex, and get orderedattrdict
make conda                                        # Create conda
anaconda upload /opt/conda/conda-bld/linux-64/gramex-*.tar.bz2    # Log in as gramener

Gramener.com administrators: re-start Gramex after deployment.

Release Gramex Enterprise Edition

Update the following and commit to master branch:

Commit and push the master branch to the server. Ensure pipeline passes.:

git commit -m"DOC: Add v1.x.x release notes"
git push                    # Push the master branch

Merge with release, create an annotated tag and push the release branch:

git checkout release
git merge master
git tag -a v1.x.x -m"One-line summary of features"
git push --follow-tags
git checkout master         # Switch back to master
git merge release

Deploy on pypi:

rm -rf dist/
python setup.py sdist
# If this fails, add '-p PASSWORD'
twine upload -u gramener dist/*

Re-start gramex on deployed servers.