Contributing to Gramex

Set up Gramex

Gramex can be developed on Python >= 3.7 on Windows or Linux. To set up the development environment:

  1. Download and install Anaconda 5.0 or later
  2. Install databases. Install PostgreSQL and MySQL. On Linux, this works:
sudo apt-get install -y git make sqlite3 postgresql postgresql-contrib libpq-dev python-dev
DEBIAN_FRONTEND=noninteractive apt-get -y -q install mysql-server

Clone and install the master branch.

git clone git@github.com:gramener/gramex.git
cd gramex
git checkout master     # Optional
pip install -e .
gramex setup --all

Test Gramex

Gramex uses nosetests for unit tests. The tests are in 2 folders:

Run make test-setup for the first time. Then you can run nosetests.

The tests take long. To test a subset, use nosetests tests.<module>:<ClassName>.<method>. For example:

make test-setup                             # Run once to install dependencies
nosetests testlib                           # Only test the libraries
nosetests testlib.test_data                 # Only run testlib/test_data.py
nosetests testlib.test_data:TestFilter      # Only run the TestFilter class
nosetests testlib.test_data:TestFilter.test_get_engine      # Run a single method

You can also see the code coverage and how long each test takes

NOSE_WITH_COVERAGE=1 nosetests      # Show code coverage. Store line-by-line results in htmlcov/
NOSE_WITH_TIMER=1 nosetests         # Show time taken for each test

Update Gramex Community Edition

In the gramex folder, create a branch for local development.

git checkout -b <branch-name>

Make your changes and check for build errors.

flake8                      # if you changed any .py files
eslint gramex/apps          # if you changed any .js files
python setup.py nosetests   # if you changed any functionality

On Windows, you may need to enable Powershell scripts.

The tests take a long time. To test a subset, use nosetests tests.<module>:<ClassName>.<method>.

Commit your changes and push your branch:

git add .
git commit -m"Your detailed description of your changes."
git push --set-upstream origin <branch-name>

Submit a pull request to the master branch. If possible:

Release Gramex Community Edition

Check build errors. Test the master branch locally on Python >= 3.7:

make test

Set up apps and & upgrade npm packages.

# yarn upgrade --cwd gramex/apps/admin/
yarn upgrade --cwd gramex/apps/admin2/
yarn upgrade --cwd gramex/apps/capture/
yarn upgrade --cwd gramex/apps/configeditor/
yarn upgrade --cwd gramex/apps/filemanager/
yarn upgrade --cwd gramex/apps/logviewer/
yarn upgrade --cwd gramex/apps/pynode/
yarn upgrade --cwd gramex/apps/ui/

Update the following and commit to master branch:

Commit and push the master branch of both repos to the server. Ensure pipeline passes.:

git commit -m"DOC: Add v1.x.x release changes"
git push                    # Push the master branch

Merge master branch with release on both repos:

git checkout release
git merge master
git tag -a v1.x.x -m"One-line summary of features"
git push --follow-tags
git push gitlab release      # To deploy into Gramener servers. See one-time setup below
git checkout master          # Switch back to master

Note: git push gitlab release requires this one-time setup:

git remote add gitlab git@code.gramener.com:cto/gramex.git        # For Gramex
git remote add gitlab git@code.gramener.com:cto/gramex-guide.git  # For Guide

Gramener.com administrators: re-start Gramex after deployment.

Deploy on gramener.com, then pypi, then docker.

make push-pypi                # Log in as gramener
make push-docker              # Log in as sanand0 / pratapvardhan
make push-docs push-coverage  # Push docs and coverage tests

Release Gramex Enterprise Edition

Update the following and commit to master branch:

Commit and push the master branch to the server. Ensure pipeline passes.:

git commit -m"DOC: Add v1.x.x release notes"
git push                    # Push the master branch

Merge with release, create an annotated tag and push the release branch:

git checkout release
git merge master
git tag -a v1.x.x -m"One-line summary of features"
git push --follow-tags
git checkout master         # Switch back to master
git merge release

Deploy on pypi:

rm -rf dist/
python setup.py sdist
# If this fails, add '-p PASSWORD'
twine upload -u gramener dist/*

Re-start gramex on deployed servers.