Contributing to Gramex

Gramex Community Edition is open source. You’re welcome to contribute by raising issues and pull requests.

Set up Gramex for development

Step 1: Install Anaconda, NodeJS, and Git. Add node and git to your PATH.

You also need bash to test. On Windows, this comes with Git.

Step 2: Clone and install the master branch.

# Create a new Conda environment with Python 3.11+
conda create -y --name gramex python=3.11
conda activate gramex

# Clone and install Gramex
git clone git@github.com:gramener/gramex.git
cd gramex
# master branch is the latest working version.
# release branch is the latest released version.
git checkout master
pip install -e . --config-settings editable_mode=strict
gramex setup --all

Step 3 (OPTIONAL): Install databases.

Test Gramex

You must install Gramex Enterprise to run tests.

# Install gramex & gramexenterprise
pip install gramex gramexenterprise
gramex license accept

# Install dependencies
pip install -r tests/requirements.txt

# Run tests
bash task lint
nosetests

Gramex uses nosetests for unit tests. The tests are in 2 folders:

The tests take long. To test a subset, use nosetests tests.<module>:<ClassName>.<method>. For example:

make test-setup                             # Run once to install dependencies
nosetests testlib                           # Only test the libraries
nosetests testlib.test_data                 # Only run testlib/test_data.py
nosetests testlib.test_data:TestFilter      # Only run the TestFilter class
nosetests testlib.test_data:TestFilter.test_get_engine      # Run a single method

You can also see the code coverage and how long each test takes

NOSE_WITH_COVERAGE=1 nosetests      # Show code coverage. Store line-by-line results in htmlcov/
NOSE_WITH_TIMER=1 nosetests         # Show time taken for each test

Release Gramex Community Edition

Run tests:

  1. Update Gramex version number in pyproject.toml and gramex/__init__.py.
  2. Test the master branch locally on Python >= 3.7
bash task test

Upgrade npm packages and audit security. Ensure reports/* don’t report errors:

bash task clean reformat update security stats

In gramex-guide:

Commit and push the master branch of both repos to the server. Ensure pipeline passes.:

export VERSION=1.x.x
git commit . -m"DOC: Add v$VERSION release changes"
git push

Merge master branch with release on both repos:

git checkout release
git merge master
git tag -a v$VERSION -m"Release v$VERSION"
git push --follow-tags
git push gitlab release master  # Deploy into Gramener servers. See one-time setup below
git checkout master

Note: git push gitlab release requires this one-time setup:

git remote add gitlab https://code.gramener.com/cto/gramex.git        # For Gramex
git remote add gitlab https://code.gramener.com/cto/gramex-guide.git  # For Guide

Then run these deployment steps on the Gramex repo:

# Deploy docs on https://gramener.com/gramex/guide/api/
bash task docs pushdocs

# Deploy on pypi: https://pypi.python.org/pypi/gramex
bash task pushpypi    # Log in as gramener

# Deploy on docker: https://hub.docker.com/r/gramener/gramex/
VERSION=1.x.x bash task builddocker pushdocker    # Log in as sanand0 / pratapvardhan

Gramener.com administrators: re-start Gramex after deployment.

Release Gramex Enterprise Edition

Update the following and commit to master branch:

Commit and push the master branch to the server. Ensure pipeline passes.:

git commit . -m"DOC: Add v1.x.x release notes"
git push                    # Push the master branch

Merge with release, create an annotated tag and push the release branch:

git checkout release
git merge master
git tag -a v1.x.x -m"One-line summary of features"
git push --follow-tags
git checkout master         # Switch back to master
git merge release

Deploy on PyPi:

rm -rf dist/
python setup.py sdist
# Set up ~/.pypirc with the API token for "gramener" account
# https://pypi.org/help/#apitoken
twine upload dist/*

Re-start gramex on deployed servers.

Committing

Begin the commit with a prefix:

Add a short subject line in imperative mood, e.g. FIX: FileHandler transforms require encoding.

Add a long body. Explain WHAT and WHY.

Video tutorials

For each section of each page on the guide, we add YouTube videos that explain that section.

For example, the Function arguments from URL page links to this video.

Decide the storyline

Record the video

Insert the video

Select “Share” on the YouTube video and copy the link, e.g. https://youtu.be/hSwQQ2wOmIk Then add this snippet to the beginning of the section:

[Video](https://youtu.be/hSwQQ2wOmIk){.youtube}

Docstrings

Docstrings are written in Markdown and compiled using MkDocs.