Deployment patterns

Development and deployment are usually on different machines with different configurations, file paths, database locations, etc. All of these can be configured in gramex.yaml using pre-defined variables.

Deployment checklist

Deploy on gramener.com servers

Gramener employees who commit to code.gramener.com can deploy to gramener.com or uat.gramener.com.

Add the following to .gitlab-ci.yml in your repository.

# Reference: http://doc.gitlab.com/ce/ci/yaml/README.html
deploy:
  stage: deploy
  script: deploy # Run the Gramener deployment script
  only: [master, /dev-.*/] # List branches to deploy as a list or RegExs
  variables:
    SERVER: ubuntu@uat.gramener.com # Deploy to uat.gramener.com/app-name/
    URL: app-name # Change this to your app-name
    SETUP: gramex setup . # You can use any setup script here
    VERSION: py3v1 # py3v1 or static
    WEBSOCKET: enabled # Optional: websocket support
    CORS: enabled # Optional: open CORS access

You may change the following keys:

Secrets

v1.64. Passwords, access tokens, and other sensitive information must be protected. There are 3 ways of doing this.

gramex.yaml secrets

If your repository and server are fully secured (i.e. only authorized people can access them) add secrets to gramex.yaml. For example:

url:
  login/google:
    pattern: /$YAMLURL/login/
    handler: GoogleAuth
    kwargs:
      key: YOURKEY
      secret: YOURSECRET

.secrets.yaml

If you can edit files on the server (directly, or via CI), add a .secrets.yaml file with your secrets, like this:

PASSWORD: your-secret-password
GOOGLE_AUTH_SECRET: your-secret-key
TWITTER_SECRET: your-secret-key
# ... etc

These variables are available in gramex.yaml as $PASSWORD, $GOOGLE_AUTH_SECRET, $TWITTER_SECRET, etc. For example:

url:
  login/google:
    pattern: /$YAMLURL/login/
    handler: GoogleAuth
    kwargs:
      key: add-your-key-directly # This is not a secret
      secret: $GOOGLE_AUTH_SECRET # This comes from .secrets.yaml

You can also access it in code (e.g. in FunctionHandler) as gramex.config.variables['PASSWORD'], etc. For example:

from gramex.config import variables

def my_function_handler():
    password = variables.get('PASSWORD', '')
    google_auth_secret = variables['GOOGLE_AUTH_SECRET']
    twitter_secret = variables['TWITTER_SECRET']
    ...

NOTE: Don’t commit the .secrets.yaml file. Everyone who can access the repo can see the secret.

.secrets.yaml imports

v1.68. .secrets.yaml can import from other files. For example:

# Imports all variables from another-secret-file.yaml
SECRETS_IMPORT: another-secret-file.yaml

# You can import from a file pattern. This imports .secrets.yaml from all
# immediate subdirectories
SECRETS_IMPORT: '*/.secrets.yaml'

# You can specify a list of imports
SECRETS_IMPORT:
  - app1/.secrets.yaml
  - app2/.secrets.yaml

# ... or a dict of imports. Keys are ignored. Values are used.
SECRETS_IMPORT:
  app1: app1/.secrets.yaml
  app2: app2/.secrets.yaml

This is useful when a Gramex instance runs multiple apps, each having its own .secrets.yaml file. The main app can use SECRETS_IMPORT: */.secrets.yaml to import secrets from all subdirectories.

Any secrets in the main file override the secrets in an imported file.

.secrets.yaml URLs

If you can’t edit files on the server, you can pick up encrypted secrets from a public URL in 3 steps.

  1. Encrypt your secrets using this secret encryption tool
  2. Store the encrypted secret anywhere publicly (e.g. at PasteBin or Github)
  3. Add the URL and the encryption secret, in .secrets.yaml:
SECRETS_URL: https://pastebin.com/raw/h75e4mWx
SECRETS_KEY: SECRETKEY # Replace with your secret key

When Gramex loads, it loads SECRETS_URL (ignoring comments beginning with #) and decrypts it using your SECRETS_KEY. The above URL has 2 secrets: REMOTE_SECRET1 and REMOTE_SECRET2, which you can use as variables.

This lets you securely modify the secrets without accessing the server.

Deploying secrets

v1.64.When deploying via Gitlab CI, add your secrets as environment variables under Settings > CI / CD > Variables. Then add this line to your .gitlab-ci.yml deployment script:

script:
  # List environment variables to export
  - "secrets KEY1 KEY2 ... > .secrets.yaml"
  # Continue with your deployment script

Docker image

To create a Docker image from an app, add this Dockerfile to your app:

FROM gramener/gramex:latest
COPY . .
RUN "$CONDA_DIR/bin/gramex" setup .

Build the image using:

docker build --pull -t my-app:latest .

Run the image using:

docker run -p 9988:9988 my-app:latest

Windows Service

Video

v1.23. To set up a Gramex application as a service, run PowerShell or the Command Prompt as administrator. Then:

cd D:\path\to\your\app
gramex service install

To start / stop the service, go to Control Panel > Administrative Tools > View Local Services. You can also do this from the command prompt as administrator:

gramex service start
gramex service stop

Once started, the application is live at the port specified in your gramex.yaml, which defaults to 9988, so visit http://localhost:9988/.

Here are additional install options:

gramex service install
    --cwd  "C:/path/to/application/"    # Run Gramex in this directory
    --user "DOMAIN\USER"                # Optional user to run as
    --password "user-password"          # Required if user is specified
    --startup manual|auto|disabled      # Default is manual

The user domain and name are stored as environment variables USERDOMAIN and USERNAME. Run echo %USERDOMAIN% %USERNAME% on the Command Prompt to see them.

You can update these parameters any time via:

gramex service update --...             # Same parameters as install

To uninstall the service, run:

gramex service remove

Troubleshooting Windows Services

If the service doesn’t run, check the log files. Log files can be accessed as follows:

Check PyWin32 paths.

PyWin32 has a common problem. When you run gramex service install, you may get this warning:

The executable at "...\Lib\site-packages\win32\PythonService.exe" is being used as a service.

This executable doesn't have pythonXX.dll and/or pywintypesXX.dll in the same
directory. This is likely to fail when used in the context of a service.

The exact environment needed will depend on which user runs the service and
where Python is installed. If the service fails to run, this will be why.

NOTE: You should consider copying this executable to the directory where these
DLLs live - "...\Lib\site-packages\win32" might be a good place.

Or, when starting the service, you may get “Error starting service: The service did not respond to the start or control request in a timely fashion”.

In that case:

  1. Copy the following files under ...\Lib\site-packages\win32\ (same location as the error above).
    • pythonXX.dll from ...\ – the root of your Conda environment. Replace XX with 37 for Python 3.7, etc.
    • pywintypesXX.dll from ...\Library\bin\. Replace XX with 37 for Python 3.7, etc.
    • Avoid python Scripts/pywin32_postinstall.py -install copies into C:\Windows\system32. It doesn’t work well across Python versions
  2. If psutil imports fails, download the relevant psutil wheel and pip install it from your conda environment.
  3. Run gramex service remove
  4. Run gramex service install to re-install. Check that the’re no warning now
  5. Run gramex service start. You should see a service.log file in the source folder with the Gramex console.logs

Check Permissions. If you get an Access is denied error like this:

pywintypes.error: (5, 'OpenSCManager', 'Access is denied.')

… then re-run from an Administrator Command Prompt.

Multiple Windows Services

To create multiple services running at different directories or ports, you can create one or more custom service classes in yourproject_service.py:

import gramex.winservice

class YourProjectGramexService(gramex.winservice.GramexService):
    _svc_name_ = 'YourServiceID'
    _svc_display_name_ = 'Your Service Display Name'
    _svc_description_ = 'Description of your service'
    _svc_port_ = 8123  # optional custom port

if __name__ == '__main__':
    import sys
    import logging
    logging.basicConfig(level=logging.INFO)
    YourProjectGramexService.setup(sys.argv[1:])

Install the service via:

python yourproject_service.py install --cwd=D:\path\to\app\

Remove the service via:

python yourproject_service.py remove

Windows administration

Here are some common Windows administration actions when deploying on Windows server:

Windows scheduled tasks

To run a scheduled task on Windows, use PowerShell v3 (ref):

$dir = "D:\app-dir"
$name = "Scheduled-Task-Name"

Unregister-ScheduledTask -TaskName $name -TaskPath $dir -Confirm:$false -ErrorAction:SilentlyContinue
$action = New-ScheduledTaskAction –Execute "D:\anaconda\bin\python.exe" -Argument "$dir\script.py" -WorkingDirectory $dir
$trigger = New-ScheduledTaskTrigger -Daily -At "5:00am"
Register-ScheduledTask –TaskName $name -TaskPath $dir -Action $action –Trigger $trigger –User 'someuser' -Password 'somepassword'

An alternative for older versions of Windows / PowerShell is schtasks.exe:

schtasks /create /tn your-task-name /sc HOURLY /tr "gramex"                # To run as current user
schtasks /create /tn your-task-name /sc HOURLY /tr "gramex" /ru SYSTEM     # To run as system user

Linux service

Set up a systemd service for Gramex. For example:

Copy this file into /etc/systemd/system/your-app.service

[Unit]
Description=Describe your Gramex application

[Service]
Type=simple
User=ubuntu                             # Run Gramex as this user, e.g. root
ExecStart=/path/to/gramex               # Path to Gramex script
WorkingDirectory=/path/to/your/app      # Path your your app
Restart=always                          # Restart app if Gramex exits
RestartSec=10                           # ... after 10 seconds

[Install]
WantedBy=multi-user.target              # Start app on reboot, after network is up

Then run sudo systemctl daemon-reload to reload the services.

Now you can run:

sudo systemctl start your-app           # Start the service
sudo systemctl status your-app          # Check status of service
sudo systemctl stop your-app            # Stop the service
sudo systemctl restart your-app         # Restart the service
sudo systemctl enable your-app          # Ensure service starts on reboot

sudo journalctl -u your-app             # See the logs on your app

Linux scheduled tasks

To run a scheduled task on Linux, use crontab

Proxy servers

Gramex is often deployed behind a reverse proxy. This allows a web server (like nginx, Apache, IIS, Tomcat) to pass requests to different ports running different applications.

Here’s a checklist

  1. Pass original HTTP headers to Gramex that capture the actual request the Proxy Server received. Gramex can redirect URLs appropriately based on this.
    • Host: Actual host the request was sent to (else proxy servers send localhost or an internal IP)
    • X-Real-IP: Remote address of the request (else proxy servers send their own IP)
    • X-Scheme: Protocol (HTTP/HTTPS) the request was made with (else proxy servers stay with HTTP)
    • X-Request-URI: Original request URL (else proxyservers send http://localhost:port/...)
    • X-Gramex-Root: Base URL of Gramex application
  2. Enable websocket proxying via proxy_pass on nginx, and mod_proxy_wstunnel on Apache
  3. Increase proxy timeout via proxy_read_timeout on nginx, and ProxyTimeout on Apache.
  4. Increase upload file size via client_max_body_size on nginx, and LimitRequestBody on Apache.
  5. Set up proxy caching via proxy_cache on nginx, and mod_cache on Apache.
  6. Enable HTTPS via certbot on nginx, and certbot on Apache.

nginx reverse proxy

Here is a minimal HTTP reverse proxy configuration:

server {
    listen 80;                              # 80 is the default HTTP port
    server_name example.com;                # http://example.com/
    server_tokens off;                      # Hide server name

    # Pass original HTTP headers
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-Request-URI $request_uri;

    # Enable websocket proxying
    proxy_http_version 1.1;
    proxy_set_header Upgrade    $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    # Increase proxy timeout
    proxy_read_timeout 300s;

    # Increase upload file size
    client_max_body_size 100m;

    location /project/ {                    # example.com/project/* maps to
        proxy_pass http://127.0.0.1:9988/;  # 127.0.0.1:9988/
        proxy_redirect ~^/ /project/;       # Redirects are sent back to /project/
        proxy_set_header X-Gramex-Root /project/;   # Tells Gramex where the app is hosted
    }
}

The use of the trailing slash makes a big difference in nginx.

    location /project/ { proxy_pass http://127.0.0.1:9988/; }   # Trailing slash
    location /project/ { proxy_pass http://127.0.0.1:9988; }    # No trailing slash

The first maps example.com/project/* to http://127.0.0.1:9988/*. The second maps it to http://127.0.0.1:9988/project/*.

If you have one Gramex running multiple applications under /app1, /app2, etc, your config file will be like:

    location /app1/ { proxy_pass http://127.0.0.1:8001; }
    location /app2/ { proxy_pass http://127.0.0.1:8001; }
    location /app3/ { proxy_pass http://127.0.0.1:8001; }

But if your have multiple Gramex instances at port 8001, 8002, etc, each running an app under their /, your config file will be like:

   location /app1/ {
        proxy_pass http://127.0.0.1:8001/;
        proxy_redirect ~^/ /app1/;
    }
    location /app2/ {
        proxy_pass http://127.0.0.1:8002/;
        proxy_redirect ~^/ /app2/;
    }
    location /app3/ {
        proxy_pass http://127.0.0.1:8003/;
        proxy_redirect ~^/ /app3/;
    }

To let nginx cache responses, use:

        # Ensure /var/cache/nginx/ is owned by nginx:nginx with 700 permissions
        proxy_cache_path /var/cache/nginx/your-project-name
                         levels=1:2
                         keys_zone=your-project-name:100m
                         inactive=10d
                         max_size=2g;
        proxy_cache your-project-name;
        proxy_cache_key "$host$request_uri";
        proxy_cache_use_stale error timeout updating http_502 http_503 http_504;

To delete specific entries from the nginx cache, use nginx-cache-purge.

nginx load balancing

To distribute load, run multiple Gramex instances on different ports on one or more servers. Add this to your nginx configuration:

upstream balancer-name {
  ip_hash;  // Same IP address goes to same port
  server 127.0.0.1:9988;
  server 127.0.0.1:9989;
}

server {
  // Instead of http://127.0.0.1:9988/ use the upstream balancer
  location / {
    proxy_pass http://balancer-name/;
  }
}

Apache reverse proxy

First, enable thee relevant proxy modules. On Linux, run:

# Required modules
sudo a2enmod proxy proxy_http proxy_wstunnel rewrite headers

On Windows, ensure that the Apache httpd.conf has the following lines:

LoadModule proxy_module          modules/mod_proxy.so
LoadModule proxy_http_module     modules/mod_proxy_http.so
LoadModule proxy_wstunnel        modules/mod_proxy_wstunnel.so
LoadModule rewrite_module        modules/mod_rewrite.so
LoadModule headers_module        modules/mod_headers.so

Here is a minimal HTTP reverse proxy configuration:

# Make sure you have a "Listen 80" in your configuration.
<VirtualHost *:80>
    ServerName example.com
    ProxyPass        /project/ http://127.0.0.1:9988/
    ProxyPassReverse /project/ http://127.0.0.1:9988/

    # Pass original HTTP headers
    ProxyPreserveHost On
    RequestHeader set X-Real-IP "%{REMOTE_ADDR}s"
    RequestHeader set X-Scheme "%{REQUEST_SCHEME}s"
    RequestHeader set X-Request-URI %{REQUEST_URI}s
    RequestHeader set X-Gramex-Root "/project/"

    # Enable websocket proxying
    RewriteEngine on
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule ^/project/(.*) "ws://127.0.0.1:9988/$1" [P,L]

    # Increase proxy timeout
    ProxyTimeout 300

    # Increase upload file size
    LimitRequestBody 102400000
</VirtualHost>

If you have one Gramex on port 8001 running multiple applications under /app1, /app2, etc, your config file will be like:

ProxyPass        /app1/ http://127.0.0.1:8001/app1/
ProxyPassReverse /app1/ http://127.0.0.1:8001/app1/

ProxyPass        /app2/ http://127.0.0.1:8001/app2/
ProxyPassReverse /app2/ http://127.0.0.1:8001/app2/

But if your have multiple Gramex instances at port 8001, 8002, etc, each running an app under their /, your config file will be like:

ProxyPass        /app1/ http://127.0.0.1:8001/
ProxyPassReverse /app1/ http://127.0.0.1:8001/

ProxyPass        /app2/ http://127.0.0.1:8002/
ProxyPassReverse /app2/ http://127.0.0.1:8002/

Apache load balancing

To distribute load, run multiple Gramex instances on different ports on one or more servers. Here is a minimal Apache configuration:

LoadModule proxy_module                modules/mod_proxy.so
LoadModule headers_module              modules/mod_headers.so
LoadModule proxy_http_module           modules/mod_proxy_http.so
LoadModule authz_core_module           modules/mod_authz_core.so
LoadModule slotmem_shm_module          modules/mod_slotmem_shm.so
LoadModule proxy_balancer_module       modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module  modules/mod_lbmethod_byrequests.so

Listen 80

<VirtualHost *:80>
    # If the server name matches one of the following, apply the configuration
    ServerName domain.server.com
    ServerAlias localhost 127.0.0.1 otherdomain.server.com

    # Load balancer set up. Give the balancer a unique name
    <Proxy "balancer://balancer-name">
        # Add multiple instances to balance load across
        BalancerMember "http://127.0.0.1:9988" route=1
        BalancerMember "http://127.0.0.1:9989" route=2
        # The ROUTEID cookie ensures that the same session goes to the same backend
        ProxySet stickysession=ROUTEID
    </Proxy>

    ProxyPass "/" "balancer://balancer-name/"
    ProxyPassReverse "/" "balancer://balancer-name/"

    # ... add other Apache proxy configurations
</VirtualHost>

Relative URL mapping

Your app may be running at http://localhost:9988/ on your system, but will be running at http://server/app/ on the server. Use relative URLs and paths to allow the application to work in both places.

Suppose /gramex.yaml imports all sub-directories:

import: */gramex.yaml   # Import all gramex.yaml from 1st-level sub-directories

… and /app/gramex.yaml has:

url:
  page-name:
    pattern: /$YAMLURL/page # Note the /$YAMLURL prefix
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/page.html # Note the $YAMLPATH prefix

When you run Gramex from /app/, the pattern becomes /page ($YAMLURL is .)

When you run Gramex from /, the pattern becomes /app/page ($YAMLURL is /app)

The correct file (/app/page.html) is rendered in both cases because $YAMLPATH points to the absolute directory of the YAML file.

You can modify the app name using ../new-app-name. For example, this pattern directs the URL /new-app-name/page to /app/page.html.

pattern: /$YAMLURL/../new-app-name/page

You also need this in redirection URLs. See this example:

url:
  auth/simple:
    pattern: /$YAMLURL/simple
    handler: SimpleAuth
    kwargs:
      credentials: { alpha: alpha }
      redirect: { url: /$YAMLURL/ } # Note the $YAMLURL here

Using /$YAMLURL/ redirects users back to this app’s home page, rather than the global home page (which may be uat.gramener.com/.)

Tips:

Using relative URLs

In your HTML code, use relative URLs where possible. For example: http://localhost:9988/ becomes . (not / – which is an absolute URL.) Similarly, /css/style.css becomes css/style.css.

Sometimes, this is not possible. For example, /main/ and /main/sub/ use the same template, you can’t specify ../style.css and ../../style.css in the same file.

Instead, in gramex.yaml, pass an APP_ROOT variable to the template that has the absolute path to the application root. For example, this can be:

url:
  deploy-url:
    pattern: /$YAMLURL/url/(.*) # Any URL under this directory
    handler: FileHandler # is rendered as a FileHandler
    kwargs:
      path: $YAMLPATH/template.html # Using this template
      transform:
        "template.html":
          # APP_ROOT is the path to the root of the application
          function: template(content, APP_ROOT='/'))

Now you can use the APP_ROOT to locate static files in the template:

<link rel="stylesheet" href="{{ APP_ROOT }}/style.css" />

If your app is behind a proxy server, the URL may not be /. In that case, pass the X-Gramex-Root HTTP header, and combine with $YAMLURL to locate your app root.

transform:
  "template.html":
    # APP_ROOT is the path to the root of the application
    function: template(content, APP_ROOT=handler.gramex_root + r'$YAMLURL')

To test this, open the following URLs:

In every case, the correct absolute path for style.css is used, irrespective of which path the app is deployed at.

Using YAMLPATH

$YAMLPATH is very similar to $YAMLURL. It is the relative path to the current gramex.yaml location.

When using a FileHandler like this:

url:
  app-home:
    pattern: / # This is hard-coded
    handler: FileHandler
    kwargs:
      path: index.html # This is hard-coded

… the locations are specified relative to where Gramex is running. To make it relative to where the gramex.yaml file is, use:

url:
  app-home:
    pattern: /$YAMLURL/
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/index.html # Path is relative to this directory

Tips:

Security

When deploying your application, go through this checklist and apply all that is relevant.

Gramex already has these security practices enabled. Don’t disable these unless required:

deploy.yaml

The most common security options are pre-configured in $GRAMEXPATH/deploy.yaml. Specifically, it:

To enable these options, add this line to your gramex.yaml:

import: $GRAMEXPATH/deploy.yaml

See deploy.yaml to understand the configurations.

nginx security

To hide the Server: nginx/* header (even if security audit teams change the method type from GET to TRACE/OPTIONS), install nginx-extras which includes headers-more-nginx. On Ubuntu, run:

sudo apt-get install nginx-extras

Then add:

location /project/ {
  more_set_headers 'Server: custom_name';
}

Restart nginx.

Content Security Policy

To generate a Content Security Policy nonce value in your template, use this code in your template:

{% from gramex.config import random_string %} {% set nonce =
random_string(size=10) %} {% set handler.set_header('Content-Security-Policy',
f"object-src 'self'; script-src 'self' 'nonce-{nonce}'") %}
<script nonce="{{ nonce }}">
  // ...
</script>

Testing

To check for application vulnerabilities, run tools such as:

HTTPS Server

The best way to set up Gramex as an HTTP server is to run it behind a Proxy Server like nginx or Apache.

server {
  listen 443 ssl http2;         # Use HTTP and HTTP/2 on port 443 with SSL
  server_name example.com;      # https://example.com/
  server_tokens off;            # Hide server version

  # Use https://certbot.eff.org/ to generate these certificates
  ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

  # Disable SSLv3 -- only IE6 needs it, and the POODLE security hole makes it vulnerable
  # Disable TLSv1 and TLSv1.1. See https://www.packetlabs.net/tls-1-1-no-longer-secure/
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers EECDH+CHACHA20:EECDH+AES;
  ssl_prefer_server_ciphers on;

  # Prevent man-in-the-middle attack by telling browsers to only use HTTPS, not HTTP, for 1 year
  # https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security (HSTS)
  add_header Strict-Transport-Security "max-age=31536000";
}

# Redirect HTTP to HTTPS

server {
  listen      80;             # Redirect port 80 on
  server_name example.com;    # http://example.com/
  location / {                # Permanently redirect to the HTTPS page
    return 301 https://$host$request_uri;
  }
  # Serve custom error HTML for 301 and 302 HTTP status codes -- to avoid reporting server name
  error_page 301 302 /error.html;
  location = /error.html {
    root /var/www/html;
    internal;
  }

Direct HTTPS server

To set up Gramex directly as a HTTPS server (not recommended for production), create a certificate file and a key file, both in PEM format. Use the following settings in gramex.yaml:

app:
  listen:
    port: 443
    ssl_options:
      certfile: "path/to/certificate.pem"
      keyfile: "path/to/privatekey.pem"

You can then connect to https://your-gramex-server/.

To generate a free HTTPS certificate for a domain, visit certbot or letsencrypt.org/.

To generate a self-signed HTTPS certificate for testing, run:

openssl genrsa -out privatekey.pem 1024
openssl req -new -key privatekey.pem -out certrequest.csr
openssl x509 -req -in certrequest.csr -signkey privatekey.pem -out certificate.pem

Or you can use these pre-created privatekey.pem and certificate.pem for localhost. (This was created with subject /C=IN/ST=KA/L=Bangalore/O=Gramener/CN=localhost/emailAddress=s.anand@gramener.com and is meant for localhost.)

All browsers will report that this connection is not trusted, since it is a self-signed certificate. Ignore the warning proceed to the website.

CORS

New in 1.78. When one server sends a request to another server via browser JavaScript, we need to enable Cross-Origin Resource Sharing (CORS).

If a Gramex app is deployed on multiple servers, or if you want a client-side app to fetch data from a URL, add cors: true to the URL’s kwargs. For example, this page returns session information to pages from any server:

url:
  deploy-cors:
    pattern: /$YAMLURL/cors
    handler: FunctionHandler
    kwargs:
      function: handler.session
      cors: true # Enable CORS

cors: true is a shortcut for:

cors:
  origins: "*" # Allow from all servers
  methods: "*" # Allow all HTTP methods
  headers: "*" # Allow all default HTTP headers
  auth: true # Allow cookies

cors: can have the following keys:

For example, this CORS page can be accessed from any server via AJAX / fetch.

Source

CORS page

On any page, you can run this JS code:

fetch("https://gramener.com/gramex/guide/deploy/cors", { method: "POST" })
  .then((r) => r.text())
  .then(console.log);

This will send a POST request to the CORS page, and print the response.

CORS POST with auth

CORS does not send cookie information by default, and workarounds were required. Since Gramex 1.78, this is handled automatically by Gramex with cors: true.

So, if you have:

… then you can send a POST request from x.example.com to y.example.com like this:

fetch("https//y.example.com", { method: "POST", credentials: "include" })
  .then((r) => r.text())
  .then(console.log);

Note: For CORS to work on CaptureHandler with authentication, pass a ?domain= argument that has the domain for which cookies are to be set.

Shared deployment

To deploy on Gramener’s UAT server, see the UAT deployment section and deployment tips.

This is a shared deployment, where multiple apps deployed on one Gramex instance. Here are common deployment errors in shared environments:

Works locally but not on server

If your app is at D:/app/, don’t run gramex from D:/app/. Run it from D:/ with this D:/gramex.yaml:

import: app/gramex.yaml

This tests the application in a shared deployment setup. The application may run from D:/app/ but fail from D:/ - giving you a chance to find out why.

403 Forbidden

$GRAMEXPATH/deploy.yaml disables all non-standard files for security. If another app imports deploy.yaml, your application may not be able to access a file - e.g. data.csv. Create a FileHandler to explicitly allow your file.

url:
  myapp/allow-files: # Create/modify a handler to allow files
    pattern: /$YAMLURL/data/(.*)
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/data/
      allow: ["data.csv", "*.xlsx"] # Explicitly allow required types

Do not use a custom deploy.yaml in your project. Import from $GRAMEXPATH instead. Reference. In case of a blanket disallow of files, refer to 403 forbidden error above and resolve.

404 Not Found

Most often, this is due to relative paths. When running locally, the app requests /style.css. But on the server, it is deployed at /app/. So the URL must be /app/style.css. To avoid this, always use relative URLs - e.g. style.css, ../style.css, etc – avoid the leading slash.

Another reason is incorrect handler name conflicts.

Handler name conflict

If your app and another app both use a URL handler called data, only one of these will be loaded.

url:            # THIS IS WRONG!
    data:       # This is defined by app1 -- only this config is loaded
        ...
    data:       # This is defined by app2 -- ignored with a warning
        ...

Ensure that each project’s URL handler is pre-fixed with a unique ID:

url: # THIS IS RIGHT
  app1/data: # This is defined by app1
    ...
  app2/data: # This is defined by app2 -- does not conflict with app1/data
    ...

Import conflict

If your app and another app both import: the same YAML script, the namespaces inside those will obviously collide:

import: # THIS IS WRONG!
  app1/ui: # app1
    path: $GRAMEXAPPS/ui/gramex.yaml # imports UI components
    YAMLURL: $YAMLURL/app1/ui/ # at /app1/ui/
  app2/ui: # app2
    path: $GRAMEXAPPS/ui/gramex.yaml # imports UI components
    YAMLURL: $YAMLURL/app2/ui/ # at /app2/ui/

Add the namespace key to avoid collision in specified sections. A safe use is namespace: [url, cache, schedule, watch]

import: # THIS IS RIGHT
  app1/ui: # app1
    namespace: [url] # ensures names in url: are unique
    path: $GRAMEXAPPS/ui/gramex.yaml
    YAMLURL: $YAMLURL/app1/ui/
  app2/ui: # app2
    namespace: [url] # ensures names in url: are unique
    path: $GRAMEXAPPS/ui/gramex.yaml
    YAMLURL: $YAMLURL/app2/ui/

Python file conflict

If your app and another app both use a Python file called common.py, only one of these is imported. Prefix the Python files with a unique name, e.g. app1_common.py.

Missing dependency

Your app may depend on an external library – e.g. a Python module, node module or R library.

The quickest way is to set these up is to run gramex setup <folder> method. This automatically installs:

Log file order

Different instances of Gramex may flush their logs at different times. Do not expect log files to be in order. For example, in this request log, the 2nd entry has a timestamp greater than the third:

1519100063656,220.227.50.9,user1@masked.com,304,1,GET,/images/bookmark.png,
1519100063680,106.209.240.105,user2@masked.com,304,55,GET,/bookmark_settings?mode=display,
1519100063678,220.227.50.9,user3@masked.com,304,1,GET,/images/filters-toggler.png,

Common errors

Here are common errors from the Gramex logs:

Port 8001 is busy

For example:

ERROR 21-May 11:41:45 __init__ Port 8001 is busy. Use --listen.port= for a different port

You are running Gramex on a port that is already running Gramex or another application.

On Linux, run lsof -i :8001 to find processes running on port 8001 and use kill to kill the process. Or start Gramex on a different port using gramex --listen.port=<new-port>

Cannot start Capture

Here is a typical error message:

Traceback (most recent call last):
  File "/home/ubuntu/gramex/gramex/handlers/capturehandler.py", line 111, in _start
    self._validate_server(r)
  File "/home/ubuntu/gramex/gramex/handlers/capturehandler.py", line 169, in _validate_server
    raise RuntimeError('Server: %s at %s is not %s' % (server, self.url, script))
RuntimeError: Server: Capture/1.0.0 at http://localhost:9900/ is not chromecapture.js

This indicates that the web application running at port 9900 is not ChromeCapture but something else. You can either:

  1. Change the CaptureHandler port to a free one (e.g. from 9900 to 9910), OR
  2. Stop what’s running at the CaptureHandler port using lsof -i 9900 and kill, and then restart Gramex.

Cannot load function

Here is an example:

ERROR 12-May 10:38:24 transforms url:box_data: Cannot load function views.get_box_data

This indicates that views.py does not have a get_box_data() function. You can locate this under the box_data: under url:. This can happen for 3 reasons:

  1. Gramex loaded a different views.py (e.g. from another project) before your views.py, and your views.py is being ignored. Solution: rename your file like <your_project>_views.py to make it unique.
  2. Gramex loaded views.py earlier and is not reloading it. This happens on rare occasions. Solution: restart Gramex.
  3. views.py does not have a get_box_data() function. Check the spelling.

Duplicate key

Here is an example:

WARNING 03-Oct 17:17:44 config Duplicate key: url.login/google

This indicates that login/google: key under url: has been repeated across different gramex.yaml files – which have been import:ed under the main gramex.yaml. The first login/google: is used. The rest are ignored.

Ensure that no keys duplicated across gramex.yaml files and restart gramex.

Other common errors