Deployment patterns

Development and deployment are usually on different machines with different configurations, file paths, database locations, etc. All of these can be configured in gramex.yaml using pre-defined variables.

Deployment checklist

Secrets

v1.64. Passwords, access tokens, and other sensitive information must be protected. There are 3 ways of doing this.

gramex.yaml secrets

If your repository and server are fully secured (i.e. only authorized people can access them) add secrets to gramex.yaml. For example:

url:
  login/google:
    pattern: /$YAMLURL/login/
    handler: GoogleAuth
    kwargs:
      key: YOURKEY
      secret: YOURSECRET

.secrets.yaml

If you can edit files on the server (directly, or via CI), add a .secrets.yaml file with your secrets, like this:

PASSWORD: your-secret-password
GOOGLE_AUTH_SECRET: your-secret-key
TWITTER_SECRET: your-secret-key
# ... etc

These variables are available in gramex.yaml as $PASSWORD, $GOOGLE_AUTH_SECRET, $TWITTER_SECRET, etc. For example:

url:
  login/google:
    pattern: /$YAMLURL/login/
    handler: GoogleAuth
    kwargs:
      key: add-your-key-directly    # This is not a secret
      secret: $GOOGLE_AUTH_SECRET   # This comes from .secrets.yaml

NOTE: Don’t commit the .secrets.yaml file. Everyone who can access the repo can see the secret.

.secrets.yaml URLs

If you can’t edit files on the server, you can pick up encrypted secrets from a public URL in 3 steps.

  1. Encrypt your secrets using this secret encryption tool
  2. Store the encrypted secret anywhere publicly (e.g. at PasteBin or Github)
  3. Add the URL and the encryption secret, in .secrets.yaml:
SECRETS_URL: https://pastebin.com/raw/h75e4mWx
SECRETS_KEY: SECRETKEY     # Replace with your secret key

When Gramex loads, it loads SECRETS_URL (ignoring comments beginning with #) and decrypts it using your SECRETS_KEY. The above URL has 2 secrets: REMOTE_SECRET1 and REMOTE_SECRET2, which you can use as variables.

This lets you securely modify the secrets without accessing the server.

Deploying secrets

v1.64.When deploying via Gitlab CI, add your secrets as environment variables under Settings > CI / CD > Variables. Then add this line to your .gitlab-ci.yml deployment script:

script:
  # List environment variables to export
  - 'secrets KEY1 KEY2 ... > .secrets.yaml'
  # Continue with your deployment script

Windows Service

v1.23. To set up a Gramex application as a service, run PowerShell or the Command Prompt as administrator. Then:

cd D:\path\to\your\app
gramex service install

To start / stop the service, go to Control Panel > Administrative Tools > View Local Services. You can also do this from the command prompt as administrator:

gramex service start
gramex service stop

Once started, the application is live at the port specified in your gramex.yaml, which defaults to 9988, so visit http://localhost:9988/.

Here are additional install options:

gramex service install
    --cwd  "C:/path/to/application/"    # Run Gramex in this directory
    --user "DOMAIN\USER"                # Optional user to run as
    --password "user-password"          # Required if user is specified
    --startup manual|auto|disabled      # Default is manual

The user domain and name are stored as environment variables USERDOMAIN and USERNAME. Run echo %USERDOMAIN% %USERNAME% on the Command Prompt to see them.

You can update these parameters any time via:

gramex service update --...             # Same parameters as install

To uninstall the service, run:

gramex service remove

Service logs can be viewed using the Windows Event Viewer. Gramex logs are at %LOCALAPPDATA%\Gramex Data\logs\ unless over-ridden by gramex.yaml.

To create multiple services running at different directories or ports, you can create one or more custom service classes in yourproject_service.py:

import gramex.winservice

class YourProjectGramexService(gramex.winservice.GramexService):
    _svc_name_ = 'YourServiceID'
    _svc_display_name_ = 'Your Service Display Name'
    _svc_description_ = 'Description of your service'
    _svc_port_ = 8123               # optional custom port

if __name__ == '__main__':
    import sys
    import logging
    logging.basicConfig(level=logging.INFO)
    YourProjectGramexService.setup(sys.argv[1:])

You can now run:

python yourproject_service.py install --cwd=...     # install the service
python yourproject_service.py remove                # uninstall the service
... etc ...

Windows administration

Here are some common Windows administration actions when deploying on Windows server:

Windows scheduled tasks

To run a scheduled task on Windows, use PowerShell v3 (ref):

$dir = "D:\app-dir"
$name = "Scheduled-Task-Name"

Unregister-ScheduledTask -TaskName $name -TaskPath $dir -Confirm:$false -ErrorAction:SilentlyContinue
$action = New-ScheduledTaskAction –Execute "D:\anaconda\bin\python.exe" -Argument "$dir\script.py" -WorkingDirectory $dir
$trigger = New-ScheduledTaskTrigger -Daily -At "5:00am"
Register-ScheduledTask –TaskName $name -TaskPath $dir -Action $action –Trigger $trigger –User 'someuser' -Password 'somepassword'

An alternative for older versions of Windows / PowerShell is schtasks.exe:

schtasks /create /tn your-task-name /sc HOURLY /tr "gramex"                # To run as current user
schtasks /create /tn your-task-name /sc HOURLY /tr "gramex" /ru SYSTEM     # To run as system user

Linux service

Set up a systemd service for Gramex. For example:

Copy this file into /etc/systemd/system/your-app.service

[Unit]
Description=Describe your Gramex application

[Service]
Type=simple
User=ubuntu                             # Run Gramex as this user, e.g. root
ExecStart=/path/to/gramex               # Path to Gramex script
WorkingDirectory=/path/to/your/app      # Path your your app
Restart=always                          # Restart app if Gramex exits
RestartSec=10                           # ... after 10 seconds

[Install]
WantedBy=multi-user.target              # Start app on reboot, after network is up

Now you can run:

sudo systemctl start your-app           # Start the service
sudo systemctl status your-app          # Check status of service
sudo systemctl stop your-app            # Stop the service
sudo systemctl restart your-app         # Restart the service
sudo systemctl enable your-app          # Ensure service starts on reboot

Linux scheduled tasks

To run a scheduled task on Linux, use crontab

Proxy servers

Gramex is often deployed behind a reverse proxy. This allows a web server (like nginx, Apache, IIS, Tomcat) to pass requests to different ports running different applications.

Here’s a checklist

  1. Pass original HTTP headers to Gramex that capture the actual request the Proxy Server received. Gramex can redirect URLs appropriately based on this.
    • Host: Actual host the request was sent to (else proxy servers send localhost or an internal IP)
    • X-Real-IP: Remote address of the request (else proxy servers send their own IP)
    • X-Scheme: Protocol (HTTP/HTTPS) the request was made with (else proxy servers stay with HTTP)
    • X-Request-URI: Original request URL (else proxyservers send http://localhost:port/...)
  2. Enable websocket proxying via proxy_pass on nginx, and mod_proxy_wstunnel on Apache
  3. Increase proxy timeout via proxy_read_timeout on nginx, and ProxyTimeout on Apache.
  4. Increase upload file size via client_max_body_size on nginx, and LimitRequestBody on Apache.
  5. Set up proxy caching via proxy_cache on nginx, and mod_cache on Apache.
  6. Enable HTTPS via certbot on nginx, and certbot on Apache.

nginx reverse proxy

Here is a minimal HTTP reverse proxy configuration:

server {
    listen 80;                              # 80 is the default HTTP port
    server_name example.com;                # http://example.com/
    server_tokens off;                      # Hide server name

    # Pass original HTTP headers
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-Request-URI $request_uri;

    # Enable websocket proxying
    proxy_http_version 1.1;
    proxy_set_header Upgrade    $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    # Increase proxy timeout
    proxy_read_timeout 300s;

    # Increase upload file size
    client_max_body_size 100m;

    location /project/ {                    # example.com/project/* maps to
        proxy_pass http://127.0.0.1:9988/;  # 127.0.0.1:9988/
        proxy_redirect ~^/ /project/;       # Redirects are sent back to /project/
    }
}

The use of the trailing slash makes a big difference in nginx.

    location /project/ { proxy_pass http://127.0.0.1:9988/; }   # Trailing slash
    location /project/ { proxy_pass http://127.0.0.1:9988; }    # No trailing slash

The first maps example.com/project/* to http://127.0.0.1:9988/*. The second maps it to http://127.0.0.1:9988/project/*.

If you have one Gramex running multiple applications under /app1, /app2, etc, your config file will be like:

    location /app1/ { proxy_pass http://127.0.0.1:8001; }
    location /app2/ { proxy_pass http://127.0.0.1:8001; }
    location /app3/ { proxy_pass http://127.0.0.1:8001; }

But if your have multiple Gramex instances at port 8001, 8002, etc, each running an app under their /, your config file will be like:

   location /app1/ {
        proxy_pass http://127.0.0.1:8001/;
        proxy_redirect ~^/ /app1/;
    }
    location /app2/ {
        proxy_pass http://127.0.0.1:8002/;
        proxy_redirect ~^/ /app2/;
    }
    location /app3/ {
        proxy_pass http://127.0.0.1:8003/;
        proxy_redirect ~^/ /app3/;
    }

To let nginx cache responses, use:

        # Ensure /var/cache/nginx/ is owned by nginx:nginx with 700 permissions
        proxy_cache_path /var/cache/nginx/your-project-name
                         levels=1:2
                         keys_zone=your-project-name:100m
                         inactive=10d
                         max_size=2g;
        proxy_cache your-project-name;
        proxy_cache_key "$host$request_uri";
        proxy_cache_use_stale error timeout updating http_502 http_503 http_504;

To delete specific entries from the nginx cache, use nginx-cache-purge.

Apache reverse proxy

First, enable thee relevant proxy modules. On Linux, run:

# Required modules
sudo a2enmod proxy proxy_http proxy_wstunnel rewrite headers

On Windows, ensure that the Apache httpd.conf has the following lines:

LoadModule proxy_module          modules/mod_proxy.so
LoadModule proxy_http_module     modules/mod_proxy_http.so
LoadModule proxy_wstunnel        modules/mod_proxy_wstunnel.so
LoadModule rewrite_module        modules/mod_rewrite.so
LoadModule headers_module        modules/mod_headers.so

Here is a minimal HTTP reverse proxy configuration:

# Make sure you have a "Listen 80" in your configuration.
<VirtualHost *:80>
    ServerName example.com
    ProxyPass        /project/ http://127.0.0.1:9988/
    ProxyPassReverse /project/ http://127.0.0.1:9988/

    # Pass original HTTP headers
    ProxyPreserveHost On
    RequestHeader set X-Real-IP "%{REMOTE_ADDR}s"
    RequestHeader set X-Scheme "%{REQUEST_SCHEME}s"
    RequestHeader set X-Request-URI %{REQUEST_URI}s

    # Enable websocket proxying
    RewriteEngine on
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule ^/project/(.*) "ws://127.0.0.1:9988/$1" [P,L]

    # Increase proxy timeout
    ProxyTimeout 300

    # Increase upload file size
    LimitRequestBody 102400000
</VirtualHost>

If you have one Gramex on port 8001 running multiple applications under /app1, /app2, etc, your config file will be like:

ProxyPass        /app1/ http://127.0.0.1:8001/app1/
ProxyPassReverse /app1/ http://127.0.0.1:8001/app1/

ProxyPass        /app2/ http://127.0.0.1:8001/app2/
ProxyPassReverse /app2/ http://127.0.0.1:8001/app2/

But if your have multiple Gramex instances at port 8001, 8002, etc, each running an app under their /, your config file will be like:

ProxyPass        /app1/ http://127.0.0.1:8001/
ProxyPassReverse /app1/ http://127.0.0.1:8001/

ProxyPass        /app2/ http://127.0.0.1:8002/
ProxyPassReverse /app2/ http://127.0.0.1:8002/

Apache load balancing

To distribute load you can run multiple Gramex instances on different ports on a single server or on multiple servers. You could configure Apache server to serve requests from multiple instances.

Here is a minimal configuration to use the Apache server for proxy load balancing:

LoadModule proxy_module                modules/mod_proxy.so
LoadModule headers_module              modules/mod_headers.so
LoadModule proxy_http_module           modules/mod_proxy_http.so
LoadModule authz_core_module           modules/mod_authz_core.so
LoadModule slotmem_shm_module          modules/mod_slotmem_shm.so
LoadModule proxy_balancer_module       modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module  modules/mod_lbmethod_byrequests.so

Listen 80

<VirtualHost *:80>
    # If the server name matches one of the following, apply the configuration
    ServerName domain.server.com
    ServerAlias localhost 127.0.0.1 otherdomain.server.com

    # Load balancer set up. Give the balancer a unique name
    <Proxy "balancer://balancer-name">
        # Add multiple instances to balance load across
        BalancerMember "http://127.0.0.1:9988" route=1
        BalancerMember "http://127.0.0.1:9989" route=2
        # The ROUTEID cookie ensures that the same session goes to the same backend
        ProxySet stickysession=ROUTEID
    </Proxy>

    ProxyPass "/" "balancer://balancer-name/"
    ProxyPassReverse "/" "balancer://balancer-name/"

    # ... add other Apache proxy configurations
</VirtualHost>

Relative URL mapping

Your app may be running at http://localhost:9988/ on your system, but will be running at http://server/app/ on the server. Use relative URLs and paths to allow the application to work in both places.

Suppose /gramex.yaml imports all sub-directories:

import: */gramex.yaml   # Import all gramex.yaml from 1st-level sub-directories

… and /app/gramex.yaml has:

url:
    page-name:
        pattern: /$YAMLURL/page         # Note the /$YAMLURL prefix
        handler: FileHandler
        kwargs:
            path: $YAMLPATH/page.html   # Note the $YAMLPATH prefix

When you run Gramex from /app/, the pattern becomes /page ($YAMLURL is .)

When you run Gramex from /, the pattern becomes /app/page ($YAMLURL is /app)

The correct file (/app/page.html) is rendered in both cases because $YAMLPATH points to the absolute directory of the YAML file.

You can modify the app name using ../new-app-name. For example, this pattern directs the URL /new-app-name/page to /app/page.html.

    pattern: /$YAMLURL/../new-app-name/page

You also need this in redirection URLs. See this example:

url:
  auth/simple:
    pattern: /$YAMLURL/simple
    handler: SimpleAuth
    kwargs:
      credentials: {alpha: alpha}
      redirect: {url: /$YAMLURL/}        # Note the $YAMLURL here

Using /$YAMLURL/ redirects users back to this app’s home page, rather than the global home page (which may be uat.gramener.com/.)

Tips:

Using relative URLs

In your HTML code, use relative URLs where possible. For example: http://localhost:9988/ becomes . (not / – which is an absolute URL.) Similarly, /css/style.css becomes css/style.css.

Sometimes, this is not desirable. For example, If you are linking to the same CSS file from different directories, you need specifying /style.css is helpful. This requires server-side templating.

You can use a Tornado template like this that using a pre-defined variable, e.g. APP_ROOT.

<link rel="stylesheet" href="/{{ APP_ROOT }}/style.css">

In gramex.yaml, we pass APP_ROOT to the that’s set to $YAMLURL. For example:

variables:
    APP_ROOT: $YAMLURL       # Pre-define APP_ROOT as the absolute URL to gramex.yaml's directory

url:
    deploy-url:
        pattern: /$YAMLURL/url/(.*)               # Any URL under this directory
        handler: FileHandler                      # is rendered as a FileHandler
        kwargs:
            path: $YAMLPATH/template.html         # Using this template
            transform:
                "template.html":
                    # Convert to a Tornado template
                    # Pass the template the APP_ROOT variable
                    function: template(content, APP_ROOT="$APP_ROOT")

To test this, open the following URLs:

In every case, the correct absolute path for /style.css is used, irrespective of which path the app is deployed at.

Using YAMLPATH

$YAMLPATH is very similar to $YAMLURL. It is the relative path to the current gramex.yaml location.

When using a FileHandler like this:

url:
  app-home:
    pattern: /                  # This is hard-coded
    handler: FileHandler
    kwargs:
      path: index.html          # This is hard-coded

… the locations are specified relative to where Gramex is running. To make it relative to where the gramex.yaml file is, use:

url:
  app-home:
    pattern: /$YAMLURL/
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/index.html        # Path is relative to this directory

Tips:

Security

When deploying your application, go through this checklist and apply all that is relevant.

Gramex already has these security practices enabled. Don’t disable these unless required:

deploy.yaml

The most common security options are pre-configured in $GRAMEXPATH/deploy.yaml. Specifically, it:

To enable these options, add this line to your gramex.yaml:

import: $GRAMEXPATH/deploy.yaml

See deploy.yaml to understand the configurations.

Testing

To check for application vulnerabilities, run tools such as:

HTTPS Server

The best way to set up Gramex as an HTTP server is to run it behind a Proxy Server like nginx or Apache. Use certbot to generate a HTTPS certificate.

But to set up Gramex directly as a HTTPS server (not recommended for production), create certificate file and a key file, both in PEM format. Use the following settings in gramex.yaml:

app:
    listen:
        port: 443
        ssl_options:
            certfile: "path/to/certificate.pem"
            keyfile: "path/to/privatekey.pem"

You can then connect to https://your-gramex-server/.

To generate a free HTTPS certificate for a domain, visit certbot or letsencrypt.org/.

To generate a self-signed HTTPS certificate for testing, run:

openssl genrsa -out privatekey.pem 1024
openssl req -new -key privatekey.pem -out certrequest.csr
openssl x509 -req -in certrequest.csr -signkey privatekey.pem -out certificate.pem

Or you can use these pre-created privatekey.pem and certificate.pem for localhost. (This was created with subject /C=IN/ST=KA/L=Bangalore/O=Gramener/CN=localhost/emailAddress=s.anand@gramener.com and is meant for localhost.)

All browsers will report that this connection is not trusted, since it is a self-signed certificate. Ignore the warning proceed to the website.

CORS

When one server sends a request to another server via AJAX, we need to enable Cross-Origin Resource Sharing (CORS).

To enable this, you need to add the Access-Control-Allow-Origin: * HTTP header. For example:

url:
  cors-page:
    pattern: /$YAMLURL/cors-page
    handler: ...
    kwargs:
      ...
      headers:
        Access-Control-Allow-Origin: '*'        # Allow CORS from any domain

Now, you can access this URL from any server via AJAX. For example:

$.getJSON('https://gramex-server/cors-page')
    .done(...)

Instead of Access-Control-Allow-Origin: *, you may specify a single domain, like: Access-Control-Allow-Origin: https://gramener.com/.

But if you want to allow multiple domains, define them in Python. For example:

url:
  cors-page:
    pattern: ...
    handler: FunctionHandler
    kwargs:
      function: mymodule.mycalc(handler)    # Headers are set by this function
# mymodule.py
def mycalc(handler):
    # The request has an 'Origin:' headers.
    origin = handler.request.headers.get('Origin')
    # If it is an allowed domain, send the same as a response.
    allowed_domains = {'https://gramener.com', 'https://learn.gramener.com'}
    if origin in allowed_domains:
        handler.set_header('Access-Control-Allow-Origin', origin)
    # ... rest of your code

CORS POST with auth

CORS does not send cookie information. Nor does it send custom HTTP headers (e.g. X-XsrfToken). So CORS does not work with POST requests with XSRF, nor with authenticated users.

To enable a Gramex client server to communicate a Gramex host server via CORS, you need to do 4 things:

  1. In the client, send the XSRF token and cookie from the HTML file. Note: this uses templates:
<script>
$.ajax('https://gramex-server/cors-page', {
  method: 'POST',
  xhrFields: {withCredentials: true}            // Send cookies
  data: { _xsrf: '{{ handler.xsrf_token }}' },  // Send XSRF token
})
</script>
  1. In the host, add additional headers in gramex.yaml:
url:
  cors:
    pattern: /$YAMLURL/cors-page
    handler: FunctionHandler
    kwargs:
      function: mymodule.mycalc(handler)
      methods: [GET, POST, OPTIONS]             # Important: Allow OPTIONS
      auth: true                                # Pick any auth conditions
      headers:
          Access-Control-Allow-Methods: GET, POST, OPTIONS      # Important
          Access-Control-Allow-Credentials: true                # Important
          # Access-Control-Allow-Origin: must be set dynamically by mycalc()
  1. In the client AND the host, enable a distributed session data mechanism like Redis, and also to share cookies:
app:
  session:
    type: redis
    path: localhost:6379:0      # Run redis on localhost at port 6379. This uses DB 0
    domain: .your-domain.com    # Allows cookies to be shared between *.your-domain.com
  1. In the host mymodule.mycalc(), set the Access-Control-Allow-Origin header:
def mycalc(handler):
    origin = handler.request.headers.get('Origin', '*')
    handler.set_header('Access-Control-Allow-Origin', origin)

Shared deployment

To deploy on Gramener’s UAT server, see the UAT deployment section and deployment tips.

This is a shared deployment, where multiple apps deployed on one Gramex instance. Here are common deployment errors in shared environments:

Works locally but not on server

If your app is at D:/app/, don’t run gramex from D:/app/. Run it from D:/ with this D:/gramex.yaml:

import: app/gramex.yaml

This tests the application in a shared deployment setup. The application may run from D:/app/ but fail from D:/ - giving you a chance to find out why.

403 Forbidden

$GRAMEXPATH/deploy.yaml disables all non-standard files for security. If another app imports deploy.yaml, your application may not be able to access a file - e.g. data.csv. Create a FileHandler to explicitly allow your file.

url:
  myapp/allow-files:                        # Create/modify a handler to allow files
    pattern: /$YAMLURL/data/(.*)
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/data/
      allow: ['data.csv', '*.xlsx']         # Explicitly allow required file types

Do not use a custom deploy.yaml in your project. Import from $GRAMEXPATH instead. Reference. In case of a blanket disallow of files, refer to 403 forbidden error above and resolve.

404 Not Found

Most often, this is due to relative paths. When running locally, the app requests /style.css. But on the server, it is deployed at /app/. So the URL must be /app/style.css. To avoid this, always use relative URLs - e.g. style.css, ../style.css, etc – avoid the leading slash.

Another reason is incorrect handler name conflicts.

Handler name conflict

If your app and another app both use a URL handler called data, only one of these will be loaded.

url:            # THIS IS WRONG!
    data:       # This is defined by app1 -- only this config is loaded
        ...
    data:       # This is defined by app2 -- this is ignored with a warning in the log
        ...

Ensure that each project’s URL handler is pre-fixed with a unique ID:

url:            # THIS IS RIGHT
    app1/data:  # This is defined by app1
        ...
    app2/data:  # This is defined by app2 -- does not conflict with app1/data
        ...

Import conflict

If your app and another app both import: the same YAML script, the namespaces inside those will obviously collide:

import:                                     # THIS IS WRONG!
    app1/ui:                                # app1
        path: $GRAMEXAPPS/ui/gramex.yaml    # imports UI components
        YAMLURL: $YAMLURL/app1/ui/          # at /app1/ui/
    app2/ui:                                # app2
        path: $GRAMEXAPPS/ui/gramex.yaml    # imports UI components
        YAMLURL: $YAMLURL/app2/ui/          # at /app2/ui/

Add the namespace key to avoid collision in specified sections. A safe use is namespace: [url, cache, schedule, watch]

import:                                     # THIS IS RIGHT
    app1/ui:                                # app1
        namespace: [url]                    # ensures that url: section names are unique
        path: $GRAMEXAPPS/ui/gramex.yaml
        YAMLURL: $YAMLURL/app1/ui/
    app2/ui:                                # app2
        namespace: [url]                    # ensures that url: section names are unique
        path: $GRAMEXAPPS/ui/gramex.yaml
        YAMLURL: $YAMLURL/app2/ui/

Python file conflict

If your app and another app both use a Python file called common.py, only one of these is imported. Prefix the Python files with a unique name, e.g. app1_common.py.

Missing dependency

Your app may depend on an external library – e.g. a Python module, node module or R library. Ensure that this is installed on the server. The preferred method is to use the gramex install method. Specifically:

Log file order

Different instances of Gramex may flush their logs at different times. Do not expect log files to be in order. For example, in this request log, the 2nd entry has a timestamp greater than the third:

1519100063656,220.227.50.9,user1@masked.com,304,1,GET,/images/bookmark.png,
1519100063680,106.209.240.105,user2@masked.com,304,55,GET,/bookmark_settings?mode=display,
1519100063678,220.227.50.9,user3@masked.com,304,1,GET,/images/filters-toggler.png,

nginx log analyzer

Goaccess is useful for server monitoring. It can be configured for Apache or nginx. Follow below steps to enable monitoring:

1) Install Goaccess

2) Enable monitoring via terminal, pointing to access.log to generate report.html

goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED

This can be run as a cronjob at periodic intervals to monitor server health.

Real-time monitoring can be enabled and exposed via websockets via --real-time-html

goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED --real-time-html --ws-url=ws://*****IP*****:7890 --ignore-crawlers --daemonize

Updated log is pushed via websockets on port 7890. --daemonize runs the task as a daemon and --ignore-crawlers ignores web crawlers.

If report.html should be accessed on an endpoint, ensure it is configured in nginx or other-related routes. To serve it in a specific project,

url:
  project_report:
    pattern: /$YAMLURL/report
    handler: FileHandler
    kwargs:
      path: $YAMLPATH/report.html   # report.html is generated from goaccess

Supporting configuration in nginx would be as below:

# nginx.conf
server {
    # here, the report is hosted on a Gramex app running on 9920 port
    location /report/  { proxy_pass http://127.0.0.1:9920/report; }
}

This report can be access at: yourapp.gramener.com/report/

GoAccess Output

Following attributes are reported in graphical and textual formats:

goaccess dashboard

Common errors

Here are common errors from the Gramex logs:

Port 8001 is busy

For example:

ERROR 21-May 11:41:45 __init__ Port 8001 is busy. Use --listen.port= for a different port

You are running Gramex on a port that is already running Gramex or another application.

On Linux, run lsof -i :8001 to find processes running on port 8001 and use kill to kill the process. Or start Gramex on a different port using gramex --listen.port=<new-port>

Cannot start Capture

Here is a typical traceback:

Traceback (most recent call last):
  File "/home/ubuntu/gramex/gramex/handlers/capturehandler.py", line 111, in _start
    self._validate_server(r)
  File "/home/ubuntu/gramex/gramex/handlers/capturehandler.py", line 169, in _validate_server
    raise RuntimeError('Server: %s at %s is not %s' % (server, self.url, script))
RuntimeError: Server: Capture/1.0.0 at http://localhost:9900/ is not chromecapture.js

This indicates that the web application running at port 9900 is not ChromeCapture but something else. You can either:

  1. Change the CaptureHandler port to a free one (e.g. from 9900 to 9910), OR
  2. Stop what’s running at the CaptureHandler port using lsof -i 9900 and kill, and then restart Gramex.

Cannot load function

Here is an example:

ERROR 12-May 10:38:24 transforms url:box_data: Cannot load function views.get_box_data

This indicates that views.py does not have a get_box_data() function. You can locate this under the box_data: under url:. This can happen for 3 reasons:

  1. Gramex loaded a different views.py (e.g. from another project) before your views.py, and your views.py is being ignored. Solution: rename your file like <your_project>_views.py to make it unique.
  2. Gramex loaded views.py earlier and is not reloading it. This happens on rare occasions. Solution: restart Gramex.
  3. views.py does not have a get_box_data() function. Check the spelling.

Duplicate key

Here is an example:

WARNING 03-Oct 17:17:44 config Duplicate key: url.login/google

This indicates that login/google: key under url: has been repeated across different gramex.yaml files – which have been import:ed under the main gramex.yaml. The first login/google: is used. The rest are ignored.

Ensure that no keys duplicated across gramex.yaml files and restart gramex.

Other common errors