Log viewer

From v1.25, Gramex ships with a log file viewer.

To use it, add this to your gramex.yaml:

import:
  logviewer:
    path: $GRAMEXAPPS/logviewer/gramex.yaml   # Source of the app
    YAMLURL: $YAMLURL/log/                    # Location to mount at
    auth: ...                                 # Restrict access as required

This configuration mounts the app at log/:

Logviewer usage

Use LOGVIEWER_* variables to configure your app.

All variables are optional.

Using Variables

Examples usage with LOGVIEWER_* variables

import:
  ui:
    path: $GRAMEXAPPS/ui/gramex.yaml
    YAMLURL: $YAMLURL/ui/
  logviewer:
    path: $GRAMEXAPPS/logviewer/gramex.yaml
    YAMLURL: $YAMLURL/log/
    LOGVIEWER_LAYOUT_KWARGS:
      auth:                                  # Add auth to layout page
        login_url: /$YAMLURL/login
    LOGVIEWER_FORMHANDLER_KWARGS:
      headers:
        Cache-Control: public, max-age=3600   # cached for 1 hour
    LOGVIEWER_FORMHANDLER_QUERIES:
      kpi-pageviews:  # overwrites existing query
        SELECT SUM(duration_count) AS value
        FROM {table} {where}
      kpi-custom-metric:  # adds new query
        SELECT AVG(duration_count) AS value
        FROM {table} {where}
    LOGVIEWER_SCHEDULER_PORT: '9006'  # run scheduler only on --listen.port=9006
    LOGVIEWER_PATH_UI: $YAMLPATH/logviewer-config.yaml    # local .yaml file
    LOGVIEWER_PATH_RENDER: $YAMLPATH/logviewer-render.js  # local js file
    LOGVIEWER_CAPTURE_KWARGS:
      timeout: 30              # Change timeout to 30
    LOGVIEWER_SCHEDULER_SETUP:
      minutes: 45              # Change minute to 45
    LOGVIEWER_SCHEDULER_KWARGS:
      transforms: # Add custom transforms, default transforms will be replaced
      - type: derive
        expr:
          col: user.id
          op: NOTIN
          value: ['-', 'dev']
        as: user.id_1

Multiple logviewer instances

When running multiple instances of the same app, each instance will start log viewer. Avoid this by specifing LOGVIEWER_SCHEDULER_PORT: <one-port>. For example:

logviewer:
    path: $GRAMEXAPPS/logviewer/gramex.yaml
    YAMLURL: $YAMLURL/log/
    LOGVIEWER_SCHEDULER_PORT: '9001'    # Run LogViewer only if the app is run on port 9001, not otherwise

Now, you can run the app on multiple ports. Only port 9001 will host the log viewer:

gramex --listen.port=9001     # Log viewer works here
gramex --listen.port=9002     # ... not here
gramex --listen.port=9003     # ... nor here

logviewer.db

Gramex logs all HTTP requests to logs/requests.csv under $GRAMEXDATA.

It logs:

Typical requests.csv looks like

1530008663609.0,::1,,200,2708.0,GET,/,
1530008665319.0,::1,,200,948.0,GET,/ui/jquery/dist/jquery.min.js,
1530008665404.0,::1,,200,81.0,GET,/script.js,
1530008665410.0,::1,,200,1.0,GET,/style.css,
1530008667319.0,::1,,404,678.0,GET,/favicon.ico,HTTPError: HTTP 404: Not Found
1530012727106.0,::1,user1,200,210.0,GET,/,
1530012729481.0,::1,,200,74.0,GET,/ui/jquery/dist/jquery.min.js,
1530012729538.0,::1,user2,200,1.0,GET,/script.js,
1530012729594.0,::1,,200,2.0,GET,/style.css,

By default, requests.csv are backed up on a weekly basis with date prefix. For eg: requests.csv.2017-11-21.

Logviewer application uses data from logviewer.db for the front-end visuals. logviewer.db stores the aggregated data (day (aggD), week (aggW), month (aggM)) tables of requests.csv*.

Data is grouped for every combination of (time (daily), user.id, ip, status, uri) and aggregrated on (duration, new_session, session_time) metrics with (_count, _sum) suffix.

Session Calculations

Every time a user logs into a gramex app, a new_session is flagged. session_time duration (in seconds) is the length of time someone spends on the app.

For example, let’s take user1

Let’s take another scenario:

Session related metrics include:

Note: You’d want to ignore test users, non-logged-in users [-, dev] for session related calulations. As they tend to skew the session duration.

Currently, logviewer application having session related visuals, is based on following

You can customize kpi-avgtimespent to consider only logged-in users

import:
  logviewer:
    path: $GRAMEXAPPS/logviewer/gramex.yaml
    YAMLURL: $YAMLURL/log/
    LOGVIEWER_FORMHANDLER_QUERIES:
      kpi-avgtimespent:
        SELECT SUM(session_time_sum)/SUM(new_session_sum) AS value
        FROM {table}
        WHERE "user.id_1" == 1 {where}

user.id_1 by default ignores ['-', 'dev'] users.

Add custom visuals

TODO

Add 3rd-party data

TODO