SAMLAuth is now a part of Gramex. Most enterprise Single-Sign-On (SSO) implementations are now SAML-enabled, including Microsoft ActiveDirectory. Gramex can integrate with all of these apps now. (@vinay.ranjan)
ProxyHandler is a new handler that lets you:
FormHandler has several enhancements:
?table=... as well as path
arguments like /database/table.headers:
kwarg.Alerts command line usage gives you access to Smart alerts give you access to alert emails from command line. You can use this as a local mail merge app.
FunctionHandler
supports the method: kwarg. You can decide which HTTP methods to
support - including PUT, DELETE, etc.
A few common queries are documented:
Stats:
FormHandler has two
major upgrades. FormHandler
charts
use the Seaborn library to generate
static charts on the server as SVG or PNG (PDF too.) Setting ?meta=y
returns metadata as HTTP headers.
The UI component library now uses Bootstrap 4 stable version. New components are:
.modal-left and .modal-right overlays$().formhandler() component. This lets you
render FormHandler data as an interactive Excel-like table
component. (@tejesh.papineni)Smart alerts can now:
email: service defined by default (you don’t need to
specify a service: every time)PPTXHandler exposes URL
query parameters in the configuration. This lets you generate
presentations whose content can be updated by the URL. For example, you
can create a certificate template, and set ?name= to update the
recipient’s name. (@ranjan.balappa).
FunctionHandler can now return NumPy objects as well – not just Python objects. (@tejesh.papineni)
Thanks to the recent Docuthon, documentation has improved. Key changes:
Note: the condition() transform is deprecated. No known repository
uses it.
Stats:
g1 is upgraded to v0.3 which
features a $().formhandler() component. This renders FormHandlers as
Excel-like tables (sortable, filterable).
FormHandler supports a
?meta=y query parameter that returns metadata about the query. This
includes the number of rows, offset, limit, sort options, columns
excluded, etc.
The UI component library now uses Bootstrap 4 Beta 3. New components are:
.arrow-tail which adds a tail to arrows.border-radius-sm, .border-radius-lg, etc which create rounded
corners of different sizes.btn-xs for extra-small buttons.modal-xl for extra-large modals.cursor-pointer as a utility class for cursor: pointer.switch and .uploadThe UI component library page itself is more usable:
CaptureHandler
supports repeated ?dpi= arguments for PPTX that allow creating
multiple slides with differently sized images (@pragnya.reddy).
?title_size= sets the title font size for pptx.
Using auth handlers for AJAX login is now well documented.
A new session
data store type
called type: sqlite is available. This is a bit slower, but allows
multiple Gramex instances to share session data.
Smart Alerts templates can now access the Gramex config. This lets you re-use templates across different alerts, changing static content in the YAML configuration (@mukul.taneja).
Gramex supports a docker install option. The documentation also features common Windows administration options used when deploying Gramex.
Stats:
This release features an upgrade to the UI component library:
/ui/bootstraptheme.css is minified by
defaultCaptureHandler for
Chrome supports a window.renderComplete option. Set
?delay=renderComplete. This waits until window.renderComplete is
true and then captures the page.
DBAuth can use a CSV file as its database.
PPTXHandler handles edge cases better:
This release also adds better debugging features:
gramex --settings.debug sets the console log level to DEBUG. This
makes it easier to start Gramex in debug mode.gramex.debug.cache print the executed query in debug modeGramex now uses bandit to test for internal vulnerabilities like SQL injection.
Stats:
This release features Smart Alerts - a rule-based email alert service. This can be used to:
The UI component library has been extended with several custom components:
Also, D3 4.0 is now part of the UI components library.
A basic Log viewer app is part of Gramex. It shows the history of all pages accessed on Gramex.
Gramex console logs are more informative. Each request prints the name of the handler used to process it. This tells you whether the correct handler processed the URL or not. Also, when starting up, the list of all handler classes and priorities and shown.
To enable debug mode from the command prompt, run
gramex --settings.debug.
Credits:
Stats:
Note: Before installing this release, you install
node 8.x or above, and also run
npm install -g yarn. Also run pip install with a --verbose option.
Gramex installs several UI libraries and the installation is slow. Yarn
speeds up the installation. --verbose lets you see progress.
This release adds a UI component library that includes a series of standard front-end libraries and a Gramex-customized version of Bootstrap 4. By @bhanu.kamapantula
All auth handlers support a inactive expiry feature that closes a session if no requests were made for a certain period.
DBAuth supports a Sign up feature that lets users create their own accounts. By @nikhil.kabbin
DBAuth used to
ignore the redirect: key when directly POSTing via AJAX. So the
response would always redirect to /. If / is not a valid URL, it
would return an error. This is now fixed –
DBAuth always
uses redirect:.
PPTXHandler pptgen supports text styles, heatgrid order, pie/donut colors, and a number of other features. By @sanjay.yadav
FormHandler and
gramex.data.filter accept a queryfile: parameter that lets you
specify queries in a separate SQL file. This makes indentation and
syntax highlighting easier, making it easier to debug queries.
gramex init and all Gramex installations use Yarn in offline mode if
possible - prefering Yarn over npm. This is to optimize installations.
A few developer enhancements and bugfixes:
gramex.cache.open can open XML, RSS and Atom files using lxml. It
returns an etree object.handler.get_arg(key) method that is exactly
like Tornado’s handler.get_argument(key), but supports Unicodegramex.cache.Subprocess waits for return code and then exitsStats:
This is an interim release with minor features and major bugfixes.
gramex init is the new way of initializing Gramex repos. It just
copies the minimal files required to get started, but will soon
include boilerplates.prepare: function. You can decrypt browser-encrypted passwords,
prefix a DOMAIN\ to the username, or restrict access by IP. Issue
180gramex.debug.print - is a smarter replacement
for print. It also prints the file and line where you inserted the
print statement, making it easier to trace flow.gramex.debug.trace() decorator makes
it very easy to see which lines in a function were executed.The bugfixes are:
gramex.cache.open used to cache based on the file and its type,
not arguments. So gramex.cache.open('data.csv', encoding='cp1252')
and gramex.cache.open('data.csv', encoding='utf-8') would return
the same cached result. This is fixed. Issue
171schema.table.) Issue
185 and Issue
186Stats:
This release adds Gramex as a Windows
service,
making it easier for Windows administrators to auto-start and manage
Gramex. Run gramex service install from the app directory to create a
service.
FormHandler has improved – you won’t need FunctionHandler even to edit data.
table format that is an Excel-like viewer for any data.
(Future releases will allow embedding this component into
templates.)pptx format to download as a PPTX?download=filenameCaptureHandler supports Chrome as a backend engine. This allows screenshots that are far more accurate than PhantomJS.
Running gramex setup <directory> lets you set up
apps by running
npm, bower, pip install and any other relevant installations in
the target directory. This can also set up pre-installed apps like
formhandler or capture.
Logging is standardized. All logs are logged to $GRAMEXDATA/logs.
There are 3 types of logs, out-of-box:
logs/gramex.loglogs/requests.csvlogs/user.csvAll logs are auto-rotated weekly by default, and the location and fields can be configured. All logging is now through the standard Python logging mechanism.
Auth handlers can now implement a “Remember me” option when users log in, and set up different session expiry values based on the user’s choice.
LDAPAuth fetches LDAP attributes with direct LDAP login. (Earlier, this was possible only through bind LDAP login.)
DBAuth has an
email_as key that sends forgot password emails from a specific email
ID.
Gramex configurations support conditions. Sections will be included only in specific environments.
YAML imports allow overriding the \$YAMLURL value. This lets you mount applications from any place into any URL. Imports also support lists.
There are several API improvements. The most important are:
gramex.cache.open guesses file type from its extension. So
gramex.cache.open('data.csv') now works – you don’t need to
specify csv as the second parameter.gramex.data.filter updates the meta object to add 2 attribute:
count which reports the number of records matched / updated, and
excluded which reports excluded columnsgramex.services.SMTPMailer supports open email servers without
passwords.For security purposes, Gramex deletes all old session keys without an expiry value. (These originate from Gramex versions prior to Gramex 1.20.)
There are several bug fixes, documentation enhancements and test cases added.
This release adds Windows IntegratedAuth. This allows Windows domain users to log into Gramex automatically without entering and ID or password.
FormHandler has improved - you won’t need FunctionHandler to process data.
table: key. If you specify a simple query here, the results
will be cached based on that queryCaptureHandler
supports a ?debug=1 URL query parameter that logs HTTP responses and
PhantomJS messages to the console. ?debug=2 also logs HTTP requests
made. The Guide also features a live example. CaptureHandler’s
selector parameter is improved and captures portions of a page better.
The default error pages shown for HTTP 500 (Internal Server Error), 404 (Not Found) and 403 (Forbidden) are a little more informattive and better designed.
All auth handlers support a custom session expiry duration. You can increase / decrease the cookie’s expiry duration.
This release also features an undocumented PPTXHandler that generates PowerPoint presentations. But the API will change. This handler not meant for general use yet. A future release will define and document the specs.
There are some enhancements to the API:
gramex.cache.Subprocess returns the stdout and stderr values if
no streams are specifiedgramex.transforms.twitterstream.TwitterStream supports a flush=
option that saves the stream data periodicallygramex.cache.query does not cache queries by default. It caches
only if a state= is specified. (This may change.)gramex.data.filter ignores empty query parameters, which is the
expected behaviourThere are some changes to Gramex behaviour that may impact your application:
name.ext.<time> to
name.<time>.extServer: HTTP header for securityhandler.kwargs attribute that has the
kwargs: configuration passed to the handlerStats:
This is a major release with new functionality. There are two new handlers.
UploadHandler is also improved. Specifically:
All requests are now logged under $GRAMEXDATA/logs/requests.csv,
independent of the console display. This will be used in the next
release to show app usage.
When writing code, there are a few new features:
import: filename.yaml instead of
import: {key: filename.yaml}.handler.args dict that has the URL arguments.
?x=1 sets handlers.args to {'x': ["1"]}. Unlike Tornado’s
.get_arguments(), this supports Unicode keys.handler.argparse(), which lets you convert arguments to the
right type, restrict values and so on.X-HTTP-Method-Override: POST
header or ?x-http-method-override=POST to the URL to convert GET
to POST.gramex.data.filter lets you filter DataFrames using URL
arguments. This is the powerful filtering mechanism behind
FormHandler.gramex.data.download helps create downloadable CSV, XLSX, JSON or
HTML files from one or more DataFrames.gramex.cache.Subprocess. This is
an async method and does not block other requests.gramex.conf.variables.GRAMEXPATH can be used to identify the PATH
where Gramex source libraries are located.Documentation is also improved to cover:
There are a number of bugfixes on this release. The most important are:
gramex.cache.open returns separate results for different
transformslog: configuration has an error, Gramex does not stop
workingThere is one deprecation this release. handler.kwargs is now
handler.conf.kwargs. (This is a largely unused feature of Gramex.)
UPDATE: this was re-introduced in 1.22.
This is a major release with some critical enhancements and fixes.
(NOTE: This release supports Python 2, not Python 3 due to a temporary bug.)
Firstly, caching is improved.
gramex.cache.open accepts a transform= parameters that lets you
post-process the returned result.
gramex.cache.open('data.xlsx', 'xlsx', transform=process_data)
ensures that process_data(data) is called only if the data.xlsx
has changed.gramex.cache.open supports a rel=True parameter. If specified,
it loads the file from the path relative to the calling file. So if
module.py calls gramex.cache.open('data.xlsx', 'xlsx', rel=True)
loads data.xlsx in the same directory as module.py, not relative
to gramex.gramex.cache.open supports a 'config' mode that loads YAML
files just like Gramex does – i.e. with environment variables
support, and returning the values as AttrDict instead of dict.Gramex supports inline images in HTML email. This is useful when sending visualizations as images in emails.
There is better support for programmatic authentication.
X-Gramex-Key header lets you override
users by
specifying an encrypted JSON object for the user. (Documentation
pending)password: function in
DBAuth can
now accept a handler object apart from the content (which is the
password)There are a few security enhancements.
delay: in gramex.yaml.The performance of sessions has been improved as well.
Command line usage of Gramex is improved.
gramex --help shows Gramex command line usage. gramex -V shows
the version.Ctrl+B for opening the browser and Ctrl+D for debugging.)url: sections have duplicate keys, and
override one another. This helps when running on shared instances
like uat.gramener.com.There are a couple of obscure fixes to DataHandler.
?city=.posttransform method. It now works for PUT method as
well as POST, but continues to be undocumented.Finally, there are a few documentation updates.
This is a minor enhancement release with
args: and kwargs: in gramex.yaml, you can now
use function: method(arg, arg, key=val, ...).gramex.cache.query caches SQL query resultsgramex.cache.open supports new formats: md for Markdown, xls
or xlsx for Excel, and template for Tornado templates.gramex.cache.opener makes it easier to create callbacks for
gramex.cache.open.gramex.config.CustomEncoder is a custom JSON encoder that encodes
objects that contain DataFrames. This makes it easy to JSON dump
objects that contain DataFrames.This is a minor enhancement release with several critical bugfixes.
conda update conda and then conda update anaconda.pip (no conda).gramex.cache.reload_module(). You can refresh
Python files without restarting Gramex.gramex.cache.open() now supports loading Tornado templates, apart
from various data / text files. It also supports loading the same
file via multiple callbacks (e.g. loading a CSV file as csv and
text.)allow: and ignore:. It should have been
checking file paths. As a result, the deploy
yaml was
disabling sub-directories. Also, the deploy
yaml file was
not getting installed. Both are fixed.This is a maintenance release with a few minor enhancements:
A series of important bugfixes are addressed:
tornado.routing.Router Class
instead of handlers. This requires an alternate way of clearing
existing handlers.--upgrade when running pip install.This version has a breaking change. The default login URL is /login/
instead of /login. This makes it easier to create custom login pages
using FileHandler (e.g. /login/index.html). If your application
breaks, in your gramex.yaml app: section, add login_url: /login to
revert the change.
?filename= parameter to specify a download filenamelogin_url: key.?next= by defaultgramex.cache.open() methodhandlers: section percolates to other
handlers?q= parameter that searches all text columns?format=xlsx to download as Excel. In QueryHandler,
multiple SQL queries translate to multiple sheetsgramex.transforms.flattener transform that flattens JSON
hierarchies based on a custom field mappinggramex.init() supports a force_reload=True that reloads services.
To support this, gramex.transforms.build_transform is no longer
cached.?count=1 parameter and returns an X-Count HTTP header that has
the number of rows in the query (ignoring limit/offset).xsrf_cookies: false to disable XSRF
cookies for a specific handler.template: "*.html" to
FileHandler kwargs
to render all HTML files as Tornado templates. template: true
renders all files as templates.redirect: config (used by many handlers) to redirect
the user after the process is executed.thread: false. This switches to a synchronous version
that is (currently) less buggy.POST and other HTTP methods via the methods:
configuration. POST is now available by default.cache: key supports user attributes. You can cache responses
based on the user.gramex install databrowser and
then gramex run databrowser to run it at any time.gramex setup can be run in any directory to run the
apps setup. It runs
setup.sh, setup.py, Makefile, npm install, bower install,
etc.requirements.txt, the
apps setup also runs
pip install -r requirements.txt.template: config is now optional for
LDAPAuth and
DBAuth. A
built-in (but minimal) login screen is available by default.redirect: config (used by many handlers) supports relative
URLs.gramex.yaml changes. This speeds up DataHandler considerably.set_xsrf: true to the kwargs: of any URL handler.
This sets the XSRF cookie when the URL is loaded.gramex.yaml has duplicate keys, Gramex raises an error, warning
you up-front.handlers.BaseHandler.log.format config lets you define the
application log format. The default value is
'%(status)d %(method)s %(uri)s (%(ip)s) %(duration).1fms %(user)s'.
It can be overridden to use any other format.handler.session is a persistent
dictionary that you can use to store information against that user
session.eventlog: service. It checks the Gramex update
page daily for updates, and
uploads the event log.none pre-defined
log handler is
available. It ignores log events.gramex update <app> re-installs the app.Ctrl+B on the console to start the browser (in case you
forgot --browser.)gramex install <app> <url> to install an app from a folder, git
repo, URL, etc. Apps can define setup scripts (such as bower
install, etc.) which will be executed after the app is installed.
gramex uninstall <app> uninstalls the appgramex run <app>. Local apps are run via
gramex run <app> --target=DIR. Any command line options (e.g.
--listen.port=8888 or --browser=true) will be stored and re-used
with the next gramex run <app>.gramex.debug.timer and gramex.debug.Timer, and
a line profiler decorator gramex.debug.lineprofile. These will
help profile your functions.Ctrl+D on the Gramex console to start the interactive
IPython debugger. This freezes Gramex and lets you run commands
inside Gramex.gramex --debug.exception=true to start the debugger when any
handler encounters an exception.gramex.yaml can use two new variables: $GRAMEXPATH – the path
where Gramex is installed, and $GRAMEXDATA – the path where
Gramex apps are installed by default.import: in gramex.yaml.Ctrl+C will shutdown Gramex gracefully. You no longer need
Ctrl+Break.There are two changes that may disrupt your code:
gramex.yaml, Gramex will no
longer run. Remove or fix them.default FileHandler are cached on the
browser for 1 minute. Press Ctrl+F5 to reload. Override the
default FileHandler to change this behaviour.thread: true configuration runs the
scheduled task in a separate thread.console).access-log writes information logs to a CSV file access.csvwarn-log writes warnings to a CSV file warn.csvthreadpool: service has been added. This is used internally
by services to run code in a separate thread. You can use
threapool.workers to specify the number of concurrent threads that
are allowed.name and conf parameter which
identifies the name and configuration used to create them.AuthHandler falls back to weaker HTTPS certificate
verification –specifically if Google authentication fails due to
older HTTPS certificates on systems.app: section, the browser: key accepts either true or
any URL. If a URL is provided, it opens the browser at that URL on
startup. If true, it opens the browser to the home page of the
application.variables: section) may contain
other variables. For example, you can define a variable HOME in a
config.yaml. This can be re-used in the variables section of an
imported YAML file as $HOME.function: parameter.
For example, VAR: {function: module.fn} will run module.fn() and
assign $VAR the returned value.index_template: key that allows customized directory
listings. It can be any custom-styled HTML file that uses $path
and $body respectively to represent the full path to the directory
and the contents of the directory.stdout and stderr parameters. These can be false to
ignore the output, or set to any file name (to save the output /
errors in that file.) The default for stdout and stderr is
pipe, which sends the output to the browser.variables: section and use it as $VARIABLE
anywhere in the YAML file, its imports or in subsequent layers. They
default to environment variables.$YAMLFILE (current YAML file
name), $YAMLPATH (current YAML directory), and $YAMLURL
(relative URL path from where Gramex is running to current YAML
directory) in your template.app: configuration. So running
gramex --listen.port=8999 from the command line will run Gramex on
port 8999, irrespective of the port configuration.browser: true to automatically start the browser on Gramex
launch. You can also use gramex --browser=true.yield strings that will be immediately written and flushed,
providing a streaming interface.path as an array. The output of these files are
concatenated after transformated.pattern: /abc instead of pattern: /(abc) if
you are mapping a single URL to a single path.function: template in the transforms section. This treats
the file as a tornado template and renders the output.like and notlike operations.gramex.yaml allows you to trigger events when files are changed.DirectoryHandler (both will work)DirectoryHandler:priority:DirectoryHandler transforms files (e.g. converting Markdown or
YAML to HTML)gramex.transforms.badgerfish transform converts YAML to HTMLgramex.handlers.DirectoryHandler)gramex.config and
gramex.confutil