Gramex FAQ

Integration

What databases does Gramex support?

Gramex can read data from all relational and most on-relational databases. See the full list.

In addition, Gramex can read from all databases that support an ODBC interface and are compliant with SQLAlchemy standard.

What input file formats does Gramex support?

Gramex can read from most standard data file formats. See the full list.

Gramex can also read from any file format that has a Python reader library.

What output formats does Gramex support?

Gramex can render data and visuals into the following formats:

What applications does Gramex integrate with out-of-box?

Gramex integrates with social media applications out-of-box. Specifically Twitter, Facebook and Google.

Gramex can also access the data from applications either directly via the database or via a data dump. This includes:

How does Gramex integrate with R?

Gramex leverages R’s analytical capability natively without sacrificing its advantages of speed and out-of-memory processing. This is achieved through:

A typical workflow with R involves:

The process allows independence of development. Developers changing the R scripts need not be aware of the Gramex environment, and vice versa.

Deployment

How are Gramex applications deployed?

Each Gramex instance runs on a single core. and we recommend reserving a core for the server. So if you run 8 Gramex instances, it can be on a single 8-core server, or two 4-core servers, and so on.

Deployment architecture{@width=600}

Gramex applications, or templates, can run on one or more instances of Gramex. Depending on the size of data and number of users, you have 3 options for deploying a new Gramex application:

  1. Run it on an existing Gramex instance. This is preferred for very small applications, with the same audience and security needs as existing applications.
  2. Run it on a new Gramex instance. This is ideal for most applications.
  3. Run it across multiple Gramex instances. This is required for applications with large datasets or large user bases.

Where does it fit in my enterprise’s architecture?

Gramex applications fit alongside your existing BI applications.

BI Ecosystem{@width=600}

Gramex sources data from a variety of sources:

  1. Source systems. If the source transaction system exposes an API to pull data from, and/or is a popular system that Gramex has a connector to (e.g. SAP), data can be accessed directly from the source system.
  2. Third party systems. Data from any external system that provide an API can be imported into Gramex in real-time.
  3. Data warehouse. If your organization has a data warehouse, that is the preferred way of accessing data. This ensures that the data is consistent with the other applications – including BI applications – that are sourcing data from the warehouse.
  4. Ad hoc files. Often, organizations have spreadsheets that contain transaction or reference data that are maintained outside of source systems. These can be accessed by Gramex as well.

The server that hosts Gramex is typically housed in the same network as the systems that it accesses data from.

Gramex also feeds a number of applications. Apart from end users who can consume the visuals directly from its web interface, or via mobile, the output can also be fed into:

  1. Intranet portals. The visualizations can be embedded as part of an existing web application (SharePoint, for example) in a seamless manner
  2. Custom applications. The visuals can be rendered as part of a native mobile application, as part of an Excel model, inside a Flash or a Java application, and so on.
  3. Sharing systems. The output of these visuals can be fed into a distribution system such as email, shared folders, collaboration software, etc.

What environment is required for Gramex applications?

We’ve often found that our clients use one server per department that consumes these visualizations.

In addition to a production server, a test server with a configuration similar to the production server is required. This needs to be live for the duration that the project is under development, but can be reused for other projects later.

While Gramex applications are built, Gramex needs to be installed on the systems the developers are working on. The hardware specifications for development systems are not stringent. Any available system would

Example Hardware Requirements

Here are some examples of hardware requirements.

If you were building just one normal Gramex application, you’d probably need:

If you plan to build eight Gramex applications across 4 departments (two applications per quarter), you’d probably need:

What does the Gramener Architecture look like?

This is a component architecture of Gramex.

Gramex Architecture{@width=600}

How does Gramex load and store data?

Gramex uses has two data sourcing mechanisms:

  1. Querying. If the source system that Gramex gets data from provides a querying mechanism, Gramex can query the system for data as required. This allows the data to reside in the source system, without a permanent copy on Gramex.
  2. Loading. If data volumes are large, or if the source system does not permit querying readily, data can be extracted in bulk and loaded into Gramex.

The choice of method varies based on the use (for example, querying is better for reporting, while loading is better for analytics) as well as the nature of data and source systems.

When data is loaded into Gramex, several options for internal storage are available. Gramex can be configured to use:

  1. An internal in-memory database-like structure (the most common option for small data)
  2. An external database (SQLite, Oracle, SQL Server, etc - the most common option for large data)
  3. An external in-memory database (Redis, for example)
  4. An external file (CSV, XML, HDF5, etc)

How does Gramex transform data?

When data is queried, most transformations and processing happen on the source system. For example, if the source system is a database, the SQL query handles the bulk of the data transformation. Gramex performs minimal post-processing required for analytics or reporting.

When data is loaded, all transformations and processing happen within Gramex, in-memory. This requires higher processing power and RAM on the system that Gramex is housed in, but reduces the load on the source systems.

How is data backed up?

Gramex applications by themselves do not act as a master for data. It is a processing system. Therefore, there is no need to backup the data in Gramex. Any transaction data, including user-generated comments, etc. are written back to a database that can be backed up as per your enterprises backup policy.

If the Gramex server is destroyed, it can be reinstalled and re-run. The Gramex application will automatically re-calculate all the values and re-creates the visuals.

How is the Gramex source code maintained and updated?

Gramex follows twice a month release cycle . Enterprises that purchase a Gramex license have the option of upgrading to a newer version. While newer versions of Gramex are mostly backward compatible and tested, there may be a cycle of deprecation of features. So we strongly urge a test phase before deploying a new version, to ensure that existing templates are compatible with it.

When developing a new application, using the latest version of Gramex is ideal, since it provides the latest features, and has the benefit of going through a test cycle in any case.

When the application template needs to be modified, a revised set of template files will be deployed first on the test server. Once all tests pass, these are deployed on the production server, overwriting the prior template. This may require restarting the Gramex instance (but not the server itself.)

How much data volume can Gramex handle?

Gramex can theoretically handle an unlimited volume of data when processing is delegated to an external engine. For in-memory processing, Gramex can handle up to 64 TB of data.

Typical Gramex projects involving large scale data process data in the terabyte scale.

How many users can Gramex handle?

Gramex can serve up to 450 requests per second per instance. This may reduce based on the complexity of the response. This can be increased with multiple instances serving the same application.

There are no hard limits to the number of concurrent users an application can serve. The largest volume of requests Gramex has handled was during the 2014 Indian Elections when the application handled 11 million requests over 8 hours with over 400 concurrent requests per second at its peak.

What testing do Gramex applications go through?

Gramener provides a secure testing environment for applications at UAT. Access is provided to authorized client roles for feedback on the application design and development. Gramener’s QA team performs the following types of tests:

Gramex Deployment Manual - Windows Server

Gramex installation will be provided to clients as a single .msi file. The single file will incorporate all the required installation modules for Gramex and the custom Solution.

  1. Administrative permission is required to install the package.

  2. Gramex can start/stop/restart via Windows Event Service. The events are added to the list during the Gramex installation.

  3. The default installation folder can be changed to meet the client need.
  4. Gramex upgrades are provided to client via a new .msi file. The new file has to be swapped with the prior version. The current solution will unavailable during the upgrade process. The installation will result in a downtime for the current solution.
  5. Custom packages are upgraded and managed via versioning of .msi file.
  6. Changing server resources will not have any impact as long as there it does not result in downtime of the solution.
  7. When there is a server downtime the solution will not be available for use.

Security

Does Gramex have a Policy that prohibits sharing of individual accounts and passwords?

Gramex can restrict the number of concurrent sessions. When a new session is created, all other sessions created for the user can be disallowed.

How does Gramex encrypt password?

By default passwords are encrypted via SHA-256. Gramex can be configured to choose from a number of algorithms including - DSA, RSA, DCDSA, SHA-512, AES, Blowfish etc;

What password policies does Gramex support?

Gramex supports:

Does Gramex use HTTP POST method when transmitting authentication credentials over HTTP?

Yes. All authentication requests are transmitted via HTTP POST by default.

Does Gramex enforce enabling SSL on landing pages that have authentication function?

Yes. Some clients have requested a deployment on non-SSL-enabled environments. In such cases, the platform also provides a flexibility to disable SSL enforcement if required.

How does Gramex defend from Cross-site Request Forgery (CSRF) attacks?

Gramex requires all form requests to hold an XSRF token generated by the server. This token is validated by the server before performing any operation.

How does Gramex defend from Cross-site Scripting (XSS) attacks.?

Gramex sets an X-XSS-Protection: 1 HTTP header by default. Browsers prevent cross-site scripting when this header is encountered.

How does Gramex defend from Cross-frame Scripting (XFS) attacks?

Gramex sets an X-Frame-Options: SAMEORIGIN HTTP header by default. This prevents Gramex content from being placed inside another frame.

How does Gramex defend from header injection attacks (e.g. CRLF)?

Gramex disallows any HTTP header with invalid values, specifically characters in the range of 00 to 1F (hex), which includes CR and LF.

How does Gramex defend from URL redirection attacks?

Gramex disables external redirects by default. By default, applications may only redirect within the same host. This prevents malicious hosts from taking control.

How does Gramex defend against brute force login attacks?

Gramex slows down its responses on repeated login failures. This limits the number of password guesses that can be made. The rate of slowdown is configurable.

How does Gramex defend from directory traversal attacks?

Gramex normalizes file access requests and rejects user access to any file outside the sandboxed deployment environment. Files outside the application folder cannot be accessed by the user.

Does Gramex expose any private object references (e.g. primary keys)?

Gramex does not expose any primary keys by default. Applications can be configured to expose non-sensitive primary keys.

How does Gramex manage Digital certificates and their authentication?

Gramex secures access to digital certificates using OS-level encryption / access restrictions. For example, on Linux, digital certificates are stored in a secure directory with access restricted to applications using ACL via AppArmor or similar security infrastructure.

How does Gramex handle Encryption keys. Elaborate on creation, regeneration, transmit mechanism?

Gramex uses encryption keys for cookies, HTTPS certificates, etc. These are generated by administrators outside of Gramex. They are stored outside the Gramex environment, e.g. as environment variables, secured files, etc. Gramex uses these keys to encrypt / decrypt content that is typically transmitted over HTTPS / SSL.

How does Gramex handle network (data-in-transit) security and storage (data-at-rest) security?

Gramener applications delegates the encryption and decryption of data to specialized algorithms suited for this purpose. A typical application environment would have the following setup:

What is Gramex’s logging mechanism? What information is captured? Who can access the logs?

Gramex logs the following information on any access:

This information is accessible to administrators of the system Gramex is deployed on.

Logs are rotated at a configurable interval, e.g. hourly, daily, weekly etc. The duration of log retention is configurable and only limited by system disk space. These can be backed up / archived using a standard file backup mechanism.

Does Gramex log any confidential and sensitive personal information like government issued IDs, passwords etc

The fields mentioned in above question are logged, no additional information is logged. If any of the logged fields are deemed sensitive, they can be configured to remove from the log.

What anti-virus protections does Gramex provide or use?

Gramex relies on anti-virus software deployed on the underlying OS. It is compatible with all anti-virus software, and can be deployed with the antivirus choice of clients.

Describe the session management approach

Gramex creates a time-bound encrypted session token on any request. Tokens are encrypted using SHA-256. The encrypted tokens are stored on the client side as expiring cookies on the browser with the HttpOnly and Secure flags enabled, and restricted by domain and path as required. On the server-side, session tokens are stored in database with protection configurable by the administrator.

The expiry duration is controlled by the application. Cookies are validated for expiry by the server on every request. Expired cookies are purged. The user is presented with a new session with no credentials associated with it.

What authentication mechanisms does Gramex support?

Gramex supports simple authentication and Google authentication.

Gramex Enterprise supports a built-in ID & password-based authentication mechanism. It also integrates with these federated authentication mechanisms:

Does Gramex support two-step registration process or two-step authentication?

No. Gramex does not support two-factor authentication natively built-in. However, Gramex works with single-sign-on (SSO) engines that support two-factor authentication using protocols such as LDAP, OAuth 2.0 and SAML. 2FA based on SMS verification can be customized if required

Does Gramex support automatic login function (remember me function to store credentials)?

Yes. Gramex stores user login information against a secure session cookie with a configurable expiration time.

How does Gramex restrict access based on business purpose? Does Gramex support authorization control for functions or services?**

Yes. Gramex controls access to all resources via a configuration editable by the administrator. The configuration defines the conditions under which a user can access the resource, e.g.

In addition, Gramex applications have the ability to expose different content based on the user’s role. For example, managers may be able to view the total sales of their entire team, while an agent may only be able to view their own sales.

What are the role-based access controls that are supported by Gramex? Are there any default roles or default permissions?

Gramex is used to build applications that allow any kind of role-based access control mechanisms to be implemented. There are no pre-defined roles.

How are Gramex and Gramex applications audited for security compliance?

The security page explains how Gramex is audited for security.

Gramener runs industry standard security audit tools such as OWASP Zed Attack Proxy on applications before deployment. Clients may run their security tests on the application as well prior to deployment. Application are deployed only after relevant security issues are resolved.

What open source libraries and licenses does Gramex use?

Gramex uses open source components that are licensed under one of these licenses:

The full list of libraries is here.

Development

Who develops Gramex?

Gramex is developed by the Gramener Platform Team, informally called the “CTO Team”. The majority of the contributions to Gramex are from this team.

What does the Gramex team work on?

  1. Security: Updating dependencies and fixing vulnerabilities quickly. For example:
  2. Dependencies. Keeping code working even when dependencies change. For example:
  3. Bug Fixes. Making sure code works as expected. For example:
  4. Adding features. Helping developers by standardizing commonly used code.
  5. Supporting developers. Answering questions to speed up others’ work. For example:

How is Gramex funded?

Gramex is funded by the license fee on the Enterprise Edition and support fees.