1 of 100

v7.2.4

Overview

HEAVY.AI is an analytics platform designed to handle very large datasets. It leverages the processing power of GPUs alongside traditional CPUs to achieve very high performance. HEAVY.AI combines an open-source SQL engine (HeavyDB), server-side rendering (HeavyRender), and web-based data visualization (Heavy Immerse) to provide a comprehensive platform for data analysis.

HeavyDB

The foundation of the platform is HeavyDB, an open-source, GPU-accelerated database. HeavyDB harnesses GPU processing power and returns SQL query results in milliseconds, even on tables with billions of rows. HeavyDB delivers high performance with rapid query compilation, query vectorization, and advanced memory management.

Native SQL

With native SQL support, HeavyDB returns query results hundreds of times faster than CPU-only analytical database platforms. Use your existing SQL knowledge to query data. You can use the standalone SQL engine with the command line, or the SQL editor that is part of the Heavy Immerse visual analytics interface. Your SQL query results can output to Heavy Immerse or to third-party software such as Birst, Power BI, Qlik, or Tableau.

Geospatial Data

HeavyDB can store and query data using native Open Geospatial Consortium (OGC) types, including POINT, LINESTRING, POLYGON, and MULTIPOLYGON. With geo type support, you can query geo data at scale using special geospatial functions. Using the power of GPU processing, you can quickly and interactively calculate distances between two points and intersections between objects.

Open Source

HeavyDB is open source and encourages contribution and innovation from a global community of users. It is available on Github under the Apache 2.0 license, along with components like a Python interface (heavyai) and JavaScript infrastructure (mapd-connector, mapd-charting), making HEAVY.AI the leader in open-source analytics.

HeavyRender

HeavyRender works on the server side, using GPU buffer caching, graphics APIs, and a Vega-based interface to generate custom pointmaps, heatmaps, choropleths, scatterplots, and other visualizations. HEAVY.AI enables data exploration by creating and sending lightweight PNG images to the web browser, avoiding high-volume data transfers. Fast SQL queries make metadata in the visualizations appear as if the data exists on the browser side.

Network bandwidth is a bottleneck for complex chart data, so HEAVY.AI uses in-situ rendering of on-GPU query results to accelerate visual rendering. This differentiates HEAVY.AI from systems that execute queries quickly but then transfer the results to the client for rendering, which slows performance.

Geospatial Analysis

Efficient geospatial analysis requires fast data-rendering of complex shapes on a map. HEAVY.AI can import and display millions of lines or polygons on a geo chart with minimal lag time. Server-side rendering technology prevents slowdowns associated with transferring data over the network to the client. You can select location shapes down to a local level, like census tracts or building footprints, and cross-filter interactively.

Visualize with Vega

Complex server-side visualizations are specified using an adaptation of the Vega Visualization Grammar. Heavy Immerse generates Vega rendering specifications behind the scenes; however, you can also generate custom visualizations using the same API. This customizable visualization system combines the agility of a lightweight frontend with the power of a GPU engine.

Heavy Immerse

Heavy Immerse is a web-based data visualization interface that uses HeavyDB and HeavyRender for visual interaction. Intuitive and easy to use, Heavy Immerse provides standard visualizations, such as line, bar, and pie charts, as well as complex data visualizations, such as geo point maps, geo heat maps, choropleths, and scatter plots. Heavy Immerse provides quick insights and makes them easy to recognize.

Dashboards

Use dashboards to create and organize your charts. Dashboards automatically cross-filter when interacting with data, and refresh with zero latency. You can create dashboards and interact with conventional charts and data tables, as well as scatterplots and geo charts created by HeavyRender. You can also create your own queries in the SQL editor.

Charts

Heavy Immerse lets you create a variety of different chart types. You can display pointmaps, heatmaps, and choropleths alongside non-geographic charts, graphs, and tables. When you zoom into any map, visualizations refresh immediately to show data filtered by that geographic context. Multiple sources of geographic data can be rendered as different layers on the same map, making it easy to find the spatial relationships between them.

Create geo charts with multiple layers of data to visualize the relationship between factors within a geographic area. Each layer represents a distinct metric overlaid on the same map. Those different metrics can come from the same or a different underlying dataset. You can manipulate the layers in various ways, including reorder, show or hide, adjust opacity, or add or remove legends.

Use Multiple Sources

Heavy Immerse can visually display dozens of datasets in the same dashboard, allowing you to find multi-factor relationships that you might not otherwise consider. Each chart (or groups of charts) in a dashboard can point to a different table, and filters are applied at the dataset level. Multisource dashboards make it easier to quickly compare across datasets, without merging the underlying tables.

Streaming Data

Heavy Immerse is ideal for high-velocity data that is constantly streaming; for example, sensor, clickstream, telematics, or network data. You can see the latest data to spot anomalies and trend variances rapidly. Immerse auto-refresh automatically updates dashboards at flexible intervals that you can tailor to your use case.

Ready to Get Started?

I want to...

See...

Install HEAVY.AI

Upgrade to the latest version

Configure HEAVY.AI

See some tutorials and demos to help get up and running

Learn more about charts in Heavy Immerse

Use HEAVY.AI in the cloud

See what APIs work with HEAVY.AI

Learn about features and resolved issues for each release

Know what issues and limitations to look out for

See answers to frequently asked questions

Release Notes

Release notes for currently supported releases

Use of HEAVY.AI is subject to the terms of the HEAVY.AI End User License Agreement (EULA).

The latest release of HEAVY.AI is 7.2.4.

Currently Supported Releases

7.2.3 | 7.2.2 | 7.2.1 | 7.2.0 | 7.1.2 | 7.1.1 | 7.1.0 | 7.0.2 | 7.0.1 | 7.0.0 | 6.4.4 | 6.4.3 | 6.4.2 | 6.4.1 | 6.4.0 | 6.2.7 | 6.2.5 | 6.2.4 | 6.2.1 | 6.2.0

For release notes for releases that are no longer supported, as well as links to documentation for those releases, see Archived Release Notes.

As with any software upgrade, it is important to back up your data before you upgrade HEAVY.AI. In addition, we recommend testing new releases before deploying in a production environment.

For assistance during the upgrade process, contact HEAVY.AI Support by logging a request through the HEAVY.AI Support Portal.

Release 7.x.x - Important Information

IMPORTANT - In HeavyDB Release 7.x.x, the “render groups” mechanism, part of the previous implementation of polygon rendering, has been removed. When you upgrade to HeavyDB Release 7.x.x, all existing tables that have a POLYGON or MULTIPOLYGON geo column are automatically migrated to remove a hidden column containing "render groups" metadata.

This operation is performed on all tables in all catalogs at first startup, and the results are recorded in the INFO log.

Once a table has been migrated in this manner, it is not backwards-compatible with earlier versions of HeavyDB. If you revert to an earlier version, the table may appear to have missing columns and behavior will be undefined. Attempting to query or render the POLYGON or MULTIPOLYGON data with the earlier version may fail or cause a server crash.

As always, HEAVY.AI strongly recommends that all databases be backed up, or at the very least, dumps are made of tables with POLYGON or MULTIPOLYGON columns using the existing HeavyDB version, before upgrading to HeavyDB Release 7.x.x.

Dumps of POLYGON and MULTIPOLYGON tables made with earlier versions can still be restored into HeavyDB Release 7.x.x. The superfluous metadata is automatically discarded. However, dumps of POLYGON and MULTIPOLYGON tables made with HeavyDB Release 7.x.x are not backwards-compatible with earlier versions.

This applies only to tables with POLYGON or MULTIPOLYGON columns. Tables that contain other geo column types (POINT, LINESTRING, etc.), or only non-geo column types, do not require migration and remain backwards-compatible with earlier relea

For Ubuntu installations, install libncurses5 with the following command:

sudo apt install libncurses5

Release 7.2.4 - March 20, 2024

HeavyDB - Fixed Issues

Adds a new option for enabling or disabling the use of virtual addressing when accessing an S3 compatible endpoint for import or HeavyConnect.
Improves logging related to system locks.

Heavy Immerse - Fixed Issues

Fixes issue with SAML authentication.

Release 7.2.3 - February 5, 2024

HeavyDB - New Features and Improvements

Improves performance of foreign tables that are backed by Parquet files in AWS S3.
Improves logging related to GPU memory allocations and data transfers.

HeavyDB - Fixed Issues

Fixes a crash that could occur for certain query patterns with intermediate geometry projections.
Fixes a crash that could occur for certain query patterns containing IN operators with string function operands.
Fixes a crash that could occur for equi join queries that use functions as operands.
Fixes an intermittent error that could occur in distributed configurations when executing count distinct queries.
Fixes an issue where certain query patterns with LIMIT and OFFSET clauses could return wrong results.
Fixes a crash that could occur for certain query patterns with left joins on Common Table Expressions.
Fixes a crash that could occur for certain queries with window functions containing repeated window frames.

Heavy Render - Fixed Issues

Fix several crashes that could occur during out-of-gpu memory error recovery

Heavy Immerse - Fixed Issues

Fixed dashboard load error when switching tabs.
Fixed table reference in size measure of a client-side join data source for linemap chart.
Fixed client-side join name reference.

Release 7.2.2 - December 15, 2023

HeavyDB - New Features and Improvements

Adds support for output/result set buffer allocations via the "cpu-buffer-mem-bytes" configured CPU memory buffer pool. This feature can be enabled using the "use-cpu-mem-pool-for-output-buffers" server configuration parameter.
Adds a "ndv-group-estimator-multiplier" server configuration parameter that determines how the number of unique groups are estimated for specific query patterns.
Adds "default-cpu-slab-size" and "default-gpu-slab-size" server configuration parameters that are used to determine the default slab allocation size. The default size was previously based on the "max-cpu-slab-size" and "max-gpu-slab-size" configuration parameters.
Improves memory utilization when querying the "dashboards" system table.
Improves memory utilization in certain cases where queries are retried on CPU.
Improves error messages that are returned for some unsupported correlated subquery use cases.

HeavyDB - Fixed Issues

Fixes an issue where allocations could go beyond the configured "cpu-buffer-mem-bytes" value when fetching table chunks.
Fixes a crash that could occur when executing concurrent sort queries.
Fixes a crash that could occur when invalid geometry literals are passed to ST functions.

Heavy Immerse - Fixed Issues

Fix for rendering a gauge chart using a parameterized source (join sources, custom sources).

Release 7.2.1 - December 4, 2023

HeavyDB - New Features and Improvements

Improves instrumentation around Parquet import and HeavyConnect.

HeavyDB - Fixed Issues

Fixes a crash that could occur for join queries that result in many bounding box overlaps.
Fixes a crash that could occur in certain cases for queries containing an IN operator with a subquery parameter.
Fixes an issue where the ST_POINTN function could return wrong results when called with negative indexes.
Fixes an issue where a hang could occur while parsing a complex query.

Heavy Render - Fixed Issues

Fixed error when setting render-mem-bytes greater than 4gb.

Heavy Immerse - Fixed Issues

Clamp contour interval size on the Contour Chart to prevent a modulo operation error.
Filter outlier values in the Contour Chart that skew color range.
Fixed sample ratio query ordering to address a pointmap rendering issue.
Fixed layer naming in the Hide Layer menu.

Release 7.2.0 - November 16, 2023

HeavyDB - New Features and Improvements

Adds support for URL_ENCODE, URL_DECODE, REGEXP_COUNT, and HASH string functions.
Enables log based system tables by default.
Adds support for log based system tables auto refresh behind a flag (Beta).
Improves the pre-flight query row count estimation process for projection queries without filters.
Improves the performance of the LIKE operator.

HeavyDB - Fixed Issues

General

Fixes errors that could occur when the REPLACE clause is applied to SQL DDL commands that do not support it.
Fixes an issue where the HeavyDB startup script could ignore command line arguments in certain cases.
Fixes a crash that could occur when requests were made to the detect_column_types API for Parquet files containing list columns.
Fixes a crash that could occur in heavysql when the \detect command is executed for Parquet files containing string list columns.
Fixes a crash that could occur when attempting to cast to text column types in SELECT queries.
Fixes a crash that could occur in certain cases where window functions were called with literal arguments.
Fixes a crash that could occur when executing the ENCODE_TEXT function on NULL values.
Fixes an issue where queries involving temporary tables could return wrong results due to incorrect cache invalidation.

Geo

Fixes an issue where the ST_Distance function could return wrong results when at least one of its arguments is NULL.
Fixes an issue where the ST_Point function could return wrong results when the "y" argument is NULL.
Fixes an issue where the ST_NPoints function could return wrong results for NULL geometries.
Fixes a crash that could occur when the ST_PointN function is called with out-of-bounds index values.
Fixes an issue where the ST_Intersects and ST_Contains functions could incorrectly result in loop joins based on table order.
Fixes an issue where the ST_Transform function could return wrong results for NULL geometries.
Fixes an error that could occur for tables with polygon columns created from the output of user-defined table functions.

Heavy Immerse - New Features and Improvements

[Beta] Geo Joins - Immerse now supports “contains” and “intersects” conditions for common geometry combinations when creating a join datasource in the no code join editor.
Join datasource crossfilter support: Charts that use single table data sources will now crossfilter and be crossfiltered by charts that use join data sources.
Layer Drawer - In layered map charts immerse now has a quick to access Layer Drawer, which allows for layer toggling, reordering, renaming, opacity, zoom visibility controls.
Zoom to filters - Map charts in immerse now support “zoom to filters” functionality, either on an individual chart layer (via the Layer Drawer) or on the whole chart.
Image support in map rollovers - URLs pointing to images will automatically be rendered as a scaled image, with clickthrough support to the full size image.

Heavy Immerse - Fixed Issues

Choropleth/Line Map join datasource support - Significantly improves performance in Choropleth and Line Map charts when using join data sources. Auto aggregates measures on geometry.
Fixes issue where sql editor will horizontally scroll with long query strings

Release 7.1.2 - October 4, 2023

HeavyDB - New Features and Improvements

Improves how memory is allocated for the APPROX_MEDIAN aggregate function.

HeavyDB - Fixed Issues

Fixes a crash that could occur when the DISTINCT qualifier is specified for aggregate functions that do not support the distinct operation.
Fixes an issue where wrong results could be returned for queries with window functions that return null values.
Fixes a crash that could occur in certain cases where queries have multiple aggregate functions.
Fixes a crash that could occur when tables are created with invalid options.
Fixes a potential data race that could occur when logging cache sizes.

Release 7.1.1 - September 15, 2023

HeavyDB - New Features and Improvements

Adds an EXPLAIN CALCITE DETAILED command that displays more details about referenced columns in the query plan.
Improved logging around system memory utilization for each query.
Adds an option to SQLImporter for disabling logging of connection strings.
Adds a "gpu-code-cache-max-size-in-bytes" server configuration parameter for limiting the amount of memory that can be used by the GPU code cache.
Improves column name representation in Parquet validation error messages.

HeavyDB - Fixed Issues

Fixes a parser error that could occur for queries containing a NOT ILIKE clause.
Fixes a multiplication overflow error that could occur when retrying queries on CPU.
Fixes an issue where table dumps do not preserve quoted column names.
Fixes a "cannot start a transaction within a transaction" error that could occur in certain cases.
Fixes a crash that could occur for certain query patterns involving division by COUNT aggregation functions.
Removes a warning that is displayed on server startup when HeavyIQ is not configured.
Removes spurious warnings for CURSOR type checks when there are both cursor and scalar overloads for a user-defined table function.

Heavy Render - New Features and Improvements

Adds hit testing support for custom measures that reference multiple tables.

Heavy Immerse - Fixed Issues

Fixes SAML authentication regression in 7.1.0
Fixes chart export regression in 7.1.0

Release 7.1.0 - August 22, 2023

HeavyDB - New Features and Improvements

Geospatial

Exposes new geo overlaps function ST_INTERSECTSBOX for very fast bounding box intersection detections.
Adds support for the max_reject COPY FROM option when importing raster files. This ensures that imports from large multi-file raster datasets continue after minor errors, but provides adjustable notification upon major ones.
Adds a new ST_AsBinary (also aliased as ST_AsWKB) function that returns the Well-Known Binary (WKB) representation of geometry values. This highly-efficient format is used by postGIS newer versions of Geopandas.
Adds a new ST_AsText (also aliased AS ST_AsWKT) function that returns the Well-Known Text (WKT) representation of geometry values. This is less efficient than WKB but compatible even with nonspatial databases.
Adds support for loading geometry values using the load_table_binary_arrow Thrift API.
New version of HeavyAI python library with direct Geopandas support.
New version of rbc-project with geo column support allowing extensions which input or output any geometric type.

Core SQL

New JAROWINKLER_SIMILARITY string operator for fuzzy matching between string columns and values. This is a case insensitive measure including edit transitions and (slightly) sensitive to white space.
New LEVENSHTEIN_DISTANCE string operator for fuzzy matching between string columns and values. This is case insensitive and represents the number of edits needed to make two strings identical. An “edit” is defined by either an insertion of a character, a deletion of a character, or a replacement of a character.
Extends the ALTER COLUMN TYPE command to support string dictionary encoding size reduction.
Improves the error message returned when out of bound values are inserted into FLOAT and DOUBLE columns.
Adds a "watchdog-max-projected-rows-per-device" server configuration parameter and query hint that determines the maximum number of rows that can be projected by each GPU and CPU device.
Adds a "preflight-count-query-threshold" server configuration parameter and query hint that determines the threshold at which the preflight count query optimization should be executed.
Optimizes memory utilization for projection queries on instances with multiple GPUs.

Predictive Modeling with HeavyML

Support for PCA models and PCA_PROJECT operator.
Support SHOW MODEL FEATURE DETAILS to show per-feature info for models, including regression coefficients and variable importance scores, if applicable.
Support for TRAIN_FRACTION option to specify proportion of the input data to a CREATE MODEL statement that should be trained on.
Support creation of models with only categorical predictors.
Enable categorical and numeric predictors to be specified in any order for CREATE MODEL statements and subsequent inference operations.
Enable Torch table functions (requires client to specify libtorch.so).
Add tf_torch_raster_object_detect for raster object detections (requires client to specify libtorch.so and provide trained model in torchscript format).

Extensions Framework

Allow Array literals as arguments to scalar UDFs
Support table function (UDTF) output row sizes up to 16 trillion rows
Adds support for Column<TextEncodingNone> and ColumnList<TextEncodingNone> table function inputs and outputs.

Performance Optimizations

SQL projections now are sized per GPU/CPU core instead of globally, meaning that projections are more memory efficient as a function of the number of GPUs/CPU threads used for a query. In particular, this means that various forms of in-situ rendering, for example, non-grouped pointmaps, renders can scale to N GPUs more points or use N GPUs less memory, depending on the configuration.
Better parallelize construction of metadata for subquery results for improved performance
Enables result set caching for queries with LIMIT clauses.
Enables the bounding box intersection optimization for certain spatial join operators and geometry types by default.

HeavyDB - Fixed Issues

Fix potential crash when concatenating strings with the output of a UDF.
Fixes an issue where deleted rows with malformed data can prevent ALTER COLUMN TYPE command execution.
Fixes an error that could occur when parsing odbcinst.ini configuration files containing only one installed driver entry.
Fixes a table data corruption issue that could occur when the server crashes multiple times while executing write queries.
Fixes a crash that could occur when attempting to do a union of a string dictionary encoded text column and a none encoded text column.
Fixes a crash that could occur when the output of a table function is used as an argument to the strtok_to_array function.
Fixes a crash that could occur for queries involving projections of both geometry columns and geometry function expressions.
Fixes an issue where wrong results could be returned when the output of the DATE_TRUNC function is used as an argument to the count distinct function.
Fixes an issue where an error occurs if the COUNT_IF function is used in an arithmetic expression.
Fixes a crash that could occur when the WIDTH_BUCKET function is called with decimal columns.
Fixes an issue where the WIDTH_BUCKET function could return wrong results when called with decimal values close to the upper and lower boundary values.
Fixes a crash that could occur for queries with redundant projection steps in the query plan.

Heavy Render - Fixed Issues

Fixes a crash that could occur on multi-gpu systems while handling an out of GPU memory error.

Heavy Immerse - New Features and Improvements

Zoom to filters, setting map bounding box to extent of current filter set.
Image preview in map chart popups where image URLs are present.

Heavy Immerse - Fixed Issues

Fixed error thrown by choropleth chart on polygon hover.

Release 7.0.2 - June 28, 2023

HeavyDB - New Features and Improvements

Adds support for nested window function expressions.
Adds support for exception propagation from table functions.

HeavyDB - Fixed Issues

Fixes a crash that could occur when accessing 8-bit or 16-bit string dictionary encoded text columns on ODBC backed foreign tables.
Fixes unexpected GPU execution and memory allocations that could occur when executing sort queries with the CPU mode query hint.
Fixes an issue that could occur when inserting empty strings for geometry columns.
Fixes an issue that could occur when out of bounds fragment sizes are specified on table creation.
Fixes an issue where system dashboards could contain unexpected cached data.
Fixes a crash that could occur when executing aggregate functions over the result of join operations on scalar subqueries.
Fixes a server hang that could occur when GPU code compilation errors occur for user-defined table functions.
Fixes a data race that could occur when logging query plan cache size.

Heavy Render - New Features and Improvements

Add support for rendering 1D “terrain” cross-section overlays.
Rewrite 2D cross-section mesh generation as a table function.
Further improvements to system state logging when a render out of memory error occurs, and move it to the ERROR log for guaranteed visibility.
Enable auto-clear-render-mem by default for any render-vega call taking < 10 seconds.

Heavy Render - Fixed Issues

Render requests with 0 width or height could lead to a CHECK failure in encodePNG. Invalid image sizes now throw a non-fatal error during vega parsing.

Heavy Immerse - New Features and Improvements

Visualize terrain at the base of atmospheric cross sections in the Cross Section chart with the new Base Terrain chart layer type.

Heavy Immerse - Fixed Issues

Fixed local timezone issue with Chart Animation using cross filter replay.

Release 7.0.1 - June 8, 2023

HeavyDB - New Features and Improvements

Improves instrumentation around CPU and GPU memory utilization and certain crash scenarios.

HeavyDB - Fixed Issues

Fixes a crash that could occur for GPU executed join queries on dictionary encoded text columns with NULL values.

Heavy Render - New Features and Improvements

Improve instrumentation and logging related to gpu memory utilization, particularly with polygon rendering, as well as command timeout issues

Heavy Render - Fixed Issues

Fix a potential segfault when a Vulkan device lost error occurs

Release 7.0.0 - May 1, 2023

HeavyDB - New Features and Improvements

IMPORTANT - In HeavyDB Release 7.0, the “render groups” mechanism, part of the previous implementation of polygon rendering, has been removed. When you upgrade to HeavyDB Release 7.0, all existing tables that have a POLYGON or MULTIPOLYGON geo column are automatically migrated to remove a hidden column containing "render groups" metadata.

This operation is performed on all tables in all catalogs at first startup, and the results are recorded in the INFO log.

Dumps of POLYGON and MULTIPOLYGON tables made with earlier versions can still be restored into HeavyDB Release 7.0. The superfluous metadata is automatically discarded. However, dumps of POLYGON and MULTIPOLYGON tables made with HeavyDB Release 7.0 are not backwards-compatible with earlier versions.

For Ubuntu installations, install libncurses5 with the following command:

sudo apt install libncurses5

Adds new Executor Resource Manager enabling parallel CPU and CPU-GPU query execution, and support for CPU execution on data inputs larger than fit in memory.
Adds HeavyML, a suite of machine learning capabilities accessible directly in SQL, including support for linear regression, random forest, gradient boosted trees, and decision tree regression models, and KMeans and DBScan clustering methods. (BETA)
Adds HeavyConnect support for MULTIPOINT and MULTILINESTRING columns.
Adds ALTER COLUMN TYPE support for text columns.
Adds a REASSIGN ALL OWNED command that allows for object ownership change across all databases.
Adds an option for validating POLYGON and MULTIPOLYGON columns when importing using the COPY FROM command or when using HeavyConnect.
Adds support for CONDITIONAL_CHANGE_EVENT window function.
Adds support for automatic casting of table function CURSOR arguments.
Adds support for Column<GeoMultiPolygon>, Column<GeoMultiLineString>, and Column<GeoMultiPoint> table function inputs and outputs.
Adds support for none encoded text column, geometry column, and array column projections from the right table in left join queries.
Adds support for literal text scalar subqueries.
Adds support for ST_X and ST_Y function output cast to text.
Improves concurrent execution of DDL and SHOW commands.
Improves error messaging for when the storage directory is missing.
Optimizes memory utilization for auto-vacuuming after delete queries.

HeavyDB - Fixed Issues

Fixes an issue where the root user could be deleted in certain cases.
Fixes an issue where staging directories for S3 import could remain when imports failed.
Fixes a crash that could occur when accessing the "tables" system table on instances containing tables with many columns.
Fixes a crash that could occur when accessing CSV and regex parsed file foreign tables that previously errored out during cache recovery.
Fixes an issue where dumping table foreign tables would produce an empty table.
Fixes an intermittent crash that could occur when accessing CSV and regex parsed file foreign tables that are backed by large files.
Fixes a "Ran out of slots in the query output buffer" exception that could occur when using stale cached cardinality values.
Fixes an issue where user defined table functions are erroneously categorized as ambiguous.
Fixes an error that could occur when a group by clause includes an alias that matches a column name.
Fixes a crash that could occur on GPUs with the Pascal architecture when executing join queries with case expression projections.
Fixes a crash that could occur when using the LAG_IN_FRAME window function.
Fixes a crash that could occur when projecting geospatial columns from the tf_raster_contour_polygons table function.
Fixes an issue that could occur when calling window functions on encoded date columns.
Fixes a crash that could occur when the coalesce function is called with geospatial or array columns.
Fixes a crash that could occur when projecting case expressions with geospatial or array columns.
Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
Fixes a crash that could occur in certain cases where left join queries are executed on GPU.
Fixes a crash that could occur for queries with joins on encoded date columns.
Fixes a crash that could occur when using the SAMPLE function on a geospatial column.
Fixes a crash that could occur for table functions with cursor arguments that specify no field type.
Fixes an issue where automatic casting does not work correctly for table function calls with ColumnList input arguments.
Fixes an issue where table function argument types are not correctly inferred when arithmetic operations are applied.
Fixes an intermittent crash that could occur for join queries due to a race condition when changing hash table layouts.
Fixes an out of CPU memory error that could occur when executing a query with a count distinct function call on a high cardinality column.
Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.
Fixes a crash that could occur in certain cases when HeavyConnect is used with Parquet files containing null string values.
Fixes potentially inaccurate calculation of vertical attenuation from antenna patterns in HeavyRF.

Heavy Render - New Features and Improvements

Add support for rendering a 1d cross-section as a line
Package the Vulkan loader libVulkan1 alongside heavydb

Heavy Render - Fixed Issues

Fix a device lost error that could occur with complex polygon renders

Heavy Immerse - New Features and Improvements

Data source Joins as a new custom data source type. (BETA)
Adds improved query performance defaults for the Contour Chart.
Adds access to new control panel to users with role "immerse_control_panel", even if the user is not a superuser.
Adds custom naming of map layers.
Adds custom map layer limit option using flag “ui/max_map_layers” which can be set explicitly (defaults to 8) or to -1 to remove the limit.

Heavy Immerse - Fixed Issues

Renames role from “immerse_trial_mode” to “immerse_export_disabled” and renames corresponding flag from “ui/enable_trial_mode” to “ui/user_export_disabled”.
Various minor UI fixes and polishing.
Fixes an issue where changing parameter value causes Choropleth popup to lose selected popup columns.
Fixes an issue where changing parameter value causes Pointmap to lose selected popup columns.
Fixes an issue where building a Skew-T chart results in a blank browser page.
Fixes an issue where Skew-T chart did not display wind barbs.
Fixes an issue with default date and time formatting.
Fixes an issue where setting flag "ui/enable_map_exports" to false unexpectedly disabled table chart export.
Fixes an issue with date filter presets.
Fixes an issue where filters "Does Not Contain" or "Does not equal" did not work on Crosslinked Columns.
Fixes an issue where charts were not redrawing to show the current bounding box filter set by the Linemap chart.

Release 6.4.4 - May 2, 2023

HeavyDB - New Features and Improvements

Adds support for literal text scalar subqueries.

HeavyDB - Fixed Issues

Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
Fixes a crash that could occur for queries with joins on encoded date columns.
Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.

Heavy Immerse - Fixed Issues

Fixed issue where Skew-T chart would not render when nulls were used in selected data.
Fixed issue where wind barbs were not visible on Skew-T chart.

Release 6.4.3 - February 27, 2023

Heavy Immerse - New Features and Improvements

Added feature flag ui/session_create_timeout with a default value of 10000 (10 seconds) for modifying login request timeout.

Release 6.4.2 - February 15, 2023

HeavyDB - New Features and Improvements

Adds the HeavyDB server configuration parameter enable-foreign-table-scheduled-refresh for enabling or disabling automated foreign table scheduled refreshes..

HeavyDB - Fixed Issues

Fixes a crash that could occur when S3 CSV-backed foreign tables with append refreshes are refreshed multiple times.
Fixes a crash that could occur when foreign tables with geospatial columns are refreshed after cache evictions.
Fixes a crash that could occur when querying foreign tables backed by Parquet files with empty row groups.
Fixes an error that could occur when select queries used in ODBC foreign tables reference case sensitive column names.
Fixes a crash that could occur when CSV backed foreign tables with geospatial columns are refreshed without updates to the underlying CSV files.
Fixes a crash that could occur in heavysql when executing the \detect command with geospatial files.
Fixes a casting error that could occur when executing left join queries.
Fixes a crash that could occur when accessing the disk cache on HeavyDB servers with the read-only configuration parameter enabled.
Fixes an error that could occur when executing queries that project geospatial columns.
Fixes a crash that could occur when executing the EXTRACT function with the ISODOW date_part parameter on GPUs.
Fixes an error that could occur when importing CSV or Parquet files with text columns containing more than 32,767 characters into HeavyDB NONE ENCODED text columns.

Heavy Render - Fixed Issues

Fixes a Vulkan Device Lost error that could occur when rendering complex polygon data with thousands of polygons in a single pixel.

Release 6.4.1 - January 30, 2023

HeavyDB - New Features and Improvements

Optimizes result set buffer allocations for CPU group by queries.
Enables trimming of white spaces in quoted fields during CSV file imports, when both the trim_spaces and quoted options are set.

HeavyDB - Fixed Issues

Fixes an error that could occur when importing CSV files with quoted fields that are surrounded by white spaces.
Fixes a crash that could occur when tables are reordered for range join queries.
Fixes a crash that could occur for join queries with intermediate projections.
Fixes a crash that could occur for queries with geospatial join predicate functions that use literal parameters.
Fixes an issue where queries could intermittently and incorrectly return error responses.
Fixes an issue where queries could return incorrect results when filter push-down through joins is enabled.
Fixes a crash that could occur for queries with join predicates that compare string dictionary encoded and nonencoded text columns.
Fixes an issue where hash table optimizations could ignore the max-cacheable-hashtable-size-bytes and hashtable-cache-total-bytes server configuration parameters.
Fixes an issue where sharded table join queries that are executed on multiple GPUs could return incorrect results.
Fixes a crash that could occur when sharded table join queries are executed on multiple GPUs with the from-table-reordering server configuration parameter enabled.

Heavy Immerse - New Features and Improvements

Multilayer support for Contour and Windbarb charts.
Enable Contour charts by default (feature flag: ui/enable_contour_chart).
Support custom SQL measures in Contour charts.
Restrict export from Heavy Immerse by enabling trial mode (feature flag: ui/enable_trial_mode). Trial mode enables a super user to restrict export capabilities for users who have the immerse_trial_mode role.

Heavy Immerse - Fixed Issues

Allow MULTILINESTRING to be used in selectors for Linemap charts.
Allow MULTILINESTRING to be used in Immerse SQL Editor.

Release 6.4.0 - December 16, 2022

This release features general availability of data connectors for PostGreSQL, beta Immerse connectors for Snowflake and Redshift, and SQL support for Google BigQuery and Hive (beta). These managed data connections let you use HEAVY.AI as an acceleration platform wherever your source data may live. Scheduling and automated caching ensure that fast analytics are always running on the latest available data.

Immerse features four new chart types: Contour, Cross-section, Wind barb, and Skew-t. While especially useful for atmospheric and geotechnical data visualization, Contour and Cross-section also have more general application.

Major improvements for time series analysis have been added. This includes an Immerse user interface for time series, and a large number of SQL window function additions and performance enhancements.

The release also includes two major architectural improvements:

The ability to perform cross-database queries, both in SQL and in Immerse, increasing flexibility across the board. For example, you can now easily build an Immerse dashboard showing system usage combined with business data. You might also make a read-only database of data shared across a set of users.
Render queries no longer block other GPU queries. In many use cases, renders can be significantly slower than other common queries. This should result in significant performance gains, particularly in map-heavy dashboards.

HeavyDB - New Features and Improvements

Core SQL

Adds support for cross database SELECT, UPDATE, and DELETE queries.
Support for MODE SQL aggregate.
Add support for strtok_to_array.
Support for ST_NumGeometries().
Support ST_TRANSFORM applied to literal geo types.
Enhanced query tracing ensures all child operations for a query_id are properly logged with that ID.

Data Linking and Import

Adds support for BigQuery and Hive HeavyConnect and import.
Adds support for table restore from S3 archive files.
Improves integer column type detection in Snowflake import/HeavyConnect data preview.
Adds HeavyConnect and import support for Parquet required scalar fields.
Improves import status error message when an invalid request is made.

Table Function Enhancements

Support POINT, LINESTRING, and POLYGON input and output types in table functions.
Support default values for scalar table function arguments.
Add tf_raster_contour table function to generate contours given x, y, and z arguments. This function is exposed in Immerse, but has additional capabilities available in SQL, such as supporting floating point contour intervals.
Return file path and file name from tf_point_cloud_metadata table function.
Previous length limit of 32K characters per values for none-encoded text columns has been lifted, and now none-encoded text values can be up to 2^31 - 1 characters (approximately 2.1billion characters).
Support array column outputs from table functions.
Add TEXT ENCODING DICT and Array<TEXT ENCODING DICT> type support for runtime functions/UDFs.
Allow transient TEXT ENCODING DICT column inputs into table functions.

Window Function Enhancements

Support COUNT_IF function.
Support SUM_IF function.
Support NTH_VALUE window function.
Support NTH_VALUE_IN_FRAME window function.
Support FIRST_VALUE_IN_FRAME and LAST_VALUE_IN_FRAME window functions.
Support CONDITIONAL_TRUE_EVENT.
Support ForwardFill and BackwardFill window functions to fill in missing (null) values based on previous non-null values in window.

HeavyDB - Fixed Issues

Fixes an issue where databases with duplicate names but different capitalization could be created.
Fixes an issue where raster imports could fail due to inconsistent band names.
Fixes an issue that could occur when DUMP/RESTORE commands were executed concurrently.
Fixes an issue where certain session updates do not occur when licenses are updated.
Fixes an issue where import/HeavyConnect data preview could return unsupported decimal types.
Fixes an issue where import/HeavyConnect data preview for PostgreSQL queries involving variable length columns could result in an error.
Fixes an issue where NULL elements in array columns with the NOT NULL constraint were not projected correctly.
Fixes a crash that could occur in certain scenarios where UPDATE and DELETE queries contain subqueries.
Fixes an issue where ingesting ODBC unsigned SQL_BIGINT into HeavyDB BIGINT columns using HeavyConnect or import could result in storage of incorrect data.
Fixes a crash that could occur in distributed configurations, when switching databases and accessing log based system tables with rolled off logs.
Fixes an error that occurred when importing Parquet files that did not contain statistics metadata.
Ensure query hint is propagated to subqueries.
Fix crash that could occur when LAG_IN_FRAME or LEAD_IN_FRAME were missing order-by or frame clause.
Fix bug where LAST_VALUE window function could return wrong results.
Fix issue where “Cannot use fast path for COUNT DISTINCT” could be reported from a count distinct operation.
Various bug fixes for support of VALUES() clause.
Improve handling of generic input expressions for window aggregate functions.
Fix bug where COUNT(*) and COUNT(1) over window frame could cause crash.
Fix wrong coordinate used for origin_y_bin in tf_raster_graph_shortest_slope_weighted_path.
Speed up table function binding in cases with no ColumnList arguments.
Support arrays of transient encoded strings into table functions.

Heavy Render - New Features and Improvements

Render queries no longer block parallel execution queue for other queries.

Heavy Immerse - New Features and Improvements

The Immerse PostgreSQL connector is now generally available, and is joined by public betas of Redshift and Snowflake.
New chart types:
- Contour chart. Contours can be applied to any geo point data, but are especially useful when applied to smoothly-varying pressure and elevation data. They can help reveal general patterns even in noisy primary data. Contours can be based on any point data, including that from regular raster grids like a temperature surface, or from sparse points like LiDAR data.
- Cross-section chart. As the name suggests, this allows a new view on 2.5D or 3D datasets, where a selected data dimension is plotted on the vertical axis for a slice of geographic data. In addition to looking in profile at parts of the atmosphere in weather modeling, this can also be used to look at geological sections below terrain.
- Representing vector force fields takes a step forward with the Wind barb plot. Wind barbs are multidimensional symbols which convey at a glance both strength and direction.
- Skew-T is a highly specialized multidimensional chart used primarily by meteorologists. Skew-Ts are heavily used in weather modeling and can help predict, for example, where thunderstorms or dry lightning are likely to occur.
Initial support for window functions in Immerse, enabling time lag analysis in charts. For example, you can now plot month-over-month or quarter-over-quarter sales or web traffic volume.
For categorical data, in addition to supporting aggregations based on the number of unique values, MODE is now supported. This supports the creation of groups based on the most-common value.

Release 6.2.7 - November 1, 2022

HeavyDB - Fixed Issues

Fixed an issue where a restarted server can potentially deadlock if the first two queries are executed at the same time and use different executors.

Release 6.2.5 - October 26, 2022

HeavyDB - Fixed Issues

Fixed an issue where COUNT DISTINCT or APPROX_COUNT_DISTINCT, when run on a CASE statement that outputs literal strings, could cause a crash.

Release 6.2.4 - October 12, 2022

HeavyDB - Fixed Issues

Fixes a crash when using COUNT() or COUNT(1) with the window function, i.e., COUNT(*) OVER (PARTITION BY x).
Fixes an incorrect result when using a date column as a partition key, like SUM(x) OVER (PARTITION BY DATE_COL).
Improves the performance of window functions when a literal expression is used as one of the input expressions of window functions like LAG(x, 1).
Improves query execution preparation phase by preventing redundant processing of the same nodes, especially when a complex input query is evaluated.
Fixes geometry type checking for range join operator that could cause a crash in some cases.
Resolves a query that may return an incorrect result when it has many projection expressions (for example, more than 50 8-byte output expressions) when using a window function expression.
Fixes an issue where the Resultset recycler ignores the server configuration size metrics.
Fixes a race condition where multiple catalogs could be created on initialization, resulting in possible deadlocks, server hangs, increased memory pressure, and slow performance.

Release 6.2.1 - September 27, 2022

HeavyDB - Fixed Issues

Fixes a crash encountered during some SQL queries when the read-only setting was enabled.
Fixes an issue in tf_raster_graph_shortest_slope_weighted_path table function that would lead some inputs to be incorrectly rejected.

6.2.0 - September 23, 2022

In Release 6.2.0, Heavy Immerse adds animation and a control panel system. HeavyConnect now includes connectors for Redshift, Snowflake, and PostGIS. The SQL system is extended with support for casting and time-based window functions. GeoSQL gets direct LiDAR import, multipoints, and multilinestrings, as well as graph network algorithms. Other enhancements include performance improvements and reduced memory requirements across the product.

HeavyDB - New Features and Improvements

SQL Improvements

TRY_CAST support for string to numeric, timestamp, date, and time casts.
Implicit and explicit CAST support for numeric, timestamp, date, and time to TEXT type.
CAST support from Timestamp(0|3|6|9) types to Time(0) type.
Concat (||) operator now supports multiple nonliteral inputs.
JSON_VALUE operator to extract fields from JSON string columns.
BASE64_ENCODE and BASE64_DECODE operators for BASE64 encoding/decoding of string columns.
POSITION operator to extract index of search string from strings.
Add hash-based count distinct operator to better handle case of sparse columns.

Geospatial

Support MULTILINESTRING OGC geospatial type.
Support MULTIPOINT OGC geospatial type.
Support ST_NumGeometries.
Support ST_ConvexHull and ST_ConcaveHull.
Improved table reordering to maximize invocation of accelerated geo joins.
Support ST_POINT, ST_TRANSFORM and ST_SETSRID as expressions for probing columns in point-to-point distance joins.
Support accelerated overlaps hash join for ST_DWITHIN clause comparing two POINT columns.
Support for POLYGON to MULTIPOLYGON promotion in SQLImporter.

Window Functions

RANGE window function FRAME support for Time, Date, and Timestamp types.
Support LEAD_IN_FRAME / LAG_IN_FRAME window functions that compute LEAD / LAG in reference to a window frame.

Extension Functions

Add TextEncodingNone support for scalar UDF and extension functions.
Support array inputs and outputs to table functions.
Support literal interval types for UDTFs.
Add support for table functions range annotations for literal inputs

Performance and Control

Make max CPU threads configurable via a startup flag.
Support array types for Arrow/select_ipc endpoints.
Add support for query hint to control dynamic watchdog.
Add query hint to control Cuda block and grid size for query.
Adds an echo all option to heavysql that prints all executed commands and queries.
Improved decimal precision error messages during table creation.

HeavyConnect

Add support for file roll offs to HeavyConnect local and S3 file use cases.
Add HeavyConnect support for non-AWS S3-compatible endpoints.

Advanced Analytics

LiDAR

Add tf_point_cloud_metadata table function to read metadata from one or more LiDAR/point cloud files, optionally filtered by a bounding box.
Add tf_load_point_cloud table function to load data from one or more LiDAR/point cloud files, optionally filtered by bounding box and optionally cached in memory for subsequent queries.

Graph and Path Functions

Add tf_graph_shortest_path table function to compute shortest edge-weighted path between two points in a graph constructed from an input edge list
Add tf_graph_shortest_paths_distances table function to compute the shortest edge-weighted distances between a starting point and all other points in a graph constructed from an input edge list.
Add tf_grid_graph_shortest_slope_weighted_path table function to compute the shortest slope-weighted path between two points along rasterized data.

Enhanced Spatial Aggregations

Support configurable aggregation types for tf_geo_rasterize and tf_geo_rasterize_slope table functions, allowing for AVG, MIN, MAX, SUM, and COUNT aggregations.
Support two-pass gaussian blur aggregation post-processing for tf_geo_rasterize and tf_geo_rasterize_slope table functions.

RF Propagation Extension Improvements

Add dynamic ray splitting to tf_rf_prop_max_signal table function for improved performance and terrain coverage.
Add variant of tf_rf_prop_max_signal table function that takes per-RF source/tower transmission power (watts) and frequency (MHz).
Add variant of generate_series table function that generates series of timestamps between a start and end timestamp at specified time intervals.

Fixed Issues

ST_Centroid now automatically picks up SRID of underlying geometry.
Fixed a crash that occurred when ST_DISTANCE had an ST_POINT input for its hash table probe column.
Fixed an issue where a query hint would not propagate to a subquery.
Improved overloaded table function type deduction eliminates type mismatches when table function outputs are used downstream.
Properly handle cases of RF sources outside of terrain bounding box for tf_rf_prop_max_signal.
Fixed an issue where specification of unsupported GEOMETRY column type during table creation could lead to a crash.
Fixed a crash that could occur due to execution of concurrent create and drop table commands.
Fixed a crash that could occur when accessing the Dashboards system table.
Fixed a crash that could occur as a result of type mismatches in ITAS queries.
Fixed an issue that could occur due to band name sanitization during raster imports.
Fixed a memory leak that could occur when dropping temporary tables.
Fixed a crash that could occur due to concurrent execution of a select query and long-running write query on the same table.

Heavy Render - New Features and Improvements

Disables render group assignment by default.
Supports rendering of MULTILINESTRING geometries.
Memory footprint required for compositing renders on multi-GPU systems is significantly reduced. Any multi-GPU system will see improvements, but is most noticeable on systems with 4 or more GPUs. For example, rendering a 1400 x 1400 image results in ~450mb of memory saved when using 8 GPUs for a query. Multi-gpu system configurations should be able to set the res-gpu-mem configuration flag value lower as a result, freeing memory for other subsystems.
Adds INFO logging of peak render memory usage for the lifetime of the server process. The render memory logged is peak render query output buffer size (controlled with the render-mem-bytes configuration flag) and peak render buffer usage (controlled with the res-gpu-mem configuration flag). These peaks are logged in the INFO log on server shutdown, when GPU memory is cleared via clear_gpu_memory endpoint, or when a new peak is reached. These logged peaks can be useful to adjust the render-mem-bytes and res-gpu-mem configuration flags to improve memory utilization by avoiding reserving memory that might go unused. Examples of the log messages:
- When a new peak render-mem-bytes is reached: New peak render buffer usage (render-mem-bytes):37206200 of 1000000000
- When a new peak res-gpu-mem is reached: New peak render memory usage (res-gpu-mem): 166033024
- Peaks logged on server shutdown or on clear_gpu_memory: Render memory peak utilization:
  Query result buffer (render-mem-bytes): 37206200 of 1000000000 Images and buffers (res-gpu_mem): 660330240 Total allocated: 1660330240

Heavy Render - Fixed Issues

Fixed an issue the occurred when trying to hit-test a multiline SQL expression.

Heavy Immerse - New Features and Improvements

Dashboard and chart image export
Crossfilter replay
Improved popup support in the base 3D chart
New Multilayer CPU rendered Geo charts: Pointmap, Linemap, and Choropleth (Beta)
Control Panel (Beta)
Redshift, Snowflake, and PostGIS HeavyConnect support (Beta)
Skew-T chart (Beta)
Support for limiting the number of charts in a dashboard through the ui/limit_charts_per_dashboard feature flag. The default value is 0 (no limit).

Heavy Immerse - Fixed Issues

Fixed duplicate column names importer error.
Various bug fixes and user-interface improvements.

Installation and Configuration

System Requirements

Hardware Reference

Optimal GPUs on which to run the HEAVY.AI platform include:

NVIDIA Tesla A100
NVIDIA Tesla V100 v2
NVIDIA Tesla V100 v1
NVIDIA Tesla P100
NVIDIA Tesla P40
NVIDIA Testa T4

The following configurations are valid for systems using any of these GPUs as the building blocks of your system. For production systems, use Tesla enterprise-grade cards. Avoid mixing card types in the same system; use a consistent card model across your environment.

Primary factors to consider when choosing GPU cards are:

The amount of GPU RAM available on each card
The number of GPU cores
Memory bandwidth

Newer cards like the Tesla V100 have higher double-precision compute performance, which is important in geospatial analytics. The Tesla V100 models support the NVLink interconnect, which can provide a significant speed increase for some query workloads.

For advice on optimal GPU hardware for your particular use case, ask your HEAVY.AI sales representative.

HeavyDB Architecture

Before considering hardware details, this topic describes the HeavyDB architecture.

HeavyDB is a hybrid compute architecture that utilizes GPU, CPU, and storage. GPU and CPU are the Compute Layer, and SSD storage is the Storage Layer.

When determining the optimal hardware, make sure to consider the storage and compute layers separately.

Loading raw data into HeavyDB ingests data onto disk, so you can load as much data as you have disk space available, allowing some overhead.

When queries are executed, HeavyDB optimizer utilizes GPU RAM first if it is available. You can view GPU RAM as an L1 cache conceptually similar to modern CPU architectures. HeavyDB attempts to cache the hot data. If GPU RAM is unavailable or filled, HeavyDB optimizer utilizes CPU RAM (L2). If both L1 and L2 are filled, query records overflow to disk (L3). To minimize latency, use SSDs for the Storage Layer.

You can run a query on a record set that spans both GPU RAM and CPU RAM as shown in the diagram above, which also shows the relative performance improvement you can expect based on whether the records all fit into L1, a mix of L1 and L2, only L2, or some combination of L1, L2, and L3.

Hot Records and Columns

The server is not limited to any number of hot records. You can store as much data on disk as you want. The system can also store and query records in CPU RAM, but with higher latency. The hot records represent the number of records on which you can perform zero-latency queries.

Projection-only Columns

CPU RAM

The amount of CPU RAM should equal four to eight times the amount of total available GPU memory. Each NVIDIA Tesla P40 has 24 GB of onboard RAM available, so if you determine that your application requires four NVIDIA P40 cards, you need between 4 x 24 GB x 4 (384 GB) and 4 x 24 GB x 8 (768 GB) of CPU RAM. This correlation between GPU RAM and CPU RAM exists because the HeavyDB uses CPU RAM in certain operations for columns that are not filtered or aggregated.

SSD Storage

A HEAVY.AI deployment should be provisioned with enough SSD storage to reliably store the required data on disk, both in compressed format and in HEAVY.AI itself. HEAVY.AI requires 30% overhead beyond compressed data volumes. HEAVY.AI recommends drives such as the Intel® SSD DC S3610 Series, or similar, in any size that meets your requirements.

For maximum ingestion speed, HEAVY.AI recommends ingesting data from files stored on the HEAVY.AI instance.
Most public cloud environments’ default storage is too small for the data volume HEAVY.AI ingests. Estimate your storage requirements and provision accordingly.

Hardware Sizing Schedule

If you already have your data in a database, you can look at the largest fact table, get a count of those records, and compare that with this schedule.

If you have a .csv file, you need to get a count of the number of lines and compare it with this schedule.

CPU Cores

HEAVY.AI uses the CPU in addition to the GPU for some database operations. GPUs are the primary performance driver; CPUs are utilized secondarily. More cores provide better performance but increase the cost. Intel CPUs with 10 cores offer good performance for the price. For example, so you could configure your system with a single NVIDIA P40 GPU and two 10-core CPUs. Similarly, you can configure a server with eight P40s and two 10-core CPUs.

Suggested CPUs:

Intel® Xeon® E5-2650 v3 2.3GHz, 10 cores
Intel® Xeon® E5-2660 v3 2.6GHz, 10 cores
Intel® Xeon® E5-2687 v3 3.1GHz, 10 cores
Intel® Xeon® E5-2667 v3 3.2GHz, 8 cores

PCI Express (PCIe)

GPUs are typically connected to the motherboard using PCIe slots. The PCIe connection is based on the concept of a lane, which is a single-bit, full-duplex, high-speed serial communication channel. The most common numbers of lanes are x4, x8, and x16. The current PCIe 3.0 version with an x16 connection has a bandwidth of 16 GB/s. PCIe 2.0 bandwidth is half the PCIe 3.0 bandwidth, and PCIe 1.0 is half the PCIe 2.0 bandwidth. Use a motherboard that supports the highest bandwidth, preferably, PCIe 3.0. To achieve maximum performance, the GPU and the PCIe controller should have the same version number.

The PCIe specification permits slots with different physical sizes, depending on the number of lanes connected to the slot. For example, a slot with an x1 connection uses a smaller slot, saving space on the motherboard. However, bigger slots can actually have fewer lanes than their physical designation. For example, motherboards can have x16 slots connected to x8, x4, or even x1 lanes. With bigger slots, check to see if their physical sizes correspond to the number of lanes. Additionally, some slots downgrade speeds when lanes are shared. This occurs most commonly on motherboards with two or more x16 slots. Some motherboards have only 16 lanes connecting the first two x16 slots to the PCIe controller. This means that when you install a single GPU, it has the full x16 bandwidth available, but two installed GPUs each have x8 bandwidth.

HEAVY.AI does not recommend adding GPUs to a system that is not certified to support the cards. For example, to run eight GPU cards in a machine, the BIOS register the additional address space required for the number of cards. Other considerations include power routing, power supply rating, and air movement through the chassis and cards for temperature control.

NVLink

NVLink is a bus technology developed by NVIDIA. Compared to PCIe, NVLink offers higher bandwidth between host CPU and GPU and between the GPU processors. NVLink-enabled servers, such as the IBM S822LC Minsky server, can provide up to 160 GB/sec bidirectional bandwidth to the GPUs, a significant increase over PCIe. Because Intel does not currently support NVLink, the technology is available only on IBM Power servers. Servers like the NVIDIA-manufactured DGX-1 offer NVLink between the GPUs but not between the host and the GPUs.

System Examples

A variety of hardware manufacturers make suitable GPU systems. For more information, follow these links to their product specifications.

Software Requirements

Operating Systems
- CentOS/RHEL 7.0 or later
- Ubuntu 20.04 or later

Ubuntu 22.04 is not currently supported.

Additional Components
- OpenJDK version 8 or higher
- EPEL
- wget or curl
- Kernel headers
- Kernel development packages
- log4j 2.15.0 or higher
NVIDIA hardware and software (for GPU installs only)
- Hardware: Ampere, Turing, Volta, or Pascal series GPU cards. HEAVY.AI recommends that each GPU card in a server or distributed environment be of the same series.
- Software:
  - NVIDIA CUDA drivers version 520 and Cuda 11.8 or higher. Run nvidia-smi to determine the currently running driver.
  - Up-to-date Vulkan drivers.
Supported web browsers (Enterprise Edition, Immerse). Latest stable release of:
- Chrome
- FireFox
- Safari version 15.x or higher

Some features in Heavy Immerse are not supported in the Internet Explorer browser due to performance issues in IE. HEAVY.AI recommends that you use a different browser to experience the latest Immerse features.

Installation

The CPU (no GPUs) install does not support backend rendering. For example, Pointmap and Scatterplot charts are not available. The GPU install supports all chart types.

The Open Source options do not require a license, and do not include Heavy Immerse.

Free Version

HEAVY.AI Free is a full-featured version of the HEAVY.AI platform available at no cost for non-hosted commercial use.

To get started with HEAVY.AI Free:

On the Get HEAVY.AI Free page, enter your email address and click I Agree.
Open the HEAVY.AI Free Edition Activation Link email that you receive from HEAVY.AI, and click Click Here to view and download the free edition license. You will need this license to run HEAVY.AI after you install it. A copy of the license is also sent to your email.

Add Users

You can create additional HEAVY.AI users to collaborate with.

Connect to Immerse using a web browser connected to your host machine on port 6273. For example, http://heavyai.mycompany.com:6273.

Installing on CentOS

In this section, you will find recipes to install HEAVY.AI platform and NVIDIA drivers using package manager like yum or tarball.

HEAVY.AI Installation on CentOS/RHEL

This is an end-to-end recipe for installing HEAVY.AI on a CentOS/RHEL 7 machine using CPU and GPU devices.

The order of these instructions is significant. To avoid problems, install each component in the order presented.

Assumptions

These instructions assume the following:

You are installing on a “clean” CentOS/RHEL 7 host machine with only the operating system installed.
Your HEAVY.AI host only runs the daemons and services required to support HEAVY.AI.
Your HEAVY.AI host is connected to the Internet.

Preparation

Prepare your Centos/RHEL machine by updating your system and optionally enabling or configuring a firewall.

Update and Reboot

Update the entire system and reboot the system if needed.

Install the utilities needed to create HEAVY.AI repositories and download archives

JDK

Open a terminal on the host machine.
Install the headless JDK using the following command:

Create the HEAVY.AI User

Create a group called heavyai and a user named heavyai, who will own HEAVY.AI software and data on the file system.

You can create the group, user, and home directory using the useradd command with the --user-group and --create-home switches:

Set a password for the user:

Installation

Install HEAVY.AI using yum or a tarball.

The installation using the yum package manager is recommended to those who want a more automated install and upgrade procedure.

Install Nvidia Drivers ᴳᴾᵁ ᴼᴾᵀᴵᴼᴺ

If your system includes NVIDIA GPUs, but the drivers are not installed, install them now.

Installing with Yum

Create a yum repository depending on the edition (Enterprise, Free, or Open Source) and execution device (GPU or CPU) you are going to use.

Add the GPG-key to the newly added repository.

Use yum to install the latest version of HEAVY.AI.

If you need to install a specific version of HEAVY.AI because you are upgrading from Omnisci, or for different reasons, you must run the following command:

Installing with a Tarball

First create the installation directory.

Download the archive and install the latest version of the software. A different archive is downloaded depending on the edition (Enterprise, Free, or Open Source) and the device used for runtime.

Configuration

Follow these steps to prepare your HEAVY.AI environment.

Set Environment Variables

For your convenience, you can update .bashrc with these environment variables

Although this step is optional, you will find references to the HEAVYAI_BASE and HEAVYAI_PATH variables. These variables contain respectively the paths where configuration, license, and data files are stored and where the software is installed. Installing them is strongly recommended.

Initialization

Run the systemd installer to initialize the HEAVY.AI services and the database storage.

Accept the default values provided or make changes as needed.

The script creates a data directory in $HEAVYAI_BASE/storage (typically /var/lib/heavyai) with the directories catalogs, data, export and log. The directory import is created when you insert data the first time. If you are a HeavyDB administrator, the log directory is of particular interest.

Activation

Note that Heavy Immerse is not available in the OS SEdition, so if running the OSS Edition the systemctl command using the heavy_web_server has no effect.

Enable the automatic startup of the service at reboot and start the HEAVY.AI services.

Configure the Firewall ᴼᴾᵀᴵᴼᴺᴬᴸ

If a firewall is not already installed and you want to harden your system, install and start firewalld.

To use Heavy Immerse or other third-party tools, you must prepare your host machine to accept incoming HTTP(S) connections. Configure your firewall for external access:

Most cloud providers use a different mechanism for firewall configuration. The commands above might not run in cloud deployments.

Licensing HEAVY.AI ᵉᵉ⁻ᶠʳᵉᵉ ᵒⁿˡʸ

Open a terminal window.
Enter cd ~/ to go to your home directory.
Open .bashrc in a text editor. For example, vi .bashrc.
Edit the .bashrc file. Add the following export commands under “User specific aliases and functions.”
Save the .bashrc file. For example, in vi enter[esc]:x!
Open a new terminal window to use your changes.
Connect to Heavy Immerse using a web browser connected to your host machine on port 6273. For example, http://heavyai.mycompany.com:6273.
When prompted, paste your license key in the text box and click Apply.
Log into Heavy Immerse by entering the default username (admin) and password (HyperInteractive), and then click Connect.

The $HEAVYAI_BASE directory must be dedicated to HEAVYAI; do not set it to a directory shared by other packages.

Final Checks

Load Sample Data and Run a Simple Query

HEAVY.AI ships with two sample datasets of airline flight information collected in 2008, and a census of New York City trees. To install sample data, run the following command.

Connect to HeavyDB by entering the following command in a terminal on the host machine (default password is HyperInteractive):

Enter a SQL query such as the following:

The results should be similar to the results below.

After installing Enterprise or Free Edition, check if Heavy Immerse is running as intended.

Connect to Heavy Immerse using a web browser connected to your host machine on port 6273. For example, http://heavyai.mycompany.com:6273.
Log into Heavy Immerse by entering the default username (admin) and password (HyperInteractive), and then click Connect.

Create a new dashboard and a Scatter Plot to verify that backend rendering is working.

Click New Dashboard.
Click Add Chart.
Click SCATTER.
Click Add Data Source.
Choose the flights_2008_10k table as the data source.
Click X Axis +Add Measure.
Choose depdelay.
Click Y Axis +Add Measure.
Choose arrdelay.
Click Size +Add Measure.
Choose airtime.
Click Color +Add Measure.
Choose dest_state.

The resulting chart shows, unsurprisingly, that there is a correlation between departure delay and arrival delay.

Create a new dashboard and a Table chart to verify that Heavy Immerse is working.

Click New Dashboard.
Click Add Chart.
Click Bubble.
Click Select Data Source.
Choose the flights_2008_10k table as the data sour
Click Add Dimension.
Choose carrier_name.
Click Add Measure.
Choose depdelay.
Click Add Measure.
Choose arrdelay.
Click Add Measure.
Choose #Records.

The resulting chart shows, unsurprisingly, that also the average departure delay is correlated to the average of arrival delay, while there is quite a difference between Carriers.

¹ In the OS Edition, Heavy Immerse is unavailable.

² The OS Edition does not require a license key.

Install NVIDIA Drivers and Vulkan on CentOS/RHEL

Install Prerequisites

Install the Extra Packages for Enterprise Linux (EPEL) repository and other packages before installing NVIDIA drivers.

For CentOS, use yum to install the epel-release package.

sudo yum install epel-release

Use the following install command for RHEL.

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

RHEL-based distributions require Dynamic Kernel Module Support (DKMS) to build the GPU driver kernel modules. For more information, see https://fedoraproject.org/wiki/EPEL. Upgrade the kernel and restart the machine.

sudo yum upgrade kernel
sudo reboot now

Install Kernel Headers

Install kernel headers and development packages:

sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r)

If installing kernel headers does not work correctly, follow these steps instead:

Identify the Linux kernel you are using by issuing the uname -r command.
Use the name of the kernel (3.10.0-862.11.6.el7.x86_64 in the following code example) to install kernel headers and development packages:

sudo yum install \
kernel-devel-3.10.0-862.11.6.el7.x86_64 \
kernel-headers-3.10.0-862.11.6.el7.x86_64

Install the dependencies and extra packages:

sudo yum install kernel-devel kernel-headers pciutils dkms

Install NVIDIA Drivers and Vulkan

CUDA is a parallel computing platform and application programming interface (API) model. It uses a CUDA-enabled graphics processing unit (GPU) for general-purpose processing. The CUDA platform provides direct access to the GPU virtual instruction set and parallel computation elements. For more information on CUDA unrelated to installing HEAVY.AI, see https://developer.nvidia.com/cuda-zone. You can install drivers in multiple ways. This section provides installation information using the NVIDIA website or using yum.

Although using the NVIDIA website is more time consuming and less automated, you are assured that the driver is certified for your GPU. Use this method if you are not sure which driver to install. If you prefer a more automated method and are confident that the driver is certified, you can use the package-manager method.

Install NVIDIA Drivers Using the NVIDIA Website

Install the CUDA package for your platform and operating system according to the instructions on the NVIDIA website (https://developer.nvidia.com/cuda-downloads).

If you do not know the GPU model installed on your system, run this command:

lspci -v | egrep "3D|VGA*.NVIDIA" | awk -F '\[|\]' ' { print $2 } '

The output shows the product type, series, and model. In this example, the product type is Tesla, the series is T (as Turing), and the model is T4.

Tesla T4

Select the product type shown after running the command above.
Select the correct product series and model for your installation.
In the Operating System dropdown list, select Linux 64-bit.
In the CUDA Toolkit dropdown list, click a supported version (11.4 or higher).
Click Search.
On the resulting page, verify the download information and click Download.

Please check that the driver's version you are downloading meets the HEAVI.AI minimum requirements.

Move the downloaded file to the server, change the permissions, and run the installation.

chmod +x NVIDIA-Linux-x86_64-*.run
sudo ./NVIDIA-Linux--x86_64-*.runYou might get the following error during installation:

You might receive the following error during installation:

ERROR: The Nouveau kernel driver is currently in use by your system. This driver is incompatible with the NVIDIA driver, and must be disabled before proceeding. Please consult the NVIDIA driver README and your Linux distribution's documentation for details on how to correctly disable the Nouveau kernel driver.

If you receive this error, blacklist the Nouveau driver by editing the /etc/modprobe.d/blacklist-nouveau.conffile, adding the following lines at the end:

blacklist nouveau blacklist lbm-nouveau options nouveau modeset=0 alias nouveau off alias lbm-nouveau off

Install NVIDIA Drivers Using Yum

Install a specific version of the driver for your GPU by installing the NVIDIA repository and using the yum package manager.

When installing the driver, ensure that your GPU model is supported and meets the HEAVI.AI minimum requirements.

Add the NVIDIA network repository to your system.

sudo yum-config-manager --add-repo \
https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

Run the available drivers for the download.

yum list nvidia-driver-branch-[0-9][0-9][0-9].x86_64

Available Packages
nvidia-driver-branch-418.x86_64    3:418.226.00-1.el7     cuda-rhel7-x86_64 
nvidia-driver-branch-440.x86_64    3:440.118.02-1.el7     cuda-rhel7-x86_64 
nvidia-driver-branch-450.x86_64    3:450.191.01-1.el7     cuda-rhel7-x86_64 
nvidia-driver-branch-455.x86_64    3:455.45.01-1.el7      cuda-rhel7-x86_64 
nvidia-driver-branch-460.x86_64    3:460.106.00-1.el7     cuda-rhel7-x86_64 
nvidia-driver-branch-465.x86_64    3:465.19.01-1.el7      cuda-rhel7-x86_64 
nvidia-driver-branch-495.x86_64    3:495.29.05-1.el7      cuda-rhel7-x86_64 
nvidia-driver-branch-510.x86_64    3:510.73.08-1.el7      cuda-rhel7-x86_64 
nvidia-driver-branch-515.x86_64    3:515.43.04-1.el7      cuda-rhel7-x86_64

Install the driver version needed with yum.

sudo yum install nvidia-driver-branch-<version>.x86_64

Reboot your system to ensure that the new version of the driver is loaded.

sudo reboot

Check NVIDIA Driver Installation

Run nvidia-smi to verify that your drivers are installed correctly and recognize the GPUs in your environment. Depending on your environment, you should see something like this to confirm that your NVIDIA GPUs and drivers are present:

If you see an error like the following, the NVIDIA drivers are probably installed incorrectly:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. 
Make sure that the latest NVIDIA driver is installed and running.

Review the Install NVIDIA Drivers section and correct any errors.

Install Vulkan

To work correctly, the back-end renderer requires a Vulkan-enabled driver and the Vulkan library. Without these components, the database cannot start without disabling the back-end renderer.

Install the Vulkan library and its dependencies using yum both CentOS and RHEL.

sudo yum install vulkan

If installing on RHEL, you must obtain and manually install the vulkan-filesystem package manually. Perform these additional steps:

Download the rpm file

wget http://mirror.centos.org/centos/7/os/x86_64/Packages/vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm

Install the rpm file

sudo rpm --install vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm

You might see a warning similar to the following:

warning: cuda-repo-rhel7-10.0.130-1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 7fa2af80: NOKEY

Ignore it now; you can verify NVIDIA driver installation here.

For more information about troubleshooting Vulkan, see the Vulkan Renderer section.

Install CUDA Toolkit ᴼᴾᵀᴵᴼᴺᴬᴸ

You must install the CUDA Toolkit if you use advanced features like C++ User-Defined Functions or User-Defined Table Functions to extend the database capabilities.

Add the NVIDIA network repository to your system:

sudo yum-config-manager --add-repo \
https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

2. List the available CUDA Toolkit versions:

yum list cuda-toolkit-* | egrep -v config

Available Packages
cuda-toolkit-10-0.x86_64                     10.0.130-1        cuda-rhel7-x86_64
cuda-toolkit-10-1.x86_64                     10.1.243-1        cuda-rhel7-x86_64
cuda-toolkit-10-2.x86_64                     10.2.89-1         cuda-rhel7-x86_64
cuda-toolkit-11-0.x86_64                     11.0.3-1          cuda-rhel7-x86_64
cuda-toolkit-11-1.x86_64                     11.1.1-1          cuda-rhel7-x86_64
cuda-toolkit-11-2.x86_64                     11.2.2-1          cuda-rhel7-x86_64
cuda-toolkit-11-3.x86_64                     11.3.1-1          cuda-rhel7-x86_64
cuda-toolkit-11-4.x86_64                     11.4.4-1          cuda-rhel7-x86_64
cuda-toolkit-11-5.x86_64                     11.5.2-1          cuda-rhel7-x86_64
cuda-toolkit-11-6.x86_64                     11.6.2-1          cuda-rhel7-x86_64

3. Install the CUDA Toolkit using yum:

sudo yum install cuda-toolkit-<version>.x86_64

4. Check that everything is working correctly:

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

Installing on Ubuntu

In this section, you will find recipes to install HEAVY.AI platform and NVIDIA drivers using package manager like apt or tarball.

HEAVY.AI Installation on Ubuntu

This is an end-to-end recipe for installing HEAVY.AI on a Ubuntu 20.04 machine using CPU and GPU devices.

The order of these instructions is significant. To avoid problems, install each component in the order presented.

Assumptions

These instructions assume the following:

You are installing on a “clean” Ubuntu 20.04 host machine with only the operating system installed.
Your HEAVY.AI host only runs the daemons and services required to support HEAVY.AI.
Your HEAVY.AI host is connected to the Internet.

Preparation

Prepare your Ubuntu machine by updating your system, creating the HEAVY.AI user (named heavyai), installing kernel headers, installing CUDA drivers, and optionally enabling the firewall.

Update and Reboot

Update the entire system:

2. Install the utilities needed to create Heavy.ai repositories and download archives:

3. Install the headless JDK and the utility apt-transport-https:

4. Reboot to activate the latest kernel:

Create the HEAVY.AI User

Create a group called heavyai and a user named heavyai, who will be the owner of the HEAVY.AI software and data on the filesystem.

Create the group, user, and home directory using the useradd command with the --user-group and --create-home switches.

2. Set a password for the user:

3. Log in with the newly created user:

Installation

Install the HEAVY.AI using APT and a tarball.

The installation using the APT package manager is recommended to those who want a more automated install and upgrade procedure.

Install Nvidia Drivers ᴳᴾᵁ ᴼᴾᵀᴵᴼᴺ

Installing with APT

Download and add a GPG key to APT.

Add a source apt depending on the edition (Enterprise, Free, or Open Source) and execution device (GPU or CPU) you are going to use.

Use apt to install the latest version of HEAVY.AI.

If you need to install a specific version of HEAVY.AI, because you are upgrading from Omnisci or for different reasons, you must run the following command:

Installing with a Tarball

First create the installation directory.

Download the archive and install the software. A different archive is downloaded depending on the Edition (Enterprise, Free, or Open Source) and the device used for runtime (GPU or CPU).

Configuration

Follow these steps to prepare your HEAVY.AI environment.

Set Environment Variables

For convenience, you can update .bashrc with these environment variables

Initialization

Run the systemd installer to create heavyai services, a minimal config file, and initialize the data storage.

Accept the default values provided or make changes as needed.

The script creates a data directory in $HEAVYAI_BASE/storage (default /var/lib/heavyai/storage) with the directories catalogs, data, export and log.The import directory is created when you insert data the first time. If you are HEAVY.AI administrator, the log directory is of particular interest.

Activation

Heavy Immerse is not available in the OSS Edition, so if running the OSS Edition the systemctl command using the heavy_web_server has no effect.

Enable the automatic startup of the service at reboot and start the HEAVY.AI services.

Configure Firewall ᴼᴾᵀᴵᴼᴺᴬᴸ

If a firewall is not already installed and you want to harden your system, install theufw.

To use Heavy Immerse or other third-party tools, you must prepare your host machine to accept incoming HTTP(S) connections. Configure your firewall for external access.

Most cloud providers use a different mechanism for firewall configuration. The commands above might not run in cloud deployments.

Licensing HEAVY.AI ᵉᵉ⁻ᶠʳᵉᵉ ᵒⁿˡʸ

If you are using Enterprise or Free Edition, you need to validate your HEAVY.AI instance with your license key.

Copy your license key of Enterprise or Free Edition from the registration email message. If you do not have a license and you want to evaluate HEAVI.AI in an unlimited
Connect to Heavy Immerse using a web browser connected to your host machine on port 6273. For example, http://heavyai.mycompany.com:6273.
When prompted, paste your license key in the text box and click Apply.
Log into Heavy Immerse by entering the default username (admin) and password (HyperInteractive), and then click Connect.
.

Final Checks

Load Sample Data and Run a Simple Query

HEAVY.AI ships with two sample datasets of airline flight information collected in 2008, and a census of New York City trees. To install sample data, run the following command.

Connect to HeavyDB by entering the following command in a terminal on the host machine (default password is HyperInteractive):

Enter a SQL query such as the following

The results should be similar to the results below.

After installing Enterprise or Free Edition, check if Heavy Immerse is running as intended.

Connect to Heavy Immerse using a web browser connected to your host machine on port 6273. For example, http://heavyai.mycompany.com:6273.
Log into Heavy Immerse by entering the default username (admin) and password (HyperInteractive), and then click Connect.

Create a new dashboard and a Scatter Plot to verify that backend rendering is working.

Click New Dashboard.
Click Add Chart.
Click SCATTER.
Click Add Data Source.
Choose the flights_2008_10k table as the data source.
Click X Axis +Add Measure.
Choose depdelay.
Click Y Axis +Add Measure.
Choose arrdelay.
Click Size +Add Measure.
Choose airtime.
Click Color +Add Measure.
Choose dest_state.

The resulting chart shows, unsurprisingly, that there is a correlation between departure delay and arrival delay.

Create a new dashboard and a Table chart to verify that Heavy Immerse is working.

Click New Dashboard.
Click Add Chart.
Click Bubble.
Click Select Data Source.
Choose the flights_2008_10k table as the data source.
Click Add Dimension.
Choose carrier_name.
Click Add Measure.
Choose depdelay.
Click Add Measure.
Choose arrdelay.
Click Add Measure.
Choose #Records.

The resulting chart shows, unsurprisingly, that also the average departure delay is correlated to the average of arrival delay, while there is quite a difference between Carriers.

¹ In the OS Edition, Heavy Immerse is unavailable.

² The OS Edition does not require a license key.

Install NVIDIA Drivers and Vulkan on Ubuntu

Installation Prerequisites

Upgrade the system and the kernel, then the machine if needed.

Install Kernel Headers

Install kernel headers and development packages.

Install the extra packages.

Installing Vulkan Library

The rendering engine of HEAVY.AI (present in Enterprise Editions) requires a Vulkan-enabled driver and the Vulkan library. Without these components, the database itself may not be able to start.

Install the Vulkan library and its dependencies using apt.

Installing NVIDIA Drivers

Installing NVIDIA drivers with support for the CUDA platform is required to run GPU-enabled versions of HEAVY.AI.

You can install NVIDIA drivers in multiple ways, we've outlined three available options below. If you would prefer not to decide, we recommend Option 1.

It is advisable to keep a record of the installation method used, as upgrading NVIDIA drivers will require the utilization of the same method for successful results.

What is CUDA? What is the CUDA toolkit?

The CUDA Toolkit from NVIDIA provides everything you need to develop GPU-accelerated applications. The CUDA Toolkit includes GPU-accelerated libraries, a compiler, development tools and the CUDA runtime. The CUDA Toolkit is not required to run HEAVY.AI, but you must install the CUDA toolkit if you use advanced features like C++ User-Defined Functions and or User-Defined Table Functions to extend the database capabilities.

Option 1: Install NVIDIA Drivers with CUDA Toolkit from NVIDIA Website

The minimum CUDA version supported by HEAVY.AI is 11.4. We recommend using a release that has been available for at least two months.

In the "Target Platform" section, follow these steps:

For "Operating System" select Linux
For Architecture" select x86_64
For "Distribution" select Ubuntu
For "Version" select the version of your operating system (20.04)
For "Installer Type" choose deb (network) **
One by one, run the presented commands in the Installer Instructions section on your server.

** You may optionally use any of the "Installer Type" options available.

If you choose to use the .run file option, prior to running the installer you will need to manually install build-essentials using apt and change permissions of the downloaded .run file to allow execution.

Option 2: Install NVIDIA Drivers via .run file using the NVIDIA Website

If you don't know the exact GPU model in your system run this command

You'll get an output in the format Product Type, Series and Model

In this example, the Product type is Tesla the Series is T (as Turing), and the model is T4.

Select the Product Type as the one you got with the command.
Select the correct Product Series and Product Type for your installation.
In the Operating System dropdown list, select Linux 64-bit.
In the CUDA Toolkit dropdown list, click a supported version (11.4 or higher).
Click Search.
On the resulting page, verify the download information and click Download
On the subsequent page, if you agree to the terms, right click on "Agree and Download" and select "Copy Link Address". You may also manually download and transfer to your server, skipping the next step.
On your server, type wget and paste the URL you copied in the previous step. Press enter to download.

Install the tools needed for installation.

Change the permissions of the downloaded .run file to allow execution, and run the installation.

Option 3: Install NVIDIA drivers using APT

Install a specific version of the driver for your GPU by installing the NVIDIA repository and using the apt package manager.

Run the command to get a list of the available driver's version

Install the driver version needed with apt

NVIDIA Driver Post-Installation steps

Reboot your system to ensure the new version of the driver is loaded

Verify Successful NVIDIA driver installation

If you see an error like the following, the NVIDIA drivers are probably installed incorrectly:

Review the installation instructions, specifically checking for completion of install prerequisites, and correct any errors.

Install Vulkan library

The rendering engine of HEAVY.AI requires a Vulkan-enabled driver and the Vulkan library. Without these components, the database itself can't even start without disabling the back-end renderer.

Install the Vulkan library and its dependencies using apt.

Advanced Installation

You must install the CUDA toolkit and Clang if you use advanced features like C++ User-Defined Functions and or User-Defined Table Functions to extend the database capabilities.

Install CUDA Toolkit ᴼᴾᵀᴵᴼᴺᴬᴸ

Install the NVIDIA public repository GPG key.

Add the repository.

List the available Cuda toolkit versions.

Install the CUDA toolkit using apt.

Verification

Check that everything is working and the toolkit has been installed.

Install Clang ᴼᴾᵀᴵᴼᴺᴬᴸ

You must install Clang if you use advanced features like C++ User-Defined Functions and or User-Defined Table Functions to extend the database capabilities. Install Clang and LLVM dependencies using apt.

Verification

Check that the software is installed and in the execution path.

Loading and Exporting Data

Supported Data Sources

Command Line

SQL

Heavy Immerse

Release Notes

Release notes for currently supported releases

Use of HEAVY.AI is subject to the terms of the HEAVY.AI End User License Agreement (EULA).

The latest release of HEAVY.AI is 7.2.4.

Currently Supported Releases

For release notes for releases that are no longer supported, as well as links to documentation for those releases, see Archived Release Notes.

As with any software upgrade, it is important to back up your data before you upgrade HEAVY.AI. In addition, we recommend testing new releases before deploying in a production environment.

For assistance during the upgrade process, contact HEAVY.AI Support by logging a request through the HEAVY.AI Support Portal.

Release 7.x.x - Important Information

This operation is performed on all tables in all catalogs at first startup, and the results are recorded in the INFO log.

For Ubuntu installations, install libncurses5 with the following command:

sudo apt install libncurses5

Release 7.2.4 - March 20, 2024

HeavyDB - Fixed Issues

Adds a new option for enabling or disabling the use of virtual addressing when accessing an S3 compatible endpoint for import or HeavyConnect.
Improves logging related to system locks.

Heavy Immerse - Fixed Issues

Fixes issue with SAML authentication.

Release 7.2.3 - February 5, 2024

HeavyDB - New Features and Improvements

Improves performance of foreign tables that are backed by Parquet files in AWS S3.
Improves logging related to GPU memory allocations and data transfers.

HeavyDB - Fixed Issues

Fixes a crash that could occur for certain query patterns with intermediate geometry projections.
Fixes a crash that could occur for certain query patterns containing IN operators with string function operands.
Fixes a crash that could occur for equi join queries that use functions as operands.
Fixes an intermittent error that could occur in distributed configurations when executing count distinct queries.
Fixes an issue where certain query patterns with LIMIT and OFFSET clauses could return wrong results.
Fixes a crash that could occur for certain query patterns with left joins on Common Table Expressions.
Fixes a crash that could occur for certain queries with window functions containing repeated window frames.

Heavy Render - Fixed Issues

Fix several crashes that could occur during out-of-gpu memory error recovery

Heavy Immerse - Fixed Issues

Fixed dashboard load error when switching tabs.
Fixed table reference in size measure of a client-side join data source for linemap chart.
Fixed client-side join name reference.

Release 7.2.2 - December 15, 2023

HeavyDB - New Features and Improvements

Adds support for output/result set buffer allocations via the "cpu-buffer-mem-bytes" configured CPU memory buffer pool. This feature can be enabled using the "use-cpu-mem-pool-for-output-buffers" server configuration parameter.
Adds a "ndv-group-estimator-multiplier" server configuration parameter that determines how the number of unique groups are estimated for specific query patterns.
Adds "default-cpu-slab-size" and "default-gpu-slab-size" server configuration parameters that are used to determine the default slab allocation size. The default size was previously based on the "max-cpu-slab-size" and "max-gpu-slab-size" configuration parameters.
Improves memory utilization when querying the "dashboards" system table.
Improves memory utilization in certain cases where queries are retried on CPU.
Improves error messages that are returned for some unsupported correlated subquery use cases.

HeavyDB - Fixed Issues

Fixes an issue where allocations could go beyond the configured "cpu-buffer-mem-bytes" value when fetching table chunks.
Fixes a crash that could occur when executing concurrent sort queries.
Fixes a crash that could occur when invalid geometry literals are passed to ST functions.

Heavy Immerse - Fixed Issues

Fix for rendering a gauge chart using a parameterized source (join sources, custom sources).

Release 7.2.1 - December 4, 2023

HeavyDB - New Features and Improvements

Improves instrumentation around Parquet import and HeavyConnect.

HeavyDB - Fixed Issues

Fixes a crash that could occur for join queries that result in many bounding box overlaps.
Fixes a crash that could occur in certain cases for queries containing an IN operator with a subquery parameter.
Fixes an issue where the ST_POINTN function could return wrong results when called with negative indexes.
Fixes an issue where a hang could occur while parsing a complex query.

Heavy Render - Fixed Issues

Fixed error when setting render-mem-bytes greater than 4gb.

Heavy Immerse - Fixed Issues

Clamp contour interval size on the Contour Chart to prevent a modulo operation error.
Filter outlier values in the Contour Chart that skew color range.
Fixed sample ratio query ordering to address a pointmap rendering issue.
Fixed layer naming in the Hide Layer menu.

Release 7.2.0 - November 16, 2023

HeavyDB - New Features and Improvements

Adds support for URL_ENCODE, URL_DECODE, REGEXP_COUNT, and HASH string functions.
Enables log based system tables by default.
Adds support for log based system tables auto refresh behind a flag (Beta).
Improves the pre-flight query row count estimation process for projection queries without filters.
Improves the performance of the LIKE operator.

HeavyDB - Fixed Issues

General

Fixes errors that could occur when the REPLACE clause is applied to SQL DDL commands that do not support it.
Fixes an issue where the HeavyDB startup script could ignore command line arguments in certain cases.
Fixes a crash that could occur when requests were made to the detect_column_types API for Parquet files containing list columns.
Fixes a crash that could occur in heavysql when the \detect command is executed for Parquet files containing string list columns.
Fixes a crash that could occur when attempting to cast to text column types in SELECT queries.
Fixes a crash that could occur in certain cases where window functions were called with literal arguments.
Fixes a crash that could occur when executing the ENCODE_TEXT function on NULL values.
Fixes an issue where queries involving temporary tables could return wrong results due to incorrect cache invalidation.

Geo

Fixes an issue where the ST_Distance function could return wrong results when at least one of its arguments is NULL.
Fixes an issue where the ST_Point function could return wrong results when the "y" argument is NULL.
Fixes an issue where the ST_NPoints function could return wrong results for NULL geometries.
Fixes a crash that could occur when the ST_PointN function is called with out-of-bounds index values.
Fixes an issue where the ST_Intersects and ST_Contains functions could incorrectly result in loop joins based on table order.
Fixes an issue where the ST_Transform function could return wrong results for NULL geometries.
Fixes an error that could occur for tables with polygon columns created from the output of user-defined table functions.

Heavy Immerse - New Features and Improvements

[Beta] Geo Joins - Immerse now supports “contains” and “intersects” conditions for common geometry combinations when creating a join datasource in the no code join editor.
Join datasource crossfilter support: Charts that use single table data sources will now crossfilter and be crossfiltered by charts that use join data sources.
Layer Drawer - In layered map charts immerse now has a quick to access Layer Drawer, which allows for layer toggling, reordering, renaming, opacity, zoom visibility controls.
Zoom to filters - Map charts in immerse now support “zoom to filters” functionality, either on an individual chart layer (via the Layer Drawer) or on the whole chart.
Image support in map rollovers - URLs pointing to images will automatically be rendered as a scaled image, with clickthrough support to the full size image.

Heavy Immerse - Fixed Issues

Choropleth/Line Map join datasource support - Significantly improves performance in Choropleth and Line Map charts when using join data sources. Auto aggregates measures on geometry.
Fixes issue where sql editor will horizontally scroll with long query strings

Release 7.1.2 - October 4, 2023

HeavyDB - New Features and Improvements

Improves how memory is allocated for the APPROX_MEDIAN aggregate function.

HeavyDB - Fixed Issues

Fixes a crash that could occur when the DISTINCT qualifier is specified for aggregate functions that do not support the distinct operation.
Fixes an issue where wrong results could be returned for queries with window functions that return null values.
Fixes a crash that could occur in certain cases where queries have multiple aggregate functions.
Fixes a crash that could occur when tables are created with invalid options.
Fixes a potential data race that could occur when logging cache sizes.

Release 7.1.1 - September 15, 2023

HeavyDB - New Features and Improvements

Adds an EXPLAIN CALCITE DETAILED command that displays more details about referenced columns in the query plan.
Improved logging around system memory utilization for each query.
Adds an option to SQLImporter for disabling logging of connection strings.
Adds a "gpu-code-cache-max-size-in-bytes" server configuration parameter for limiting the amount of memory that can be used by the GPU code cache.
Improves column name representation in Parquet validation error messages.

HeavyDB - Fixed Issues

Fixes a parser error that could occur for queries containing a NOT ILIKE clause.
Fixes a multiplication overflow error that could occur when retrying queries on CPU.
Fixes an issue where table dumps do not preserve quoted column names.
Fixes a "cannot start a transaction within a transaction" error that could occur in certain cases.
Fixes a crash that could occur for certain query patterns involving division by COUNT aggregation functions.
Removes a warning that is displayed on server startup when HeavyIQ is not configured.
Removes spurious warnings for CURSOR type checks when there are both cursor and scalar overloads for a user-defined table function.

Heavy Render - New Features and Improvements

Adds hit testing support for custom measures that reference multiple tables.

Heavy Immerse - Fixed Issues

Fixes SAML authentication regression in 7.1.0
Fixes chart export regression in 7.1.0

Release 7.1.0 - August 22, 2023

HeavyDB - New Features and Improvements

Geospatial

Exposes new geo overlaps function ST_INTERSECTSBOX for very fast bounding box intersection detections.
Adds support for the max_reject COPY FROM option when importing raster files. This ensures that imports from large multi-file raster datasets continue after minor errors, but provides adjustable notification upon major ones.
Adds a new ST_AsBinary (also aliased as ST_AsWKB) function that returns the Well-Known Binary (WKB) representation of geometry values. This highly-efficient format is used by postGIS newer versions of Geopandas.
Adds a new ST_AsText (also aliased AS ST_AsWKT) function that returns the Well-Known Text (WKT) representation of geometry values. This is less efficient than WKB but compatible even with nonspatial databases.
Adds support for loading geometry values using the load_table_binary_arrow Thrift API.
New version of HeavyAI python library with direct Geopandas support.
New version of rbc-project with geo column support allowing extensions which input or output any geometric type.

Core SQL

New JAROWINKLER_SIMILARITY string operator for fuzzy matching between string columns and values. This is a case insensitive measure including edit transitions and (slightly) sensitive to white space.
New LEVENSHTEIN_DISTANCE string operator for fuzzy matching between string columns and values. This is case insensitive and represents the number of edits needed to make two strings identical. An “edit” is defined by either an insertion of a character, a deletion of a character, or a replacement of a character.
Extends the ALTER COLUMN TYPE command to support string dictionary encoding size reduction.
Improves the error message returned when out of bound values are inserted into FLOAT and DOUBLE columns.
Adds a "watchdog-max-projected-rows-per-device" server configuration parameter and query hint that determines the maximum number of rows that can be projected by each GPU and CPU device.
Adds a "preflight-count-query-threshold" server configuration parameter and query hint that determines the threshold at which the preflight count query optimization should be executed.
Optimizes memory utilization for projection queries on instances with multiple GPUs.

Predictive Modeling with HeavyML

Support for PCA models and PCA_PROJECT operator.
Support SHOW MODEL FEATURE DETAILS to show per-feature info for models, including regression coefficients and variable importance scores, if applicable.
Support for TRAIN_FRACTION option to specify proportion of the input data to a CREATE MODEL statement that should be trained on.
Support creation of models with only categorical predictors.
Enable categorical and numeric predictors to be specified in any order for CREATE MODEL statements and subsequent inference operations.
Enable Torch table functions (requires client to specify libtorch.so).
Add tf_torch_raster_object_detect for raster object detections (requires client to specify libtorch.so and provide trained model in torchscript format).

Extensions Framework

Allow Array literals as arguments to scalar UDFs
Support table function (UDTF) output row sizes up to 16 trillion rows
Adds support for Column<TextEncodingNone> and ColumnList<TextEncodingNone> table function inputs and outputs.

Performance Optimizations

SQL projections now are sized per GPU/CPU core instead of globally, meaning that projections are more memory efficient as a function of the number of GPUs/CPU threads used for a query. In particular, this means that various forms of in-situ rendering, for example, non-grouped pointmaps, renders can scale to N GPUs more points or use N GPUs less memory, depending on the configuration.
Better parallelize construction of metadata for subquery results for improved performance
Enables result set caching for queries with LIMIT clauses.
Enables the bounding box intersection optimization for certain spatial join operators and geometry types by default.

HeavyDB - Fixed Issues

Fix potential crash when concatenating strings with the output of a UDF.
Fixes an issue where deleted rows with malformed data can prevent ALTER COLUMN TYPE command execution.
Fixes an error that could occur when parsing odbcinst.ini configuration files containing only one installed driver entry.
Fixes a table data corruption issue that could occur when the server crashes multiple times while executing write queries.
Fixes a crash that could occur when attempting to do a union of a string dictionary encoded text column and a none encoded text column.
Fixes a crash that could occur when the output of a table function is used as an argument to the strtok_to_array function.
Fixes a crash that could occur for queries involving projections of both geometry columns and geometry function expressions.
Fixes an issue where wrong results could be returned when the output of the DATE_TRUNC function is used as an argument to the count distinct function.
Fixes an issue where an error occurs if the COUNT_IF function is used in an arithmetic expression.
Fixes a crash that could occur when the WIDTH_BUCKET function is called with decimal columns.
Fixes an issue where the WIDTH_BUCKET function could return wrong results when called with decimal values close to the upper and lower boundary values.
Fixes a crash that could occur for queries with redundant projection steps in the query plan.

Heavy Render - Fixed Issues

Fixes a crash that could occur on multi-gpu systems while handling an out of GPU memory error.

Heavy Immerse - New Features and Improvements

Zoom to filters, setting map bounding box to extent of current filter set.
Image preview in map chart popups where image URLs are present.

Heavy Immerse - Fixed Issues

Fixed error thrown by choropleth chart on polygon hover.

Release 7.0.2 - June 28, 2023

HeavyDB - New Features and Improvements

Adds support for nested window function expressions.
Adds support for exception propagation from table functions.

HeavyDB - Fixed Issues

Fixes a crash that could occur when accessing 8-bit or 16-bit string dictionary encoded text columns on ODBC backed foreign tables.
Fixes unexpected GPU execution and memory allocations that could occur when executing sort queries with the CPU mode query hint.
Fixes an issue that could occur when inserting empty strings for geometry columns.
Fixes an issue that could occur when out of bounds fragment sizes are specified on table creation.
Fixes an issue where system dashboards could contain unexpected cached data.
Fixes a crash that could occur when executing aggregate functions over the result of join operations on scalar subqueries.
Fixes a server hang that could occur when GPU code compilation errors occur for user-defined table functions.
Fixes a data race that could occur when logging query plan cache size.

Heavy Render - New Features and Improvements

Add support for rendering 1D “terrain” cross-section overlays.
Rewrite 2D cross-section mesh generation as a table function.
Further improvements to system state logging when a render out of memory error occurs, and move it to the ERROR log for guaranteed visibility.
Enable auto-clear-render-mem by default for any render-vega call taking < 10 seconds.

Heavy Render - Fixed Issues

Render requests with 0 width or height could lead to a CHECK failure in encodePNG. Invalid image sizes now throw a non-fatal error during vega parsing.

Heavy Immerse - New Features and Improvements

Visualize terrain at the base of atmospheric cross sections in the Cross Section chart with the new Base Terrain chart layer type.

Heavy Immerse - Fixed Issues

Fixed local timezone issue with Chart Animation using cross filter replay.

Release 7.0.1 - June 8, 2023

HeavyDB - New Features and Improvements

Improves instrumentation around CPU and GPU memory utilization and certain crash scenarios.

HeavyDB - Fixed Issues

Fixes a crash that could occur for GPU executed join queries on dictionary encoded text columns with NULL values.

Heavy Render - New Features and Improvements

Improve instrumentation and logging related to gpu memory utilization, particularly with polygon rendering, as well as command timeout issues

Heavy Render - Fixed Issues

Fix a potential segfault when a Vulkan device lost error occurs

Release 7.0.0 - May 1, 2023

HeavyDB - New Features and Improvements

This operation is performed on all tables in all catalogs at first startup, and the results are recorded in the INFO log.

For Ubuntu installations, install libncurses5 with the following command:

sudo apt install libncurses5

Adds new Executor Resource Manager enabling parallel CPU and CPU-GPU query execution, and support for CPU execution on data inputs larger than fit in memory.
Adds HeavyML, a suite of machine learning capabilities accessible directly in SQL, including support for linear regression, random forest, gradient boosted trees, and decision tree regression models, and KMeans and DBScan clustering methods. (BETA)
Adds HeavyConnect support for MULTIPOINT and MULTILINESTRING columns.
Adds ALTER COLUMN TYPE support for text columns.
Adds a REASSIGN ALL OWNED command that allows for object ownership change across all databases.
Adds an option for validating POLYGON and MULTIPOLYGON columns when importing using the COPY FROM command or when using HeavyConnect.
Adds support for CONDITIONAL_CHANGE_EVENT window function.
Adds support for automatic casting of table function CURSOR arguments.
Adds support for Column<GeoMultiPolygon>, Column<GeoMultiLineString>, and Column<GeoMultiPoint> table function inputs and outputs.
Adds support for none encoded text column, geometry column, and array column projections from the right table in left join queries.
Adds support for literal text scalar subqueries.
Adds support for ST_X and ST_Y function output cast to text.
Improves concurrent execution of DDL and SHOW commands.
Improves error messaging for when the storage directory is missing.
Optimizes memory utilization for auto-vacuuming after delete queries.

HeavyDB - Fixed Issues

Fixes an issue where the root user could be deleted in certain cases.
Fixes an issue where staging directories for S3 import could remain when imports failed.
Fixes a crash that could occur when accessing the "tables" system table on instances containing tables with many columns.
Fixes a crash that could occur when accessing CSV and regex parsed file foreign tables that previously errored out during cache recovery.
Fixes an issue where dumping table foreign tables would produce an empty table.
Fixes an intermittent crash that could occur when accessing CSV and regex parsed file foreign tables that are backed by large files.
Fixes a "Ran out of slots in the query output buffer" exception that could occur when using stale cached cardinality values.
Fixes an issue where user defined table functions are erroneously categorized as ambiguous.
Fixes an error that could occur when a group by clause includes an alias that matches a column name.
Fixes a crash that could occur on GPUs with the Pascal architecture when executing join queries with case expression projections.
Fixes a crash that could occur when using the LAG_IN_FRAME window function.
Fixes a crash that could occur when projecting geospatial columns from the tf_raster_contour_polygons table function.
Fixes an issue that could occur when calling window functions on encoded date columns.
Fixes a crash that could occur when the coalesce function is called with geospatial or array columns.
Fixes a crash that could occur when projecting case expressions with geospatial or array columns.
Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
Fixes a crash that could occur in certain cases where left join queries are executed on GPU.
Fixes a crash that could occur for queries with joins on encoded date columns.
Fixes a crash that could occur when using the SAMPLE function on a geospatial column.
Fixes a crash that could occur for table functions with cursor arguments that specify no field type.
Fixes an issue where automatic casting does not work correctly for table function calls with ColumnList input arguments.
Fixes an issue where table function argument types are not correctly inferred when arithmetic operations are applied.
Fixes an intermittent crash that could occur for join queries due to a race condition when changing hash table layouts.
Fixes an out of CPU memory error that could occur when executing a query with a count distinct function call on a high cardinality column.
Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.
Fixes a crash that could occur in certain cases when HeavyConnect is used with Parquet files containing null string values.
Fixes potentially inaccurate calculation of vertical attenuation from antenna patterns in HeavyRF.

Heavy Render - New Features and Improvements

Add support for rendering a 1d cross-section as a line
Package the Vulkan loader libVulkan1 alongside heavydb

Heavy Render - Fixed Issues

Fix a device lost error that could occur with complex polygon renders

Heavy Immerse - New Features and Improvements

Data source Joins as a new custom data source type. (BETA)
Adds improved query performance defaults for the Contour Chart.
Adds access to new control panel to users with role "immerse_control_panel", even if the user is not a superuser.
Adds custom naming of map layers.
Adds custom map layer limit option using flag “ui/max_map_layers” which can be set explicitly (defaults to 8) or to -1 to remove the limit.

Heavy Immerse - Fixed Issues

Renames role from “immerse_trial_mode” to “immerse_export_disabled” and renames corresponding flag from “ui/enable_trial_mode” to “ui/user_export_disabled”.
Various minor UI fixes and polishing.
Fixes an issue where changing parameter value causes Choropleth popup to lose selected popup columns.
Fixes an issue where changing parameter value causes Pointmap to lose selected popup columns.
Fixes an issue where building a Skew-T chart results in a blank browser page.
Fixes an issue where Skew-T chart did not display wind barbs.
Fixes an issue with default date and time formatting.
Fixes an issue where setting flag "ui/enable_map_exports" to false unexpectedly disabled table chart export.
Fixes an issue with date filter presets.
Fixes an issue where filters "Does Not Contain" or "Does not equal" did not work on Crosslinked Columns.
Fixes an issue where charts were not redrawing to show the current bounding box filter set by the Linemap chart.

Release 6.4.4 - May 2, 2023

HeavyDB - New Features and Improvements

Adds support for literal text scalar subqueries.

HeavyDB - Fixed Issues

Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
Fixes a crash that could occur for queries with joins on encoded date columns.
Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.

Heavy Immerse - Fixed Issues

Fixed issue where Skew-T chart would not render when nulls were used in selected data.
Fixed issue where wind barbs were not visible on Skew-T chart.

Release 6.4.3 - February 27, 2023

Heavy Immerse - New Features and Improvements

Added feature flag ui/session_create_timeout with a default value of 10000 (10 seconds) for modifying login request timeout.

Release 6.4.2 - February 15, 2023

HeavyDB - New Features and Improvements

Adds the HeavyDB server configuration parameter enable-foreign-table-scheduled-refresh for enabling or disabling automated foreign table scheduled refreshes..

HeavyDB - Fixed Issues

Fixes a crash that could occur when S3 CSV-backed foreign tables with append refreshes are refreshed multiple times.
Fixes a crash that could occur when foreign tables with geospatial columns are refreshed after cache evictions.
Fixes a crash that could occur when querying foreign tables backed by Parquet files with empty row groups.
Fixes an error that could occur when select queries used in ODBC foreign tables reference case sensitive column names.
Fixes a crash that could occur when CSV backed foreign tables with geospatial columns are refreshed without updates to the underlying CSV files.
Fixes a crash that could occur in heavysql when executing the \detect command with geospatial files.
Fixes a casting error that could occur when executing left join queries.
Fixes a crash that could occur when accessing the disk cache on HeavyDB servers with the read-only configuration parameter enabled.
Fixes an error that could occur when executing queries that project geospatial columns.
Fixes a crash that could occur when executing the EXTRACT function with the ISODOW date_part parameter on GPUs.
Fixes an error that could occur when importing CSV or Parquet files with text columns containing more than 32,767 characters into HeavyDB NONE ENCODED text columns.

Heavy Render - Fixed Issues

Fixes a Vulkan Device Lost error that could occur when rendering complex polygon data with thousands of polygons in a single pixel.

Release 6.4.1 - January 30, 2023

HeavyDB - New Features and Improvements

Optimizes result set buffer allocations for CPU group by queries.
Enables trimming of white spaces in quoted fields during CSV file imports, when both the trim_spaces and quoted options are set.

HeavyDB - Fixed Issues

Fixes an error that could occur when importing CSV files with quoted fields that are surrounded by white spaces.
Fixes a crash that could occur when tables are reordered for range join queries.
Fixes a crash that could occur for join queries with intermediate projections.
Fixes a crash that could occur for queries with geospatial join predicate functions that use literal parameters.
Fixes an issue where queries could intermittently and incorrectly return error responses.
Fixes an issue where queries could return incorrect results when filter push-down through joins is enabled.
Fixes a crash that could occur for queries with join predicates that compare string dictionary encoded and nonencoded text columns.
Fixes an issue where hash table optimizations could ignore the max-cacheable-hashtable-size-bytes and hashtable-cache-total-bytes server configuration parameters.
Fixes an issue where sharded table join queries that are executed on multiple GPUs could return incorrect results.
Fixes a crash that could occur when sharded table join queries are executed on multiple GPUs with the from-table-reordering server configuration parameter enabled.

Heavy Immerse - New Features and Improvements

Multilayer support for Contour and Windbarb charts.
Enable Contour charts by default (feature flag: ui/enable_contour_chart).
Support custom SQL measures in Contour charts.
Restrict export from Heavy Immerse by enabling trial mode (feature flag: ui/enable_trial_mode). Trial mode enables a super user to restrict export capabilities for users who have the immerse_trial_mode role.

Heavy Immerse - Fixed Issues

Allow MULTILINESTRING to be used in selectors for Linemap charts.
Allow MULTILINESTRING to be used in Immerse SQL Editor.

Release 6.4.0 - December 16, 2022

Major improvements for time series analysis have been added. This includes an Immerse user interface for time series, and a large number of SQL window function additions and performance enhancements.

The release also includes two major architectural improvements:

The ability to perform cross-database queries, both in SQL and in Immerse, increasing flexibility across the board. For example, you can now easily build an Immerse dashboard showing system usage combined with business data. You might also make a read-only database of data shared across a set of users.
Render queries no longer block other GPU queries. In many use cases, renders can be significantly slower than other common queries. This should result in significant performance gains, particularly in map-heavy dashboards.

HeavyDB - New Features and Improvements

Core SQL

Adds support for cross database SELECT, UPDATE, and DELETE queries.
Support for MODE SQL aggregate.
Add support for strtok_to_array.
Support for ST_NumGeometries().
Support ST_TRANSFORM applied to literal geo types.
Enhanced query tracing ensures all child operations for a query_id are properly logged with that ID.

Data Linking and Import

Adds support for BigQuery and Hive HeavyConnect and import.
Adds support for table restore from S3 archive files.
Improves integer column type detection in Snowflake import/HeavyConnect data preview.
Adds HeavyConnect and import support for Parquet required scalar fields.
Improves import status error message when an invalid request is made.

Table Function Enhancements

Support POINT, LINESTRING, and POLYGON input and output types in table functions.
Support default values for scalar table function arguments.
Add tf_raster_contour table function to generate contours given x, y, and z arguments. This function is exposed in Immerse, but has additional capabilities available in SQL, such as supporting floating point contour intervals.
Return file path and file name from tf_point_cloud_metadata table function.
Previous length limit of 32K characters per values for none-encoded text columns has been lifted, and now none-encoded text values can be up to 2^31 - 1 characters (approximately 2.1billion characters).
Support array column outputs from table functions.
Add TEXT ENCODING DICT and Array<TEXT ENCODING DICT> type support for runtime functions/UDFs.
Allow transient TEXT ENCODING DICT column inputs into table functions.

Window Function Enhancements

Support COUNT_IF function.
Support SUM_IF function.
Support NTH_VALUE window function.
Support NTH_VALUE_IN_FRAME window function.
Support FIRST_VALUE_IN_FRAME and LAST_VALUE_IN_FRAME window functions.
Support CONDITIONAL_TRUE_EVENT.
Support ForwardFill and BackwardFill window functions to fill in missing (null) values based on previous non-null values in window.

HeavyDB - Fixed Issues

Fixes an issue where databases with duplicate names but different capitalization could be created.
Fixes an issue where raster imports could fail due to inconsistent band names.
Fixes an issue that could occur when DUMP/RESTORE commands were executed concurrently.
Fixes an issue where certain session updates do not occur when licenses are updated.
Fixes an issue where import/HeavyConnect data preview could return unsupported decimal types.
Fixes an issue where import/HeavyConnect data preview for PostgreSQL queries involving variable length columns could result in an error.
Fixes an issue where NULL elements in array columns with the NOT NULL constraint were not projected correctly.
Fixes a crash that could occur in certain scenarios where UPDATE and DELETE queries contain subqueries.
Fixes an issue where ingesting ODBC unsigned SQL_BIGINT into HeavyDB BIGINT columns using HeavyConnect or import could result in storage of incorrect data.
Fixes a crash that could occur in distributed configurations, when switching databases and accessing log based system tables with rolled off logs.
Fixes an error that occurred when importing Parquet files that did not contain statistics metadata.
Ensure query hint is propagated to subqueries.
Fix crash that could occur when LAG_IN_FRAME or LEAD_IN_FRAME were missing order-by or frame clause.
Fix bug where LAST_VALUE window function could return wrong results.
Fix issue where “Cannot use fast path for COUNT DISTINCT” could be reported from a count distinct operation.
Various bug fixes for support of VALUES() clause.
Improve handling of generic input expressions for window aggregate functions.
Fix bug where COUNT(*) and COUNT(1) over window frame could cause crash.
Fix wrong coordinate used for origin_y_bin in tf_raster_graph_shortest_slope_weighted_path.
Speed up table function binding in cases with no ColumnList arguments.
Support arrays of transient encoded strings into table functions.

Heavy Render - New Features and Improvements

Render queries no longer block parallel execution queue for other queries.

Heavy Immerse - New Features and Improvements

The Immerse PostgreSQL connector is now generally available, and is joined by public betas of Redshift and Snowflake.
New chart types:
- Contour chart. Contours can be applied to any geo point data, but are especially useful when applied to smoothly-varying pressure and elevation data. They can help reveal general patterns even in noisy primary data. Contours can be based on any point data, including that from regular raster grids like a temperature surface, or from sparse points like LiDAR data.
- Cross-section chart. As the name suggests, this allows a new view on 2.5D or 3D datasets, where a selected data dimension is plotted on the vertical axis for a slice of geographic data. In addition to looking in profile at parts of the atmosphere in weather modeling, this can also be used to look at geological sections below terrain.
- Representing vector force fields takes a step forward with the Wind barb plot. Wind barbs are multidimensional symbols which convey at a glance both strength and direction.
- Skew-T is a highly specialized multidimensional chart used primarily by meteorologists. Skew-Ts are heavily used in weather modeling and can help predict, for example, where thunderstorms or dry lightning are likely to occur.
Initial support for window functions in Immerse, enabling time lag analysis in charts. For example, you can now plot month-over-month or quarter-over-quarter sales or web traffic volume.
For categorical data, in addition to supporting aggregations based on the number of unique values, MODE is now supported. This supports the creation of groups based on the most-common value.

Release 6.2.7 - November 1, 2022

HeavyDB - Fixed Issues

Fixed an issue where a restarted server can potentially deadlock if the first two queries are executed at the same time and use different executors.

Release 6.2.5 - October 26, 2022

HeavyDB - Fixed Issues

Fixed an issue where COUNT DISTINCT or APPROX_COUNT_DISTINCT, when run on a CASE statement that outputs literal strings, could cause a crash.

Release 6.2.4 - October 12, 2022

HeavyDB - Fixed Issues

Fixes a crash when using COUNT() or COUNT(1) with the window function, i.e., COUNT(*) OVER (PARTITION BY x).
Fixes an incorrect result when using a date column as a partition key, like SUM(x) OVER (PARTITION BY DATE_COL).
Improves the performance of window functions when a literal expression is used as one of the input expressions of window functions like LAG(x, 1).
Improves query execution preparation phase by preventing redundant processing of the same nodes, especially when a complex input query is evaluated.
Fixes geometry type checking for range join operator that could cause a crash in some cases.
Resolves a query that may return an incorrect result when it has many projection expressions (for example, more than 50 8-byte output expressions) when using a window function expression.
Fixes an issue where the Resultset recycler ignores the server configuration size metrics.
Fixes a race condition where multiple catalogs could be created on initialization, resulting in possible deadlocks, server hangs, increased memory pressure, and slow performance.

Release 6.2.1 - September 27, 2022

HeavyDB - Fixed Issues

Fixes a crash encountered during some SQL queries when the read-only setting was enabled.
Fixes an issue in tf_raster_graph_shortest_slope_weighted_path table function that would lead some inputs to be incorrectly rejected.

6.2.0 - September 23, 2022

HeavyDB - New Features and Improvements

SQL Improvements

TRY_CAST support for string to numeric, timestamp, date, and time casts.
Implicit and explicit CAST support for numeric, timestamp, date, and time to TEXT type.
CAST support from Timestamp(0|3|6|9) types to Time(0) type.
Concat (||) operator now supports multiple nonliteral inputs.
JSON_VALUE operator to extract fields from JSON string columns.
BASE64_ENCODE and BASE64_DECODE operators for BASE64 encoding/decoding of string columns.
POSITION operator to extract index of search string from strings.
Add hash-based count distinct operator to better handle case of sparse columns.

Geospatial

Support MULTILINESTRING OGC geospatial type.
Support MULTIPOINT OGC geospatial type.
Support ST_NumGeometries.
Support ST_ConvexHull and ST_ConcaveHull.
Improved table reordering to maximize invocation of accelerated geo joins.
Support ST_POINT, ST_TRANSFORM and ST_SETSRID as expressions for probing columns in point-to-point distance joins.
Support accelerated overlaps hash join for ST_DWITHIN clause comparing two POINT columns.
Support for POLYGON to MULTIPOLYGON promotion in SQLImporter.

Window Functions

RANGE window function FRAME support for Time, Date, and Timestamp types.
Support LEAD_IN_FRAME / LAG_IN_FRAME window functions that compute LEAD / LAG in reference to a window frame.

Extension Functions

Add TextEncodingNone support for scalar UDF and extension functions.
Support array inputs and outputs to table functions.
Support literal interval types for UDTFs.
Add support for table functions range annotations for literal inputs

Performance and Control

Make max CPU threads configurable via a startup flag.
Support array types for Arrow/select_ipc endpoints.
Add support for query hint to control dynamic watchdog.
Add query hint to control Cuda block and grid size for query.
Adds an echo all option to heavysql that prints all executed commands and queries.
Improved decimal precision error messages during table creation.

HeavyConnect

Add support for file roll offs to HeavyConnect local and S3 file use cases.
Add HeavyConnect support for non-AWS S3-compatible endpoints.

Advanced Analytics

LiDAR

Add tf_point_cloud_metadata table function to read metadata from one or more LiDAR/point cloud files, optionally filtered by a bounding box.
Add tf_load_point_cloud table function to load data from one or more LiDAR/point cloud files, optionally filtered by bounding box and optionally cached in memory for subsequent queries.

Graph and Path Functions

Add tf_graph_shortest_path table function to compute shortest edge-weighted path between two points in a graph constructed from an input edge list
Add tf_graph_shortest_paths_distances table function to compute the shortest edge-weighted distances between a starting point and all other points in a graph constructed from an input edge list.
Add tf_grid_graph_shortest_slope_weighted_path table function to compute the shortest slope-weighted path between two points along rasterized data.

Enhanced Spatial Aggregations

Support configurable aggregation types for tf_geo_rasterize and tf_geo_rasterize_slope table functions, allowing for AVG, MIN, MAX, SUM, and COUNT aggregations.
Support two-pass gaussian blur aggregation post-processing for tf_geo_rasterize and tf_geo_rasterize_slope table functions.

RF Propagation Extension Improvements

Add dynamic ray splitting to tf_rf_prop_max_signal table function for improved performance and terrain coverage.
Add variant of tf_rf_prop_max_signal table function that takes per-RF source/tower transmission power (watts) and frequency (MHz).
Add variant of generate_series table function that generates series of timestamps between a start and end timestamp at specified time intervals.

Fixed Issues

ST_Centroid now automatically picks up SRID of underlying geometry.
Fixed a crash that occurred when ST_DISTANCE had an ST_POINT input for its hash table probe column.
Fixed an issue where a query hint would not propagate to a subquery.
Improved overloaded table function type deduction eliminates type mismatches when table function outputs are used downstream.
Properly handle cases of RF sources outside of terrain bounding box for tf_rf_prop_max_signal.
Fixed an issue where specification of unsupported GEOMETRY column type during table creation could lead to a crash.
Fixed a crash that could occur due to execution of concurrent create and drop table commands.
Fixed a crash that could occur when accessing the Dashboards system table.
Fixed a crash that could occur as a result of type mismatches in ITAS queries.
Fixed an issue that could occur due to band name sanitization during raster imports.
Fixed a memory leak that could occur when dropping temporary tables.
Fixed a crash that could occur due to concurrent execution of a select query and long-running write query on the same table.

Heavy Render - New Features and Improvements

Disables render group assignment by default.
Supports rendering of MULTILINESTRING geometries.
Memory footprint required for compositing renders on multi-GPU systems is significantly reduced. Any multi-GPU system will see improvements, but is most noticeable on systems with 4 or more GPUs. For example, rendering a 1400 x 1400 image results in ~450mb of memory saved when using 8 GPUs for a query. Multi-gpu system configurations should be able to set the res-gpu-mem configuration flag value lower as a result, freeing memory for other subsystems.
Adds INFO logging of peak render memory usage for the lifetime of the server process. The render memory logged is peak render query output buffer size (controlled with the render-mem-bytes configuration flag) and peak render buffer usage (controlled with the res-gpu-mem configuration flag). These peaks are logged in the INFO log on server shutdown, when GPU memory is cleared via clear_gpu_memory endpoint, or when a new peak is reached. These logged peaks can be useful to adjust the render-mem-bytes and res-gpu-mem configuration flags to improve memory utilization by avoiding reserving memory that might go unused. Examples of the log messages:
- When a new peak render-mem-bytes is reached: New peak render buffer usage (render-mem-bytes):37206200 of 1000000000
- When a new peak res-gpu-mem is reached: New peak render memory usage (res-gpu-mem): 166033024
- Peaks logged on server shutdown or on clear_gpu_memory: Render memory peak utilization:
  Query result buffer (render-mem-bytes): 37206200 of 1000000000 Images and buffers (res-gpu_mem): 660330240 Total allocated: 1660330240

Heavy Render - Fixed Issues

Fixed an issue the occurred when trying to hit-test a multiline SQL expression.

Heavy Immerse - New Features and Improvements

Dashboard and chart image export
Crossfilter replay
Improved popup support in the base 3D chart
New Multilayer CPU rendered Geo charts: Pointmap, Linemap, and Choropleth (Beta)
Control Panel (Beta)
Redshift, Snowflake, and PostGIS HeavyConnect support (Beta)
Skew-T chart (Beta)
Support for limiting the number of charts in a dashboard through the ui/limit_charts_per_dashboard feature flag. The default value is 0 (no limit).

Heavy Immerse - Fixed Issues

Fixed duplicate column names importer error.
Various bug fixes and user-interface improvements.