Release Notes
Release notes for currently supported releases
For release notes for releases that are no longer supported, as well as links to documentation for those releases, see Archived Release Notes.
As with any software upgrade, it is important to back up your data before you upgrade HEAVY.AI. In addition, we recommend testing new releases before deploying in a production environment.
For assistance during the upgrade process, contact HEAVY.AI Support by logging a request through the HEAVY.AI Support Portal.
IMPORTANT - In HeavyDB Release 7.0, the “render groups” mechanism, part of the previous implementation of polygon rendering, has been removed. When you upgrade to HeavyDB Release 7.0, all existing tables that have a POLYGON or MULTIPOLYGON geo column are automatically migrated to remove a hidden column containing "render groups" metadata.
This operation is performed on all tables in all catalogs at first startup, and the results are recorded in the INFO log.
Once a table has been migrated in this manner, it is not backwards-compatible with earlier versions of HeavyDB. If you revert to an earlier version, the table may appear to have missing columns and behavior will be undefined. Attempting to query or render the POLYGON or MULTIPOLYGON data with the earlier version may fail or cause a server crash.
As always, HEAVY.AI strongly recommends that all databases be backed up, or at the very least, dumps are made of tables with POLYGON or MULTIPOLYGON columns using the existing HeavyDB version, before upgrading to HeavyDB Release 7.0.
Dumps of POLYGON and MULTIPOLYGON tables made with earlier versions can still be restored into HeavyDB Release 7.0. The superfluous metadata is automatically discarded. However, dumps of POLYGON and MULTIPOLYGON tables made with HeavyDB Release 7.0 are not backwards-compatible with earlier versions.
This applies only to tables with POLYGON or MULTIPOLYGON columns. Tables that contain other geo column types (POINT, LINESTRING, etc.), or only non-geo column types, do not require migration and remain backwards-compatible with earlier releases..
For Ubuntu installations, install libncurses5 with the following command:
sudo apt install libncurses5- Adds new Executor Resource Manager enabling parallel CPU and CPU-GPU query execution, and support for CPU execution on data inputs larger than fit in memory.
- Adds HeavyML, a suite of machine learning capabilities accessible directly in SQL, including support for linear regression, random forest, gradient boosted trees, and decision tree regression models, and KMeans and DBScan clustering methods. (BETA)
- Adds HeavyConnect support for MULTIPOINT and MULTILINESTRING columns.
- Adds ALTER COLUMN TYPE support for text columns.
- Adds a REASSIGN ALL OWNED command that allows for object ownership change across all databases.
- Adds an option for validating POLYGON and MULTIPOLYGON columns when importing using the COPY FROM command or when using HeavyConnect.
- Adds support for CONDITIONAL_CHANGE_EVENT window function.
- Adds support for automatic casting of table function CURSOR arguments.
- Adds support for Column<GeoMultiPolygon>, Column<GeoMultiLineString>, and Column<GeoMultiPoint> table function inputs and outputs.
- Adds support for none encoded text column, geometry column, and array column projections from the right table in left join queries.
- Adds support for literal text scalar subqueries.
- Adds support for ST_X and ST_Y function output cast to text.
- Improves concurrent execution of DDL and SHOW commands.
- Improves error messaging for when the storage directory is missing.
- Optimizes memory utilization for auto-vacuuming after delete queries.
- Fixes an issue where the root user could be deleted in certain cases.
- Fixes an issue where staging directories for S3 import could remain when imports failed.
- Fixes a crash that could occur when accessing the "tables" system table on instances containing tables with many columns.
- Fixes a crash that could occur when accessing CSV and regex parsed file foreign tables that previously errored out during cache recovery.
- Fixes an issue where dumping table foreign tables would produce an empty table.
- Fixes an intermittent crash that could occur when accessing CSV and regex parsed file foreign tables that are backed by large files.
- Fixes a "Ran out of slots in the query output buffer" exception that could occur when using stale cached cardinality values.
- Fixes an issue where user defined table functions are erroneously categorized as ambiguous.
- Fixes an error that could occur when a group by clause includes an alias that matches a column name.
- Fixes a crash that could occur on GPUs with the Pascal architecture when executing join queries with case expression projections.
- Fixes a crash that could occur when using the LAG_IN_FRAME window function.
- Fixes a crash that could occur when projecting geospatial columns from the tf_raster_contour_polygons table function.
- Fixes an issue that could occur when calling window functions on encoded date columns.
- Fixes a crash that could occur when the coalesce function is called with geospatial or array columns.
- Fixes a crash that could occur when projecting case expressions with geospatial or array columns.
- Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
- Fixes a crash that could occur in certain cases where left join queries are executed on GPU.
- Fixes a crash that could occur for queries with joins on encoded date columns.
- Fixes a crash that could occur when using the SAMPLE function on a geospatial column.
- Fixes a crash that could occur for table functions with cursor arguments that specify no field type.
- Fixes an issue where automatic casting does not work correctly for table function calls with ColumnList input arguments.
- Fixes an issue where table function argument types are not correctly inferred when arithmetic operations are applied.
- Fixes an intermittent crash that could occur for join queries due to a race condition when changing hash table layouts.
- Fixes an out of CPU memory error that could occur when executing a query with a count distinct function call on a high cardinality column.
- Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
- Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.
- Fixes a crash that could occur in certain cases when HeavyConnect is used with Parquet files containing null string values.
- Fixes potentially inaccurate calculation of vertical attenuation from antenna patterns in HeavyRF.
- Add support for rendering a 1d cross-section as a line
- Package the Vulkan loader libVulkan1 alongside heavydb
- Fix a device lost error that could occur with complex polygon renders
- Data source Joins as a new custom data source type. (BETA)
- Adds improved query performance defaults for the Contour Chart.
- Adds access to new control panel to users with role "immerse_control_panel", even if the user is not a superuser.
- Adds custom naming of map layers.
- Adds custom map layer limit option using flag “ui/max_map_layers” which can be set explicitly (defaults to 8) or to -1 to remove the limit.
- Renames role from “immerse_trial_mode” to “immerse_export_disabled” and renames corresponding flag from “ui/enable_trial_mode” to “ui/user_export_disabled”.
- Various minor UI fixes and polishing.
- Fixes an issue where changing parameter value causes Choropleth popup to lose selected popup columns.
- Fixes an issue where changing parameter value causes Pointmap to lose selected popup columns.
- Fixes an issue where building a Skew-T chart results in a blank browser page.
- Fixes an issue where Skew-T chart did not display wind barbs.
- Fixes an issue with default date and time formatting.
- Fixes an issue where setting flag "ui/enable_map_exports" to false unexpectedly disabled table chart export.
- Fixes an issue with date filter presets.
- Fixes an issue where filters "Does Not Contain" or "Does not equal" did not work on Crosslinked Columns.
- Fixes an issue where charts were not redrawing to show the current bounding box filter set by the Linemap chart.
- Adds support for literal text scalar subqueries.
- Fixes a crash that could occur due to rounding error when using the WIDTH_BUCKET function.
- Fixes a crash that could occur for queries with joins on encoded date columns.
- Fixes a crash that could occur when running a HeavyDB instance in read-only mode after previously executing write queries on tables.
- Fixes an issue where the auto-vacuuming process does not immediately evict chunks that were pulled in for vacuuming.
- Fixed issue where Skew-T chart would not render when nulls were used in selected data.
- Fixed issue where wind barbs were not visible on Skew-T chart.
- Added feature flag
ui/session_create_timeoutwith a default value of 10000 (10 seconds) for modifying login request timeout.
- Adds the HeavyDB server configuration parameter
enable-foreign-table-scheduled-refreshfor enabling or disabling automated foreign table scheduled refreshes..
- Fixes a crash that could occur when S3 CSV-backed foreign tables with append refreshes are refreshed multiple times.
- Fixes a crash that could occur when foreign tables with geospatial columns are refreshed after cache evictions.
- Fixes a crash that could occur when querying foreign tables backed by Parquet files with empty row groups.
- Fixes an error that could occur when select queries used in ODBC foreign tables reference case sensitive column names.
- Fixes a crash that could occur when CSV backed foreign tables with geospatial columns are refreshed without updates to the underlying CSV files.
- Fixes a crash that could occur in heavysql when executing the \detect command with geospatial files.
- Fixes a casting error that could occur when executing left join queries.
- Fixes a crash that could occur when accessing the disk cache on HeavyDB servers with the read-only configuration parameter enabled.
- Fixes an error that could occur when executing queries that project geospatial columns.
- Fixes a crash that could occur when executing the EXTRACT function with the ISODOW
date_partparameter on GPUs. - Fixes an error that could occur when importing CSV or Parquet files with text columns containing more than 32,767 characters into HeavyDB NONE ENCODED text columns.
- Fixes a Vulkan Device Lost error that could occur when rendering complex polygon data with thousands of polygons in a single pixel.
- Optimizes result set buffer allocations for CPU group by queries.
- Enables trimming of white spaces in quoted fields during CSV file imports, when both the
trim_spacesandquotedoptions are set.
- Fixes an error that could occur when importing CSV files with quoted fields that are surrounded by white spaces.
- Fixes a crash that could occur when tables are reordered for range join queries.
- Fixes a crash that could occur for join queries with intermediate projections.
- Fixes a crash that could occur for queries with geospatial join predicate functions that use literal parameters.
- Fixes an issue where queries could intermittently and incorrectly return error responses.
- Fixes an issue where queries could return incorrect results when filter push-down through joins is enabled.
- Fixes a crash that could occur for queries with join predicates that compare string dictionary encoded and nonencoded text columns.
- Fixes an issue where hash table optimizations could ignore the
max-cacheable-hashtable-size-bytesandhashtable-cache-total-bytesserver configuration parameters. - Fixes an issue where sharded table join queries that are executed on multiple GPUs could return incorrect results.
- Fixes a crash that could occur when sharded table join queries are executed on multiple GPUs with the
from-table-reorderingserver configuration parameter enabled.
- Multilayer support for Contour and Windbarb charts.
- Support custom SQL measures in Contour charts.
- Restrict export from Heavy Immerse by enabling trial mode (feature flag:
ui/enable_trial_mode). Trial mode enables a super user to restrict export capabilities for users who have theimmerse_trial_moderole.
- Allow MULTILINESTRING to be used in selectors for Linemap charts.
- Allow MULTILINESTRING to be used in Immerse SQL Editor.
This release features general availability of data connectors for PostGreSQL, beta Immerse connectors for Snowflake and Redshift, and SQL support for Google BigQuery and Hive (beta). These managed data connections let you use HEAVY.AI as an acceleration platform wherever your source data may live. Scheduling and automated caching ensure that fast analytics are always running on the latest available data.
Immerse features four new chart types: Contour, Cross-section, Wind barb, and Skew-t. While especially useful for atmospheric and geotechnical data visualization, Contour and Cross-section also have more general application.
Major improvements for time series analysis have been added. This includes an Immerse user interface for time series, and a large number of SQL window function additions and performance enhancements.
The release also includes two major architectural improvements:
- The ability to perform cross-database queries, both in SQL and in Immerse, increasing flexibility across the board. For example, you can now easily build an Immerse dashboard showing system usage combined with business data. You might also make a read-only database of data shared across a set of users.
- Render queries no longer block other GPU queries. In many use cases, renders can be significantly slower than other common queries. This should result in significant performance gains, particularly in map-heavy dashboards.
- Adds support for cross database SELECT, UPDATE, and DELETE queries.
- Support for MODE SQL aggregate.
- Add support for strtok_to_array.
- Support for ST_NumGeometries().
- Support ST_TRANSFORM applied to literal geo types.
- Enhanced query tracing ensures all child operations for a query_id are properly logged with that ID.
- Adds support for BigQuery and Hive HeavyConnect and import.
- Adds support for table restore from S3 archive files.
- Improves integer column type detection in Snowflake import/HeavyConnect data preview.
- Adds HeavyConnect and import support for Parquet required scalar fields.
- Improves import status error message when an invalid request is made.
- Support POINT, LINESTRING, and POLYGON input and output types in table functions.
- Support default values for scalar table function arguments.
- Add tf_raster_contour table function to generate contours given x, y, and z arguments. This function is exposed in Immerse, but has additional capabilities available in SQL, such as supporting floating point contour intervals.
- Return file path and file name from tf_point_cloud_metadata table function.
- Previous length limit of 32K characters per values for none-encoded text columns has been lifted, and now none-encoded text values can be up to 2^31 - 1 characters (approximately 2.1billion characters).
- Support array column outputs from table functions.
- Add TEXT ENCODING DICT and Array<TEXT ENCODING DICT> type support for runtime functions/UDFs.
- Allow transient TEXT ENCODING DICT column inputs into table functions.
- Support COUNT_IF function.
- Support SUM_IF function.
- Support NTH_VALUE window function.
- Support NTH_VALUE_IN_FRAME window function.
- Support FIRST_VALUE_IN_FRAME and LAST_VALUE_IN_FRAME window functions.
- Support CONDITIONAL_TRUE_EVENT.
- Support ForwardFill and BackwardFill window functions to fill in missing (null) values based on previous non-null values in window.
- Fixes an issue where databases with duplicate names but different capitalization could be created.
- Fixes an issue where raster imports could fail due to inconsistent band names.
- Fixes an issue that could occur when DUMP/RESTORE commands were executed concurrently.
- Fixes an issue where certain session updates do not occur when licenses are updated.
- Fixes an issue where import/HeavyConnect data preview could return unsupported decimal types.
- Fixes an issue where import/HeavyConnect data preview for PostgreSQL queries involving variable length columns could result in an error.
- Fixes an issue where NULL elements in array columns with the NOT NULL constraint were not projected correctly.
- Fixes a crash that could occur in certain scenarios where UPDATE and DELETE queries contain subqueries.
- Fixes an issue where ingesting ODBC unsigned SQL_BIGINT into HeavyDB BIGINT columns using HeavyConnect or import could result in storage of incorrect data.
- Fixes a crash that could occur in distributed configurations, when switching databases and accessing log based system tables with rolled off logs.
- Fixes an error that occurred when importing Parquet files that did not contain statistics metadata.
- Ensure query hint is propagated to subqueries.
- Fix crash that could occur when LAG_IN_FRAME or LEAD_IN_FRAME were missing order-by or frame clause.
- Fix bug where LAST_VALUE window function could return wrong results.
- Fix issue where “Cannot use fast path for COUNT DISTINCT” could be reported from a count distinct operation.
- Various bug fixes for support of VALUES() clause.
- Improve handling of generic input expressions for window aggregate functions.
- Fix bug where COUNT(*) and COUNT(1) over window frame could cause crash.
- Fix wrong coordinate used for origin_y_bin in tf_raster_graph_shortest_slope_weighted_path.
- Speed up table function binding in cases with no ColumnList arguments.
- Support arrays of transient encoded strings into table functions.
Render queries no longer block parallel execution queue for other queries.
- The Immerse PostgreSQL connector is now generally available, and is joined by public betas of Redshift and Snowflake.
- New chart types:
- Contour chart. Contours can be applied to any geo point data, but are especially useful when applied to smoothly-varying pressure and elevation data. They can help reveal general patterns even in noisy primary data. Contours can be based on any point data, including that from regular raster grids like a temperature surface, or from sparse points like LiDAR data.
- Cross-section chart. As the name suggests, this allows a new view on 2.5D or 3D datasets, where a selected data dimension is plotted on the vertical axis for a slice of geographic data. In addition to looking in profile at parts of the atmosphere in weather modeling, this can also be used to look at geological sections below terrain.
- Representing vector force fields takes a step forward with the Wind barb plot. Wind barbs are multidimensional symbols which convey at a glance both strength and direction.
- Skew-T is a highly specialized multidimensional chart used primarily by meteorologists. Skew-Ts are heavily used in weather modeling and can help predict, for example, where thunderstorms or dry lightning are likely to occur.
- Initial support for window functions in Immerse, enabling time lag analysis in charts. For example, you can now plot month-over-month or quarter-over-quarter sales or web traffic volume.
- For categorical data, in addition to supporting aggregations based on the number of unique values, MODE is now supported. This supports the creation of groups based on the most-common value.
- Fixed an issue where a restarted server can potentially deadlock if the first two queries are executed at the same time and use different executors.
- Fixed an issue where COUNT DISTINCT or APPROX_COUNT_DISTINCT, when run on a CASE statement that outputs literal strings, could cause a crash.
- Fixes a crash when using COUNT() or COUNT(1) with the window function, i.e., COUNT(*) OVER (PARTITION BY x).
- Fixes an incorrect result when using a date column as a partition key, like SUM(x) OVER (PARTITION BY DATE_COL).
- Improves the performance of window functions when a literal expression is used as one of the input expressions of window functions like LAG(x, 1).
- Improves query execution preparation phase by preventing redundant processing of the same nodes, especially when a complex input query is evaluated.
- Fixes geometry type checking for range join operator that could cause a crash in some cases.
- Resolves a query that may return an incorrect result when it has many projection expressions (for example, more than 50 8-byte output expressions) when using a window function expression.
- Fixes an issue where the Resultset recycler ignores the server configuration size metrics.
- Fixes a race condition where multiple catalogs could be created on initialization, resulting in possible deadlocks, server hangs, increased memory pressure, and slow performance.
- Fixes a crash encountered during some SQL queries when the read-only setting was enabled.
- Fixes an issue in
tf_raster_graph_shortest_slope_weighted_pathtable function that would lead some inputs to be incorrectly rejected.
In Release 6.2.0, Heavy Immerse adds animation and a control panel system. HeavyConnect now includes connectors for Redshift, Snowflake, and PostGIS. The SQL system is extended with support for casting and time-based window functions. GeoSQL gets direct LiDAR import, multipoints, and multilinestrings, as well as graph network algorithms. Other enhancements include performance improvements and reduced memory requirements across the product.
- TRY_CAST support for string to numeric, timestamp, date, and time casts.
- Implicit and explicit CAST support for numeric, timestamp, date, and time to TEXT type.
- CAST support from Timestamp(0|3|6|9) types to Time(0) type.
- Concat (||) operator now supports multiple nonliteral inputs.
- JSON_VALUE operator to extract fields from JSON string columns.
- BASE64_ENCODE and BASE64_DECODE operators for BASE64 encoding/decoding of string columns.
- POSITION operator to extract index of search string from strings.
- Add hash-based count distinct operator to better handle case of sparse columns.
- Support MULTILINESTRING OGC geospatial type.
- Support MULTIPOINT OGC geospatial type.
- Support ST_NumGeometries.
- Support ST_ConvexHull and ST_ConcaveHull.
- Improved table reordering to maximize invocation of accelerated geo joins.
- Support ST_POINT, ST_TRANSFORM and ST_SETSRID as expressions for probing columns in point-to-point distance joins.
- Support accelerated overlaps hash join for ST_DWITHIN clause comparing two POINT columns.
- Support for POLYGON to MULTIPOLYGON promotion in SQLImporter.
- RANGE window function FRAME support for Time, Date, and Timestamp types.
- Support LEAD_IN_FRAME / LAG_IN_FRAME window functions that compute LEAD / LAG in reference to a window frame.
- Add TextEncodingNone support for scalar UDF and extension functions.
- Support array inputs and outputs to table functions.
- Support literal interval types for UDTFs.
- Add support for table functions range annotations for literal inputs
- Make max CPU threads configurable via a startup flag.
- Support array types for Arrow/select_ipc endpoints.
- Add support for query hint to control dynamic watchdog.
- Add query hint to control Cuda block and grid size for query.
- Adds an
echo alloption to heavysql that prints all executed commands and queries. - Improved decimal precision error messages during table creation.
- Add support for file roll offs to HeavyConnect local and S3 file use cases.
- Add HeavyConnect support for non-AWS S3-compatible endpoints.
LiDAR
- Add
tf_point_cloud_metadatatable function to read metadata from one or more LiDAR/point cloud files, optionally filtered by a bounding box. - Add
tf_load_point_cloudtable function to load data from one or more LiDAR/point cloud files, optionally filtered by bounding box and optionally cached in memory for subsequent queries.
Graph and Path Functions
- Add
tf_graph_shortest_pathtable function to compute shortest edge-weighted path between two points in a graph constructed from an input edge list - Add
tf_graph_shortest_paths_distancestable function to compute the shortest edge-weighted distances between a starting point and all other points in a graph constructed from an input edge list. - Add
tf_grid_graph_shortest_slope_weighted_pathtable function to compute the shortest slope-weighted path between two points along rasterized data.
Enhanced Spatial Aggregations
- Support configurable aggregation types for
tf_geo_rasterizeandtf_geo_rasterize_slopetable functions, allowing for AVG, MIN, MAX, SUM, and COUNT aggregations. - Support two-pass gaussian blur aggregation post-processing for
tf_geo_rasterizeandtf_geo_rasterize_slopetable functions.
RF Propagation Extension Improvements
- Add dynamic ray splitting to
tf_rf_prop_max_signaltable function for improved performance and terrain coverage. - Add variant of
tf_rf_prop_max_signaltable function that takes per-RF source/tower transmission power (watts) and frequency (MHz). - Add variant of
generate_seriestable function that generates series of timestamps between a start and end timestamp at specified time intervals.
- ST_Centroid now automatically picks up SRID of underlying geometry.
- Fixed a crash that occurred when ST_DISTANCE had an ST_POINT input for its hash table probe column.
- Fixed an issue where a query hint would not propagate to a subquery.
- Improved overloaded table function type deduction eliminates type mismatches when table function outputs are used downstream.
- Properly handle cases of RF sources outside of terrain bounding box for
tf_rf_prop_max_signal. - Fixed an issue where specification of unsupported GEOMETRY column type during table creation could lead to a crash.
- Fixed a crash that could occur due to execution of concurrent create and drop table commands.
- Fixed a crash that could occur when accessing the Dashboards system table.
- Fixed a crash that could occur as a result of type mismatches in ITAS queries.
- Fixed an issue that could occur due to band name sanitization during raster imports.
- Fixed a memory leak that could occur when dropping temporary tables.
- Fixed a crash that could occur due to concurrent execution of a select query and long-running write query on the same table.
- Disables render group assignment by default.
- Supports rendering of MULTILINESTRING geometries.
- Memory footprint required for compositing renders on multi-GPU systems is significantly reduced. Any multi-GPU system will see improvements, but is most noticeable on systems with 4 or more GPUs. For example, rendering a 1400 x 1400 image results in ~450mb of memory saved when using 8 GPUs for a query. Multi-gpu system configurations should be able to set the
res-gpu-memconfiguration flag value lower as a result, freeing memory for other subsystems. - Adds INFO logging of peak render memory usage for the lifetime of the server process. The render memory logged is peak render query output buffer size (controlled with the
render-mem-bytesconfiguration flag) and peak render buffer usage (controlled with theres-gpu-memconfiguration flag). These peaks are logged in the INFO log on server shutdown, when GPU memory is cleared viaclear_gpu_memoryendpoint, or when a new peak is reached. These logged peaks can be useful to adjust therender-mem-bytesandres-gpu-memconfiguration flags to improve memory utilization by avoiding reserving memory that might go unused. Examples of the log messages:- When a new peak
render-mem-bytesis reached:New peak render buffer usage (render-mem-bytes):37206200 of 1000000000 - When a new peak
res-gpu-memis reached:New peak render memory usage (res-gpu-mem): 166033024 - Peaks logged on server shutdown or on
clear_gpu_memory:Render memory peak utilization:Query result buffer (render-mem-bytes): 37206200 of 1000000000 Images and buffers (res-gpu_mem): 660330240 Total allocated: 1660330240
- Fixed an issue the occurred when trying to hit-test a multiline SQL expression.
- Dashboard and chart image export
- Crossfilter replay
- Improved popup support in the base 3D chart
- New Multilayer CPU rendered Geo charts: Pointmap, Linemap, and Choropleth (Beta)
- Control Panel (Beta)
- Redshift, Snowflake, and PostGIS HeavyConnect support (Beta)
- Skew-T chart (Beta)
- Support for limiting the number of charts in a dashboard through the
ui/limit_charts_per_dashboardfeature flag. The default value is 0 (no limit).
- Fixed duplicate column names importer error.
- Various bug fixes and user-interface improvements.
- Adds support for POLYGON to MULTIPOLYGON promotion in the load table Thrift APIs and SQLImporter.
- Fixes an issue that caused an intermittent KafkaImporter crash on CentOS 7.9.
- Fixes an issue that cause incorrect results in multiple aggregation of date columns that include COUNT DISTINCT.
- Adds support for limiting the number of charts in a dashboard through the
ui/limit_charts_per_dashboardfeature flag. The default value is 0 (no limit).
- Adds a new set of log-based (request_logs, server_logs, web_server_logs, and web_server_access_logs) system tables.
- Adds a new Request Logs and Monitoring dashboard.
- Adds a new SHOW CREATE SERVER command, which displays the create server DDL for a specified foreign server.
- Adds support for non-super-user execution of SHOW CREATE TABLE on views.
- Adds a new ALTER SESSION SET EXECUTOR_DEVICE command, which updates the type of executor device (CPU or GPU) for the current session.
- Adds a new ALTER SESSION SET CURRENT_DATABASE command, which updates the connected database for the current session.
- Adds a new ALTER DATABASE OWNER TO command, which allows super users to change the owner of a database.
- Extends the INSERT command to support inserting multiple rows at once/batch insert.
- Add support for default values on shard key columns.
- Add initial support for Window function framing, including support for BETWEEN ROWS clause for all numeric and date/time types, and BETWEEN RANGE clause for numeric types.
- Enable group-by push down for UNION ALL such that group-by and aggregate operations applied to the output of UNION ALL are evaluated on the UNION ALL inputs, improving performance.
- Add support for LCASE (alias for LOWER), UCASE (alias for UPPER), LEFT, and RIGHT string functions
- Adds a new trim_spaces option for delimited file import.
- (BETA) Adds data import/COPY FROM support from Relational Database Management Systems and Data Warehouses using the Open Database Connectivity (ODBC) interface.
- Initial support for CUDA streams to parallelize GPU computation and memory transfers.
- Increase per-GPU projection limit with watchdog enabled from 32M to 128M rows to take advantage of improvements in large projection support in recent releases.
- Add new SHOW FUNCTIONS and SHOW FUNCTIONS DETAILS commands to show registered compile-time UDFs and extension functions in the system and their arguments, and SHOW RUNTIME FUNCTIONS [DETAILS] and SHOW RUNTIME TABLE FUNCTIONS [DETAILS] to show user-defined runtime functions/table functions.
- Support timestamp inputs and outputs for table functions.
Advanced Analytics
- Add tf_compute_dwell_times table function, that given a query input input with entity keys and timestamps, and parameters specifying the minimum session time, minimum number of session records, and max inactive seconds, outputs all unique sessions found in the data with the duration of the session (dwell time).
- Add tf_feature_self_similarity table function, that given a query input of entity keys/IDs, a set of feature columns, and a metric column, scores each pair of entities based on their similarity, computed as the cosine similarity of the feature column(s) between each entity pair, which can optionally TF/IDF weighted.
- Add tf_feature_similarity table function, that given a query input of entity keys, feature columns, and a metric column, and a second query input specifying a search vector of feature columns and metric, computes the similarity of each entity in the first input to the search vector based on their similarity, computed as the cosine similarity of the feature column(s) for each entity with the feature column(s) for the search vector, which can optionally be TF/IDF weighted.
- Fixed an issue where some join queries on ODBC-backed foreign tables can return empty result sets for the first query.
- Fixed an issue where append refreshes on foreign tables backed by delimited or regex-parsed files ignore file-path filter and sort options.
- Fixed a crash that can occur when very large dates are specified for the refresh_start_date_time foreign table option.
- Fixed a crash that can occur when a foreign table’s data source is updated within a refresh window.
- Fixed an issue where databases owned by deleted user accounts are not visible, and adds a restriction that prevents dropping users who own databases.
- Fixed an issue where joins on string dictionary-encoded columns would hit spurious none-encoded string translation
- Fixed issue with certain UNION ALL query patterns, such as UNION ALL containing logical values.
- Disabled KEY_FOR_STRING for UNNEST operations on string dictionary-encoded columns, to prevent a crash.
- Fixed an issue where logged stats for raster imports could overflow.
- Fixed an issue where joins on synthetic tables (for example, created with a VALUES statement or table function without an underlying table) could crash.
- Fixed an issue where require checks used on string dictionary inputs to a table function could crash.
- Fixed a crash and/or wrong query results that can occur when a decimal literal is used in a nested query.
- Fixed a potential crash when attempting to auto-retry a render immediately after an OutOfGpuMemory exception is thrown. This crash can occur only if the
render-oom-retry-thresholdconfiguration option is set. - Fixed a regression where polygons with transparent colors are rendered opaque.
- Corrects an issue with point/symbol rendering with explicitly Vega projections where the projection was not being updated when panned/zoomed if the query did not change.
- Significant improvements in hit-testing consistency and stability when rendering queries with subqueries, window functions, or table functions.
- Font size controls.
- Borders and Zebra Striping in Table charts.
- Justify content in Table charts.
- Customization polygon border control.
- Allow measure date formatting for table charts.
- Extend y-axis on Vega combo charts to end at the next whole value past the highest data point.
- Add layer visibility toggle to kebab dropdown on multi-layer raster charts.
- Made unsaved changes modal less aggressive.
- Custom Source Table Functions Browser
- Don’t show unsaved warning modal after adding default filter set.
- (BETA) PostgreSQL connector.
- Allows maxBounds to be set in servers.json.
- Toggle dashboard unsaved when updating annotations.
- Dashboard save state behavior fixes.
- Table Chart order by group keys when present.
- Use key_for_string when ordering by known dictionary measures/dimensions.
- Add default formatting for date/time on table chart.
- Add admin feature flag to hide key manager.
- Customizable polygon border color and existing border bug fixes.
- Cannot append to table using PostgreSQL connector.
- Building a raster chart with the layer visibility toggle feature flag enabled causes a crash.
- Support for fast string functions on dictionary-encoded text columns (the default), including LOWER, UPPER, INITCAP, TRIM/LTRIM/RTRIM, LPAD/RPAD, REVERSE, REPEAT, SUBSTRING/SUBSTR, REPLACE, OVERLAY, SPLIT_PART, REGEXP_REPLACE, REGEXP_SUBSTR, AND CONCAT (||). The output of these expressions can be chained, grouped-by, and used in both the left and right side of join predicates.
- Support for fast string equality/inequality operations without the previous requirement of watchdog disablement when the two columns do not share dictionaries.
- Support for fast case statements with multiple text column inputs that do not share dictionary-encoded strings.
- Support for ENCODE_TEXT to encode none-encoded strings, which can then be grouped on and manipulated like dictionary-encoded strings. This operator is not intended for interactive use at scale but instead for ELT-like scenarios. Use the new server flag
watchdog-none-encoded-string-translation-limitto set the upper cardinality allowed for such operations (1,000,000 by default). - Support for UNION ALL is enabled by default, and now works across string columns that do not share dictionaries with significantly better performance than in the previous release.
- Window functions now support expressions in the PARTITION BY and ORDER clauses.
- Support for subqueries in CASE statement clauses.
- SHOW USER DETAILS is changed to only list those users with access to the currently-selected database. Previously, all users on the HeavyDB instance would be listed; this is still available to superusers with SHOW ALL USER DETAILS.
- 10X improvements in initial join performance (including geo joins) through faster, parallelized hash table construction, removing redundant inter-thread hash table computation.
- Improved join ordering to avoid loop joins in certain scenarios.
- Parallel compilation of queries as well as inter-executor generated code increases concurrency and throughput in common, Immerse-driven scenarios by up to 20%. Also decreases latency for a single user interacting with dashboards or issuing SQL queries in a way that required new plans to be code-generated.
- New result set recycler allows query substeps (expensive in subqueries) can be cached using SQL hints ( /*+ keep_result */), dramatically improving performance where the subquery is reused across multiple queries (for example, in Immerse) and only outer steps of the query vary.
- The default for the header option of
COPY TOto a CSV/TSV file has been changed from'false'to'true'. - Faster dictionary map in StringDictionaryProxy, accelerating various string operations involving transient entries.
- Arrow execution endpoints now use multiple executors and can run concurrently like queries issued to the Thrift endpoints.
- Addition of sparse dictionary output capability for Arrow queries, which automatically creates a subset of a string dictionary to send via Arrow when it detects that it is faster than sending the full, unfiltered dictionary. This provides orders-of-magnitude better server- and client-side performance and scalability for common cases where large dictionary-encoded text columns are filtered or top-k sorted such that only a small subset of dictionary entries are needed in the result set.
- ST_INTERSECTS now can operate directly on top of compressed (the default) coordinates, leading to 2-3X increase in speed.
- New table function framework allows for both system and user-defined table functions. Table functions can run on both CPU and GPU and are designed for efficient, scalable execution of custom algorithms in-situ on data that might be hard or impossible to implement in SQL.
- Support for generate_series table function (similar to Postgres) for easy and fast integer series generation, particularly useful for left joins against binned tables to fill in gaps, whether for visualization or downstream operations like window functions, and generate_random_strings for generation of string columns of a user-defined size and cardinality.
- Support for geo_rasterize and geo_rasterize_slope table functions to efficiently bin vector data into gap-free bins, with the optional ability to fill in null values, apply box blur, and compute slope and aspect ratios
- Initial support for HeavyRF, a module that allows for real-time, ray-based computation of signal propagation, taking inputs of both terrain data and real or hypothetical signal sources.
- Beta support for Python-defined scalar (row-level) and tabular User Defined Functions (UDFs and UDTFs), using the RBC library to translate Numba python code into LLVM IR that is JITed into query execution code for fast, scalable, custom user-defined capabilities.
- Complete redesign and rewrite of Parquet import to one that is more robust, efficient, and performant.
- Adds support for import from regex parsed files on either the server file system or S3 using the COPY FROM command.
- The
geoandparquetWITHoptions ofCOPY FROMhave been deprecated and replaced bysource_type. Using the deprecated syntax generates the following:Deprecation Warning: COPY FROM WITH (geo='true') is deprecated. Use WITH (source_type='geo_file') instead.Update any scripts you have to replace the deprecated syntax with the new syntax. For more information, see CSV/TSV Import. - (BETA) Adds support for import from RDMS/data warehouses using the COPY FROM command.
- A new default information_schema database contains 10 new system tables that provide information regarding CPU/GPU memory utilization, storage space utilization, database objects, and database object permissions.
- New system dashboards that enable intuitive visualization of system resource utilization and user roles and permissions.
- Support for Zarr and NetCDF raster file import.
- You can now import raster files with ground control points geospatial references.
- Support for file path filtering, globbing, and sorting when importing geo and raster files.
- Improved error messaging when attempting to save a dashboard that uses a duplicate dashboard name.
- Support for connections to delimited files on either the server file system or S3. S3 support includes an option to use the S3 Select API, which provides better performance but with limitations on supported column types.
- Support for connections to Parquet files on either the server file system or S3. HeavyConnect leverages Parquet metadata to provide efficient data access and row group-level filter push down.
- Parquet column type coercion. Convert Parquet column types to more memory-efficient HeavyDB column types for use cases that guarantee no loss of information.
- Connections to regex parsed files on either the server file system or S3. This enables you to query unstructured text files, such as logs, by specifying regular expression patterns that extract components of the text files into table columns.
- (BETA) Support for connections to Relational Database Management Systems and Data Warehouses, leveraging the Open Database Connectivity (ODBC) interface to provide seamless access to data.
- (BETA) ODBC column type coercion. Use HeavyConnect to convert ODBC column types to more memory-efficient HeavyDB column types for use cases that guarantee no loss of information.
- Support for scheduled data refreshes. Specify a start date time and interval at which connected data gets refreshed.
- Adds support for disk level caching. By default, data fetched by HeavyConnect are cached at the disk level in addition to normal CPU/GPU level caching. This provides better overall query performance for network based connections, such as S3, and systems with limited CPU/GPU memory capacity. Disk cache size and level can be set through HeavyDB server configuration.
- Adds support for file path filtering, globbing, and sorting for Parquet, delimited, and regex parsed file use cases.
- Complete redesign and rewrite of the Parquet detect_column_types Thrift API. The Parquet detect/data preview feature is now more robust, efficient, and performant.
- Change to query interrupt mechanism allowing certain classes of queries such as loop joins to be easily and quickly interrupted.
- Fixed crash that could occur with joins on predicates that had functions on the left hand side expression, i.e. geoToH3.
- Fix crash that could occur with Arrow queries that did not return results.
- Avoid building metadata for empty result sets.
- Fixes a crash that can occur when executing queries on GPU that involve a baseline group by and variable length column projections.
- Fixes some table query concurrency bottlenecks. Previously, queries such as INSERT, TRUNCATE, and DROP TABLE required system wide locks to execute and would therefore block execution of other unrelated queries. These kinds of queries can now be executed concurrently.
- Fixes a crash that can occur on server restart when the disk cache is enabled and tables with cached data are deleted.
- Fixes a crash that can occur when the max_rows table option is altered for an empty table.
- Fixes an issue in the JDBC driver where tables from multiple databases are listed even when a single database is specified.
- Fixes an issue where raster POINT column type import would incorrectly throw an exception.
- Fixes a crash that can occur when restoring a dump for a table with previously deleted columns.
- Updates the export COPY TO command to include headers by default.
- Removes the file_type parameter from the create_table Thrift API. This parameter was not used.
- Fixes a crash that can occur when executing SQL commands containing comments.
- Fixed the setting for default database (DEFAULT_DB) being ignored in a SAML login for a user who already exists.
- The OpenGL renderer driver has been fully removed as of this release. Vulkan is the only available driver and enables a more modern, flexible API. As a result, the
renderer-use-vulkan-driverprogram option has been removed. Remove any references to that program option from your configuration files. For more on the move to the Vulkan driver, see Vulkan Renderer. - A novel polygon rendering algorithm is now used as the default when rendering polygons. This algorithm does no triangulation nor does it require “render groups” (a hidden column to assist the old polygon rendering algorithm). However, the render groups column is still added on import as a fallback. See Importing Geospatial Data for more on render group deprecation.
- You can now hit-test certain render queries with subqueries more effectively. For example, if the subquery is only used for filter predicates, renders should now be sped up and hit-testing more flexible.
- Render times are now being logged correctly (“render_vega-COMPLETED nonce:2 Total Execution: (ms), Total Render: (ms)”). The execution time and render time were incorrectly logged as 0 in Releases 5.9 and 5.10
- Fixes a regression introduced in Release 5.10.0 when hit-testing an Immerse cohort-generated query. The hit-test would result in an error such as the following:
“Cannot find column in hit-test cache for query …” - Resolves a crash when trying to hit-test render queries with window functions or cursorless table functions.
- Fixes an issue where a multi-layer, multi-GPU render with a poly or line mark as the first layer can result in ghosting artifacts if the query associated with that layer resulted in 0 rows.
- Fixes an issue when switching between a density accumulation scale with an auto-computed range (via min/max/+-1stStdDev/+-2ndStdDev) to a scale with an explicitly defined range. In this case, the explicit case was not reflected.
- Removes a legacy constraint that prevented you from rendering a query that referenced one or more tables with more than one polygon/multipolygon column.
- Improved speed of server interface using the Thrift binary protocol.
- Data Manager has been redesigned to support HeavyConnect via S3, server file uploads, and expanded raster file support.
- Introduced the new Gauge chart type.
- Introduced a Welcome Panel and Help Center menu.
- Rebranded interface for HEAVY.AI. Updated styles for the default dark and light themes.
- Added option to toggle the legend on the New Combo chart.
- Added configuration option for setting the default chart type.
- Added configuration option for hiding specified chart types.
- Added auto-selection of geo columns and measures on geo chart types.
- Adjusted maximum bins for larger Top-N groups.
- Added support for cross-domain configuration without SSL.
- BETA: Added filter support for global custom expressions.
- BETA: Introduced the new iframe chart type.
- BETA: Introduced Arrow transport protocol for a limited number of chart types.
- Fixed various minor UI and performance issues.
- Fixed parameter creation from dashboard title in Safari browser.
- Fixed displaying of the Jupyter logo when integration is unavailable.