Release Notes
Looking for documentation from a previous release?
5.x/6.x
4.x
3.x
As with any software upgrade, it is important that you back up your data before you upgrade HEAVY.AI. Each release introduces efficiencies that are not necessarily compatible with earlier releases of the platform. HEAVY.AI is never expected to be backward compatible.
For assistance during the upgrade process, contact HEAVY.AI support at [email protected] before you upgrade your system.
- Adds support for POLYGON to MULTIPOLYGON promotion in the load table Thrift APIs and SQLImporter.
- Fixes an issue that caused an intermittent KafkaImporter crash on CentOS 7.9.
- Fixes an issue that cause incorrect results in multiple aggregation of date columns that include COUNT DISTINCT.
- Adds support for limiting the number of charts in a dashboard through the
ui/limit_charts_per_dashboardfeature flag. The default value is 0 (no limit).
- Adds a new set of log-based (request_logs, server_logs, web_server_logs, and web_server_access_logs) system tables.
- Adds a new Request Logs and Monitoring dashboard.
- Adds a new SHOW CREATE SERVER command, which displays the create server DDL for a specified foreign server.
- Adds support for non-super-user execution of SHOW CREATE TABLE on views.
- Adds a new ALTER SESSION SET EXECUTOR_DEVICE command, which updates the type of executor device (CPU or GPU) for the current session.
- Adds a new ALTER SESSION SET CURRENT_DATABASE command, which updates the connected database for the current session.
- Adds a new ALTER DATABASE OWNER TO command, which allows super users to change the owner of a database.
- Extends the INSERT command to support inserting multiple rows at once/batch insert.
- Add support for default values on shard key columns.
- Add initial support for Window function framing, including support for BETWEEN ROWS clause for all numeric and date/time types, and BETWEEN RANGE clause for numeric types.
- Enable group-by push down for UNION ALL such that group-by and aggregate operations applied to the output of UNION ALL are evaluated on the UNION ALL inputs, improving performance.
- Add support for LCASE (alias for LOWER), UCASE (alias for UPPER), LEFT, and RIGHT string functions
- Adds a new trim_spaces option for delimited file import.
- (BETA) Adds data import/COPY FROM support from Relational Database Management Systems and Data Warehouses using the Open Database Connectivity (ODBC) interface.
- Initial support for CUDA streams to parallelize GPU computation and memory transfers.
- Increase per-GPU projection limit with watchdog enabled from 32M to 128M rows to take advantage of improvements in large projection support in recent releases.
- Add new SHOW FUNCTIONS and SHOW FUNCTIONS DETAILS commands to show registered compile-time UDFs and extension functions in the system and their arguments, and SHOW RUNTIME FUNCTIONS [DETAILS] and SHOW RUNTIME TABLE FUNCTIONS [DETAILS] to show user-defined runtime functions/table functions.
- Support timestamp inputs and outputs for table functions.
New Extension Functions
- Add tf_compute_dwell_times table function, that given a query input input with entity keys and timestamps**,** and parameters specifying the minimum session time, minimum number of session records, and max inactive seconds, outputs all unique sessions found in the data with the duration of the session (dwell time).
- Add tf_feature_self_similarity table function, that given a query input of entity keys/IDs, a set of feature columns, and a metric column, scores each pair of entities based on their similarity, computed as the cosine similarity of the feature column(s) between each entity pair, which can optionally TF/IDF weighted.
- Add tf_feature_similarity table function, that given a query input of entity keys, feature columns, and a metric column, and a second query input specifying a search vector of feature columns and metric, computes the similarity of each entity in the first input to the search vector based on their similarity, computed as the cosine similarity of the feature column(s) for each entity with the feature column(s) for the search vector, which can optionally be TF/IDF weighted.
- Fixed an issue where some join queries on ODBC-backed foreign tables can return empty result sets for the first query.
- Fixed an issue where append refreshes on foreign tables backed by delimited or regex-parsed files ignore file-path filter and sort options.
- Fixed a crash that can occur when very large dates are specified for the refresh_start_date_time foreign table option.
- Fixed a crash that can occur when a foreign table’s data source is updated within a refresh window.
- Fixed an issue where databases owned by deleted user accounts are not visible, and adds a restriction that prevents dropping users who own databases.
- Fixed an issue where joins on string dictionary-encoded columns would hit spurious none-encoded string translation
- Fixed issue with certain UNION ALL query patterns, such as UNION ALL containing logical values.
- Disabled KEY_FOR_STRING for UNNEST operations on string dictionary-encoded columns, to prevent a crash.
- Fixed an issue where logged stats for raster imports could overflow.
- Fixed an issue where joins on synthetic tables (for example, created with a VALUES statement or table function without an underlying table) could crash.
- Fixed an issue where require checks used on string dictionary inputs to a table function could crash.
- Fixed a crash and/or wrong query results that can occur when a decimal literal is used in a nested query.
- Fixed a potential crash when attempting to auto-retry a render immediately after an OutOfGpuMemory exception is thrown. This crash can occur only if the
render-oom-retry-thresholdconfiguration option is set. - Fixed a regression where polygons with transparent colors are rendered opaque.
- Corrects an issue with point/symbol rendering with explicitly Vega projections where the projection was not being updated when panned/zoomed if the query did not change.
- Significant improvements in hit-testing consistency and stability when rendering queries with subqueries, window functions, or table functions.
- Font size controls.
- Borders and Zebra Striping in Table charts.
- Justify content in Table charts.
- Customization polygon border control.
- Allow measure date formatting for table charts.
- Extend y-axis on Vega combo charts to end at the next whole value past the highest data point.
- Add layer visibility toggle to kebab dropdown on multi-layer raster charts.
- Made unsaved changes modal less aggressive.
- Custom Source Table Functions Browser
- Don’t show unsaved warning modal after adding default filter set.
- (BETA) PostgreSQL connector.
- Allows maxBounds to be set in servers.json.
- Toggle dashboard unsaved when updating annotations.
- Dashboard save state behavior fixes.
- Table Chart order by group keys when present.
- Use key_for_string when ordering by known dictionary measures/dimensions.
- Add default formatting for date/time on table chart.
- Add admin feature flag to hide key manager.
- Customizable polygon border color and existing border bug fixes.
- Cannot append to table using PostgreSQL connector.
- Building a raster chart with the layer visibility toggle feature flag enabled causes a crash.
- Support for fast string functions on dictionary-encoded text columns (the default), including LOWER, UPPER, INITCAP, TRIM/LTRIM/RTRIM, LPAD/RPAD, REVERSE, REPEAT, SUBSTRING/SUBSTR, REPLACE, OVERLAY, SPLIT_PART, REGEXP_REPLACE, REGEXP_SUBSTR, AND CONCAT (||). The output of these expressions can be chained, grouped-by, and used in both the left and right side of join predicates.
- Support for fast string equality/inequality operations without the previous requirement of watchdog disablement when the two columns do not share dictionaries.
- Support for fast case statements with multiple text column inputs that do not share dictionary-encoded strings.
- Support for ENCODE_TEXT to encode none-encoded strings, which can then be grouped on and manipulated like dictionary-encoded strings. This operator is not intended for interactive use at scale but instead for ELT-like scenarios. Use the new server flag
watchdog-none-encoded-string-translation-limitto set the upper cardinality allowed for such operations (1,000,000 by default). - Support for UNION ALL is enabled by default, and now works across string columns that do not share dictionaries with significantly better performance than in the previous release.
- Window functions now support expressions in the PARTITION BY and ORDER clauses.
- Support for subqueries in CASE statement clauses.
- SHOW USER DETAILS is changed to only list those users with access to the currently-selected database. Previously, all users on the HeavyDB instance would be listed; this is still available to superusers with SHOW ALL USER DETAILS.
- 10X improvements in initial join performance (including geo joins) through faster, parallelized hash table construction, removing redundant inter-thread hash table computation.
- Improved join ordering to avoid loop joins in certain scenarios.
- Parallel compilation of queries as well as inter-executor generated code increases concurrency and throughput in common, Immerse-driven scenarios by up to 20%. Also decreases latency for a single user interacting with dashboards or issuing SQL queries in a way that required new plans to be code-generated.
- New result set recycler allows query substeps (expensive in subqueries) can be cached using SQL hints ( /*+ keep_result */), dramatically improving performance where the subquery is reused across multiple queries (for example, in Immerse) and only outer steps of the query vary.
- The default for the header option of
COPY TOto a CSV/TSV file has been changed from'false'to'true'. - Faster dictionary map in StringDictionaryProxy, accelerating various string operations involving transient entries.
- Arrow execution endpoints now use multiple executors and can run concurrently like queries issued to the Thrift endpoints.
- Addition of sparse dictionary output capability for Arrow queries, which automatically creates a subset of a string dictionary to send via Arrow when it detects that it is faster than sending the full, unfiltered dictionary. This provides orders-of-magnitude better server- and client-side performance and scalability for common cases where large dictionary-encoded text columns are filtered or top-k sorted such that only a small subset of dictionary entries are needed in the result set.
- ST_INTERSECTS now can operate directly on top of compressed (the default) coordinates, leading to 2-3X increase in speed.
- New table function framework allows for both system and user-defined table functions. Table functions can run on both CPU and GPU and are designed for efficient, scalable execution of custom algorithms in-situ on data that might be hard or impossible to implement in SQL.
- Support for generate_series table function (similar to Postgres) for easy and fast integer series generation, particularly useful for left joins against binned tables to fill in gaps, whether for visualization or downstream operations like window functions, and generate_random_strings for generation of string columns of a user-defined size and cardinality.
- Support for geo_rasterize and geo_rasterize_slope table functions to efficiently bin vector data into gap-free bins, with the optional ability to fill in null values, apply box blur, and compute slope and aspect ratios
- Initial support for HeavyRF, a module that allows for real-time, ray-based computation of signal propagation, taking inputs of both terrain data and real or hypothetical signal sources.
- Beta support for Python-defined scalar (row-level) and tabular User Defined Functions (UDFs and UDTFs), using the RBC library to translate Numba python code into LLVM IR that is JITed into query execution code for fast, scalable, custom user-defined capabilities.
- Complete redesign and rewrite of Parquet import to one that is more robust, efficient, and performant.
- Adds support for import from regex parsed files on either the server file system or S3 using the COPY FROM command.
- The
geoandparquetWITHoptions ofCOPY FROMhave been deprecated and replaced bysource_type. Using the deprecated syntax generates the following:Deprecation Warning: COPY FROM WITH (geo='true') is deprecated. Use WITH (source_type='geo_file') instead.Update any scripts you have to replace the deprecated syntax with the new syntax. For more information, see CSV/TSV Import. - (BETA) Adds support for import from RDMS/data warehouses using the COPY FROM command.
- A new default information_schema database contains 10 new system tables that provide information regarding CPU/GPU memory utilization, storage space utilization, database objects, and database object permissions.
- New system dashboards that enable intuitive visualization of system resource utilization and user roles and permissions.
- Support for Zarr and NetCDF raster file import.
- You can now import raster files with ground control points geospatial references.
- Support for file path filtering, globbing, and sorting when importing geo and raster files.
- Improved error messaging when attempting to save a dashboard that uses a duplicate dashboard name.
- Support for connections to delimited files on either the server file system or S3. S3 support includes an option to use the S3 Select API, which provides better performance but with limitations on supported column types.
- Support for connections to Parquet files on either the server file system or S3. HeavyConnect leverages Parquet metadata to provide efficient data access and row group-level filter push down.
- Parquet column type coercion. Convert Parquet column types to more memory-efficient HeavyDB column types for use cases that guarantee no loss of information.
- Connections to regex parsed files on either the server file system or S3. This enables you to query unstructured text files, such as logs, by specifying regular expression patterns that extract components of the text files into table columns.
- (BETA) Support for connections to Relational Database Management Systems and Data Warehouses, leveraging the Open Database Connectivity (ODBC) interface to provide seamless access to data.
- (BETA) ODBC column type coercion. Use HeavyConnect to convert ODBC column types to more memory-efficient HeavyDB column types for use cases that guarantee no loss of information.
- Support for scheduled data refreshes. Specify a start date time and interval at which connected data gets refreshed.
- Adds support for disk level caching. By default, data fetched by HeavyConnect are cached at the disk level in addition to normal CPU/GPU level caching. This provides better overall query performance for network based connections, such as S3, and systems with limited CPU/GPU memory capacity. Disk cache size and level can be set through HeavyDB server configuration.
- Adds support for file path filtering, globbing, and sorting for Parquet, delimited, and regex parsed file use cases.
- Complete redesign and rewrite of the Parquet detect_column_types Thrift API. The Parquet detect/data preview feature is now more robust, efficient, and performant.
- Change to query interrupt mechanism allowing certain classes of queries such as loop joins to be easily and quickly interrupted.
- Fixed crash that could occur with joins on predicates that had functions on the left hand side expression, i.e. geoToH3.
- Fix crash that could occur with Arrow queries that did not return results.
- Avoid building metadata for empty result sets.
- Fixes a crash that can occur when executing queries on GPU that involve a baseline group by and variable length column projections.
- Fixes some table query concurrency bottlenecks. Previously, queries such as INSERT, TRUNCATE, and DROP TABLE required system wide locks to execute and would therefore block execution of other unrelated queries. These kinds of queries can now be executed concurrently.
- Fixes a crash that can occur on server restart when the disk cache is enabled and tables with cached data are deleted.
- Fixes a crash that can occur when the max_rows table option is altered for an empty table.
- Fixes an issue in the JDBC driver where tables from multiple databases are listed even when a single database is specified.
- Fixes an issue where raster POINT column type import would incorrectly throw an exception.
- Fixes a crash that can occur when restoring a dump for a table with previously deleted columns.
- Updates the export COPY TO command to include headers by default.
- Removes the file_type parameter from the create_table Thrift API. This parameter was not used.
- Fixes a crash that can occur when executing SQL commands containing comments.
- Fixed the setting for default database (DEFAULT_DB) being ignored in a SAML login for a user who already exists.
- The OpenGL renderer driver has been fully removed as of this release. Vulkan is the only available driver and enables a more modern, flexible API. As a result, the
renderer-use-vulkan-driverprogram option has been removed. Remove any references to that program option from your configuration files. For more on the move to the Vulkan driver, see Vulkan Renderer. - A novel polygon rendering algorithm is now used as the default when rendering polygons. This algorithm does no triangulation nor does it require “render groups” (a hidden column to assist the old polygon rendering algorithm). However, the render groups column is still added on import as a fallback. See Importing Geospatial Data for more on render group deprecation.
- You can now hit-test certain render queries with subqueries more effectively. For example, if the subquery is only used for filter predicates, renders should now be sped up and hit-testing more flexible.
- Render times are now being logged correctly (“render_vega-COMPLETED nonce:2 Total Execution: (ms), Total Render: (ms)”). The execution time and render time were incorrectly logged as 0 in Releases 5.9 and 5.10
- Fixes a regression introduced in Release 5.10.0 when hit-testing an Immerse cohort-generated query. The hit-test would result in an error such as the following:
“Cannot find column in hit-test cache for query …” - Resolves a crash when trying to hit-test render queries with window functions or cursorless table functions.
- Fixes an issue where a multi-layer, multi-GPU render with a poly or line mark as the first layer can result in ghosting artifacts if the query associated with that layer resulted in 0 rows.
- Fixes an issue when switching between a density accumulation scale with an auto-computed range (via min/max/+-1stStdDev/+-2ndStdDev) to a scale with an explicitly defined range. In this case, the explicit case was not reflected.
- Removes a legacy constraint that prevented you from rendering a query that referenced one or more tables with more than one polygon/multipolygon column.
- Improved speed of server interface using the Thrift binary protocol.
- Data Manager has been redesigned to support HeavyConnect via S3, server file uploads, and expanded raster file support.
- Introduced the new Gauge chart type.
- Introduced a Welcome Panel and Help Center menu.
- Rebranded interface for HEAVY.AI. Updated styles for the default dark and light themes.
- Added option to toggle the legend on the New Combo chart.
- Added configuration option for setting the default chart type.
- Added configuration option for hiding specified chart types.
- Added auto-selection of geo columns and measures on geo chart types.
- Adjusted maximum bins for larger Top-N groups.
- Added support for cross-domain configuration without SSL.
- BETA: Added filter support for global custom expressions.
- BETA: Introduced the new iframe chart type.
- BETA: Introduced Arrow transport protocol for a limited number of chart types.
- Fixed various minor UI and performance issues.
- Fixed parameter creation from dashboard title in Safari browser.
- Fixed displaying of the Jupyter logo when integration is unavailable.
- The COPY TO command now exports time, date, and timestamp data types in ISO 8601 format. Previously, date and timestamp data types were exported as unix epochs.
- Fixed crash that could occur when window function partitions were sorted on multiple columns.
- Fixed crash that could occur when in-situ queries had query step punted to CPU.
- Fixed crash that could occur when columnar output is forced for queries projecting variable length types, either by command-line flag or query hint.
- Fixes hit-testing issues in single-node elated to rendering SQL with aggregate subqueries. Such queries could be generated using the cohort feature in Immerse or using custom dimensions. Previously, you could see errors such as the following in map charts:
Cannot find column <column name> in hit-test cache for query <sql query>.Most errors of that form should be resolved in single-node.
- Fixed New Combo Chart legend appearing over chart bars.
- Fixed 'Failed to Load Tables' error when navigating to the data manager.
- Fixed issue where projections with more than 1M rows of geo and other variable-length types could crash the server when –
columnar-large-projectionsis enabled. - Fixed issue with rendering results of certain subqueries when running distributed.
- Fixed data manager error occuring when global custom expressions beta feature is enabled.
- Upgrade log4j to 2.17.1 to address CVE-2021-44832.
The SAML Entitlement feature is deprecated and may be removed in a future release. Row-level security (RLS), with a role specified through SAML, provides an improved replacement.
- Row-level security (RLS): Administrators can use new commands CREATE POLICY, SHOW POLICIES, and DROP POLICY to apply security filtering to queries run as a user or with a role.
- Significantly more performant, parallelized window functions, executing up to 10X faster than in Release 5.9.
- Automatic use of columnar output (instead of the default row-wise output) for large projections, particularly benefitting window functions, subqueries, and table function calls that returned large numbers of rows, lowering query times by 5-10X in some cases. The threshold defaults to 1M rows, but can be modified with the
columnar-large-projections-thresholdflag (and turned off withcolumnar-large-projections=false). - UNION ALL now works with tables that do not share dictionary columns and supports query patterns such as grouped inputs. Previously only projections were supported.
- Large IN subqueries are now significantly faster and scalable by rewriting the associated query plans as decorrelated joins.
- Added support for full set of ST_TRANSFORM SRIDs supported by geos/proj4 library.
- Significant ST_DWithin speedups involving LINESTRING types via an optimization to clip the checked region of the LINESTRING to just the portion that can possibly fall within the specified distance of the other geometry.
- Removes a previous restriction where null values were not allowed for text array columns.
- Adds ANY_VALUE as canonical SQL alias for existing SAMPLE aggregate operator.
- Support for full set of ST_TRANSFORM SRIDs supported by geos/proj4 library.
- High performance UTM transforms (ST_TRANSFORM) available to and from web mercator and geographic coordinate systems.
- Significant ST_DWithin speedups involving LINESTRING types via an optimization to clip the checked region of the LINESTRING to just the portion that can possibly fall within the specified distance of the other geometry.
- Adds support for import from dozens of image and raster file types, such as jpeg, png, geotiff and ESRI grid, including remote files.
- Adds support for numerous vector GIS files (100+ formats supported by current GDAL release).
- Adds support for multidimensional array import from GRIB/GRIB2 formats common in science and meteorology.
When importing GRIB/GRIB2 raster files, the default multithreaded import behavior is unstable due to a bug in the GDAL library; this issue will be addressed in the next release. You must use the
WITH option threads=1 to force single-threaded operation and avoid a server crash. This restriction does not apply to any other raster formats.- Adds support for selected layers or bands within multilayer files.
- Adds a new flag to SQLImporter that allows narrowing of integer values.
- Improves point import speed regardless of source.
- Extended file metadata and column detection in omnisql to include additional formats.
- Fixes an issue in the ODBC driver where some SQL statements containing double quotes would result in an error.
- Fixes an issue where decimal value imports could fail, even when the specified number of maximum rejections has not been reached.
- Fixes a crash that occurs when the max rows option is set for an empty table.
- Fixes a potential ghosting artifact when rendering consecutive accumulation renders in a multi-gpu system using the OpenGL driver.
- Introduces dashboard-level Named Custom SQL.
- Improves Table Chart export to access all data represented by a table chart.
- Adds overridable CSS file (override.css) for custom CSS in embedded implementations.
- Panning support in the New Combo chart with alt+scroll.
- New color picker user interface and made other UI improvements.
- Alerts for unsaved dashboard changes.
- Geo-joined Choropleth charts now spatially cross-filter the fact table on zoom and pan.
- Expanded functionality of the Iframe API (Beta).
- Introduces optional higher data throughput Arrow Transport (Beta).
- Expanded functionality for Embedded UI Customization (Beta).
- Adds database-level Global Custom Expressions (Beta).
- Adds cohort support for crosslinking (Beta).
- Fixed application of Configuration UI changes to discretion charts ’ legends.
- Fixed custom SQL expression support for custom data sources.
- Fixed various cases for upgrading legacy charts to the New Combo chart.
- Fixed various minor UI issues.
- Adds IF EXISTS support to DROP USER and DROP ROLE statements.
- Significant speedup for POINT and fixed-length array imports and CTAS/ITAS--generally 5-20X faster.
- Fixes an issue where null values and empty arrays were not imported correctly for text array columns.
- Correctly account for compressed size of 8-bit and 16-bit dictionary encoded text columns and all geo type columns to avoid overly aggressive punting of queries to CPU when columns of these types are used in a query.
- Prevent overflow of columns with widths less than 4 bytes when columnar projection used in a query, which could lead to a crash.
- Improves the performance of density accumulation stats gathering when using min/max/1stStdDev/2ndStdDev ranges, up to 50x speedups in some instances.
- The PNG encoding step of a render request is no longer a blocking step, thereby improving render concurrency.
- Fixes a potential timeout in the Vulkan renderer in a multi-GPU server configuration if GPU contention caused by many queries/renders executing at the same time exists. The timeout can manifest as a VulkanDeviceLost error that can lock up renders for a minute before the renderer can recover.
- Fixes a minor memory leak when caching parsed Vega JSONs.
- Fixes a possible ghosting artifact in multilayer/multi-GPU renders when individual layers use different subsets of available GPUs when rendered.
- BETA - Adds custom expressions to table columns.
- BETA - Adds Crosslink feature with Crosslink Panel UI.
- BETA - Adds Custom SQL Source support and Custom SQL Source Manager.
- Adds support to hide deprecated chart types from add/edit chart menu.
- Improved speed of user role verification.
- Fixes minor issues in the Parameter manager UI.
- Fixes New Combo chart binning migration.
- Fixes issues in the global side navigation.
- Fixes fetching a cohort count when a custom SQL filter contains a parameter.
- Fixes Vega chart color-utils to always fetch latest color palettes.
- Fixes filter removal after changing and resetting the default value if you use a SQL custom dimension as a parameter.
- Fixes minor UI styling issues.
- Fixes custom source selection in Vega source selector.
- Fixes cross-filter support with geo-joined bounding boxes.
- Fixes resetting of parameter definitions when canceling from inside the chart editor view.
- Fixes new cases of chart title helper support.
- Fixes New Combo Bar label automatic formatting.
- Added a new SHOW ROLES command for viewing directly assigned and effective user roles.
- Fixed a regression in Release 5.8.0 where purging idle render sessions can result in server crashes with the OpenGL renderer.
- Fixes a regression in Release 5.8.0 where poly renders can result in segfaults.
- User role checks that block dashboard load and login are completed much more quickly.
Release 5.8 officially enables the new Vulkan backend renderer as the default, replacing the OpenGL renderer. For more information on Vulkan, reasons for the change, and troubleshooting, see Vulkan Renderer.
The legacy OpenGL backend renderer is still available as a fallback, but will be deprecated and removed in subsequent releases, so it is advisable to use the legacy OpenGL renderer only if you have a blocking issue with Vulkan. To disable the Vulkan renderer and enable OpenGL, add the following flag to your server configuration:
renderer-use-vulkan-driver=falseIn distributed clusters, the configuration file for each node must have this parameter set to use OpenGL.
Releases 5.7 and higher require the installation of a Vulkan API loader library to support the Vulkan renderer. This library is required regardless of whether you are using the Vulkan renderer or not. If you are upgrading from Release 5.7, you have already installed this library and no further action is required.
If you see an error similar to the following when trying to start a renderer-enabled Release 5.8 server for the first time, you need to install the Vulkan API loader:
error while loading shared libraries: libvulkan.so.1: cannot open shared object file: No such file or directoryTo install the Vulkan API loader:
- On CentOS:
sudo yum install vulkan - On Ubuntu:
sudo apt install libvulkan1
For a summary of how to install the Vulkan loader on other various Linux distributions, see https://linuxconfig.org/install-and-test-vulkan-on-linux.
- Parallel executors now on by default (with default
--num-executors=2) - Spatial joins between point types using ST_Distance type are now accelerated using overlaps join hash framework, with increased in speed up to 100x.
- Window functions can run on empty partitions and can operate over tables with multiple fragments and shards.
- Queries that need to run a query step on CPU due to memory pressure or compatibility reasons now execute only that individual step on CPU. Previously, the full query was restarted and all steps ran on CPU.
- Support WIDTH_BUCKET operator for easier numeric binning.
- Natively support ST_Transform to/from all UTM zones and EPSG:4326 (Lon/Lat) and EPSG:900913 (Web Mercator).
- Support provided for file path regex filter and for file path sort order when running the COPY FROM command.
- New ALTER SYSTEM CLEAR commands enable clearing CPU or GPU memory.
- Error messages for DUMP/RESTORE commands are improved.
- Validations and error messages are improved when specifying default column values.
- More robust handling is added for long decimal strings during import.
- Non-superusers can view and interrupt their own queries.
- Rewrote query plans where certain aggregates are performed on the same expression that is grouped-by (that is, COUNT DISTINCT column where column is being grouped) to improve performance for these queries
- Fixed an issue where window functions with a preceding filter would pull all of a table’s columns into memory, even if unused by the query.
- Disabled window functions for updates because it could crash the server. Targeted to be fixed and enabled in a future release.
- Various fixes related to aggregate window functions.
- Improvements to auto-casting logic for extension functions, user-defined row functions, and table functions.
- Avoid unnecessary translation between dictionary-encoded text columns for certain classes of hash joins when the inner and outer tables’ join columns share the same dictionary, thereby improving performance.
- Fixed crash that could occur when performing a join on multifragment input when the hash table is built on GPU but the join is executed on CPU.
- Various bug fixes and performance improvements for runtime query interrupt.
- Fixed broken Immerse dashboard import in omnisql.
- The default renderer is now Vulkan. The legacy OpenGL renderer can still be used as a fallback. See the notice at the top of these release notes for more information.
- Improved the memory footprint/performance of the multi-GPU compositor by doing overlapped, tiled transfers from render GPUs to the compositor GPU. The more GPUs on the node, the bigger potential gains.
- Added extra logging in the event of a timeout or Vulkan device lost error.
- Migrated numeric binning in Immerse to use the new WIDTH_BUCKET SQL operator, significantly improving binning/histogram performance in some cases.
- BETA: Added a new global side navigation.
- BETA: Added support for 3D terrain in 3D scatter chart.
- BETA: Support for hiding Immerse UI elements in iframed app usage is added.
- BETA: Crossfilter referencing in custom filters is supported.
- Fixed an issue where the parameters dropdown would not autocomplete.
- Fixed incorrect file headings in chart map export.
- Fixed multilayer legend value for Choropleth chart with geo heat map.
- Fixed display of long dashboard titles.
- Fixed dashboard duplication permission case.
- Fixed hit-test error on a custom measure.
- Fixed SQL syntax error on Pointmap chart with custom SQL group-by dimension.
- Fixed rendering of Scatter Plot and Pointmap charts after hovering over points.
- Fixed various minor UI bugs affecting overall usability.
- Fixed SAML login to a specified database based on login URL with database name.
- Fixed role retrieval for usernames with @ and other symbols ($ & + , ; : / = ?).
Query Capabilities
- Added support for default column values. When creating tables or adding new columns to existing tables, you can now specify column default values. This works in both SQL and with Thrift APIs.
- Added support for APPROX_QUANTILE, with performance and functionality similar to APPROX_MEDIAN, which is APPROX_QUANTILE called with a 50% quantile argument.
- Per-kernel interrupt performance significantly improved, now on by default. Queries can be interrupted using Ctrl + C in omnisql, or by calling the interrupt API.
Performance
- The Arrow data frame API now differentiates between where the query is run and where the Arrow result is requested. For example, a query that must run on CPU can be pushed to GPU for Arrow IPC sharing, and a query that would normally run on GPU can still run on GPU, even if the user requests the Arrow buffer in CPU memory via IPC sharing.
- Improved performance of high-cardinality group-by queries. Large cohorts in Immerse should show substantial performance improvement.
- Parallel executors now in public beta (set with
--num-executorsflag). In future versions of OmniSci, the number of executors will be set to 2 by default.
Administrative
- Added a new REASSIGN OWNED command, which changes ownership of database objects (tables, views, dashboards, etc.) from a user or set of users in the current database to a different user.
- Added a new SHOW USER DETAILS command for introspecting user information.
- Error messages have been updated to remove redundancy and otherwise extraneous content.
Geospatial
- Added support to ST_Transform for transformations from 900913 (web mercator) to 4326 (lon/lat).
- ST_Contains and ST_Intersect joins and filters can now run fully accelerated on dynamically constructed points using ST_Point(lon, lat).
- Geospatial projection support for points is now more robust, faster, and widely supported. For example, ST_SetSRID(ST_Point(id,id),4326) is now supported in projections.
- Major enhancements to ODBC driver, including support for Geospatial types.
- Added FlatGeoBuf import/export support, which is about 3x faster than shapefile and 8x faster than geojson.
- Improved handling of geospatial columns in intermediate results and temporary tables. Previously, a "columnar conversion not supported" error would be thrown in joins involving multi-fragment geospatial tables. Now, the geospatial column can be zipped up, allowing joins and similar operations to proceed.
Enhanced AWS Permissions and Session Management
- Added support for using IAM roles or server permissions when importing data from AWS S3. Admins can enable the use of IAM roles when running on an EC2 instance. Credentials can also be configured on the server either through AWS environment variables or credential files. Enable this option with the
allow-s3-server-privilegesserver configuration. - Added support for AWS session tokens through omnisql and Thrift import APIs.
- Improved messaging related to an error that can occur when the sample data insert script downloads data to a path outside the server data directory.
- Fixed an issue where the server can encounter an error on startup due to a pre 4.0 release migration bug.
- Fixed a race condition that can occur when SELECT queries and auto-vacuuming execute concurrently.
- Projections without limits now should not re-compile if query literals change, significantly increasing performance
- Arrow over-the-wire query requests were always being executed on CPU, now they will run on default device type (and can be overridden by the user with a query hint)
- Fix crash that could occur with empty Arrow result sets
- omnisql now returns error status codes if a single command is called and the command fails. This allows omnisql to be more easily embedded into scripts with error checking.
- Resolved an issue where join table reordering could fail for some geojoins, resulting in an error message or crash.
- Resolved a crash that occurred when inserting a NULL value into a geospatial column in distributed mode.
- Various bug fixes and performance improvements for geospatial types in Insert Into As Select / Create Table As Select queries in distributed mode.
- Resolved a crash that occurred when updating a BOOLEAN array column.
- Resolved null handling issues when grouping by BIGINT columns.
- Resolved an issue which prevented CURRENT_TIME() or NOW() from being used in aggregate queries.
- Resolved an issue involving CASE statements with string literals in one of the case branches, where the returned results could be incorrect.
- Comparison between full array and indexed array columns will throw an appropriate error. Previously, SQL with such a comparison could cause the server to crash.
- Resolved an issue where NULLS LAST in an ORDER BY clause could cause an incorrect ordering with respect to sign (e.g. negatives before positives).
- Resolved an issue where CTAS/ITAS queries with small returns in the SELECT statement could enter an infinite loop in distributed mode.
- Resolved an issue preventing the owner of a database from dropping that database.
Release 5.7 includes the official beta release of the new Vulkan-backed renderer, which will replace the current OpenGL renderer. For more information, see Vulkan and the reasons we are making the switch, see Vulkan Renderer (Beta).
Because Vulkan will be the default renderer in Release 5.8, OmniSci strongly recommends using it now. The OpenGL renderer will be deprecated and removed in subsequent releases. The Vulkan renderer is reliable and stable, and switching to Vulkan now can help reveal any unforeseen issues. Finding such issues early, while an easy fallback exists, ensures a smooth, less risky transition.
To enable the beta Vulkan renderer, set the
renderer-use-vulkan-driver configuration parameter to true.In distributed clusters, the configuration file for each node must have the parameter set.
The Vulkan library is required for Release 5.7, regardless of which renderer you use. If you do not install the renderer, you will see the following error when trying to start a renderer-enabled server for the first time:
error while loading shared libraries: libvulkan.so.1: cannot open shared object file: No such file or directoryIf you receive this error, you need to install the Vulkan API loader:
- CentOS:
sudo yum install vulkan - Ubuntu:
sudo apt install libvulkan1
For a summary of how to install the Vulkan loader on various Linux distributions, see: https://linuxconfig.org/install-and-test-vulkan-on-linux.
- Significant speedups in large polygon renders across multiple GPUs. The larger the table, and the more GPUs, the bigger the speed increase.
- Memory footprint improvements in compositing and anti-aliasing components.
- 3-4x speedup with procedural rendering (i.e. “symbol” vega mark type)
- Better pipelining of compressed geo columns and projection math; 2x memory footprint improvement in use cases
- Added a new Airplane symbol shape for Vega that can be used in symbol/legacy symbol marks.
- Significant enhancements to Immerse parameters.
- Dashboard parameter widgets:
- Side-panel parameter controllers can be added to dashboards.
- Supports sophisticated custom dashboards with no code required.
- Parameter value display across the product:
- Chart titles and chart axes labels
- Dashboard titles
- Tooltips and Legends
- Convenience methods to use parameters within:
- Chart column selectors
- Simple filters
- Quick filters
- Enhanced parameter management:
- Show/hide hidden parameters
- Improved user interface for tracking parameters usage
- Allow parameter usage in Demo mode
- Improved parameter selection autocomplete in all chart types
- Improvements in Geo charts:
- Ability to set top color category in Pointmaps.
- Improved gradient selection in Pointmaps
- Support Zoom Level in Map Chart `Zoom to` Field
- High-precision, higher-performance lasso tool:
- Now uses an
ST_Containsfilter expression for drawn polygons instead of expensive inside-triangle expressions. - Employs a dynamic level-of-detail drawing algorithm that automatically adjusts the resolution of parts of a polygon/circle that are in view so that it matches the results of the
ST_Containsfilter exactly for map charts.
- 3D Pointmap chart (Beta release).
- Embed HTML in a text chart (Beta release).
- Updated mapd-connector with latest version of Thrift.
- Unique category support for larger unique category sets.
- Apply numeric filters using parameters.
- Fixed display of parameters on new dashboards after a session timeout.
- Fixed Choropleth hit test when hovering over a polygon.
- Fixed display of boolean values in the input field of a global filter and dropdown menu.
- Fixed parameter support in Choropleth charts with a geo join.
- Fixed reversed axes labels on New Combo chart.
- Fixed BIGINT column support in cohort builder.
- Fixed display of parameter value in filter component input.
- Fixed boolean setting of column value parameters.
- Upgrade charts with column parameters set as dimension.
- Fixed errant session clearing for non-autologin instances.
- Fixed missing support for map-move crossfilter feature flags in Combo chart.
- Fixed imperial formatting in popups.
- Fixed a multilayer popup visibility when hovering over a line or point.
- Various minor UI improvements and bug fixes.
OmniSci release 5.6.4 includes a fixed issue.
- Resolved an issue where cleaning up an expired session could cause the server to crash.
OmniSci release 5.6.3 includes a new feature and an improvement.
- ITAS and CTAS queries are now protected from entering an infinite loop when small numbers of records are returned.
- Support for encrypted credentials in connectors is decoupled from authentication through Immerse.