Comment on page
Release 5.x
Archived release notes for Release 5.x
- The COPY TO command now exports time, date, and timestamp data types in ISO 8601 format. Previously, date and timestamp data types were exported as unix epochs.
- Fixed crash that could occur when window function partitions were sorted on multiple columns.
- Fixed crash that could occur when in-situ queries had query step punted to CPU.
- Fixed crash that could occur when columnar output is forced for queries projecting variable length types, either by command-line flag or query hint.
- Fixes hit-testing issues in single-node elated to rendering SQL with aggregate subqueries. Such queries could be generated using the cohort feature in Immerse or using custom dimensions. Previously, you could see errors such as the following in map charts:
Cannot find column <column name> in hit-test cache for query <sql query>.Most errors of that form should be resolved in single-node.
- Fixed New Combo Chart legend appearing over chart bars.
- Fixed 'Failed to Load Tables' error when navigating to the data manager.
- Fixed issue where projections with more than 1M rows of geo and other variable-length types could crash the server when –
columnar-large-projectionsis enabled. - Fixed issue with rendering results of certain subqueries when running distributed.
- Fixed data manager error occuring when global custom expressions beta feature is enabled.
- Upgrade log4j to 2.17.1 to address CVE-2021-44832.
The SAML Entitlement feature is deprecated and may be removed in a future release. Row-level security (RLS), with a role specified through SAML, provides an improved replacement.
- Row-level security (RLS): Administrators can use new commands CREATE POLICY, SHOW POLICIES, and DROP POLICY to apply security filtering to queries run as a user or with a role.
- Significantly more performant, parallelized window functions, executing up to 10X faster than in Release 5.9.
- Automatic use of columnar output (instead of the default row-wise output) for large projections, particularly benefitting window functions, subqueries, and table function calls that returned large numbers of rows, lowering query times by 5-10X in some cases. The threshold defaults to 1M rows, but can be modified with the
columnar-large-projections-thresholdflag (and turned off withcolumnar-large-projections=false). - UNION ALL now works with tables that do not share dictionary columns and supports query patterns such as grouped inputs. Previously only projections were supported.
- Large IN subqueries are now significantly faster and scalable by rewriting the associated query plans as decorrelated joins.
- Added support for full set of ST_TRANSFORM SRIDs supported by geos/proj4 library.
- Significant ST_DWithin speedups involving LINESTRING types via an optimization to clip the checked region of the LINESTRING to just the portion that can possibly fall within the specified distance of the other geometry.
- Removes a previous restriction where null values were not allowed for text array columns.
- Adds ANY_VALUE as canonical SQL alias for existing SAMPLE aggregate operator.
- Support for full set of ST_TRANSFORM SRIDs supported by geos/proj4 library.
- High performance UTM transforms (ST_TRANSFORM) available to and from web mercator and geographic coordinate systems.
- Significant ST_DWithin speedups involving LINESTRING types via an optimization to clip the checked region of the LINESTRING to just the portion that can possibly fall within the specified distance of the other geometry.
- Adds support for import from dozens of image and raster file types, such as jpeg, png, geotiff and ESRI grid, including remote files.
- Adds support for numerous vector GIS files (100+ formats supported by current GDAL release).
- Adds support for multidimensional array import from GRIB/GRIB2 formats common in science and meteorology.
When importing GRIB/GRIB2 raster files, the default multithreaded import behavior is unstable due to a bug in the GDAL library; this issue will be addressed in the next release. You must use the
WITH option threads=1 to force single-threaded operation and avoid a server crash. This restriction does not apply to any other raster formats.- Adds support for selected layers or bands within multilayer files.
- Adds a new flag to SQLImporter that allows narrowing of integer values.
- Improves point import speed regardless of source.
- Extended file metadata and column detection in omnisql to include additional formats.
- Fixes an issue in the ODBC driver where some SQL statements containing double quotes would result in an error.
- Fixes an issue where decimal value imports could fail, even when the specified number of maximum rejections has not been reached.
- Fixes a crash that occurs when the max rows option is set for an empty table.
- Fixes a potential ghosting artifact when rendering consecutive accumulation renders in a multi-gpu system using the OpenGL driver.
- Introduces dashboard-level Named Custom SQL.
- Improves Table Chart export to access all data represented by a table chart.
- Adds overridable CSS file (override.css) for custom CSS in embedded implementations.
- Panning support in the New Combo chart with alt+scroll.
- New color picker user interface and made other UI improvements.
- Alerts for unsaved dashboard changes.
- Geo-joined Choropleth charts now spatially cross-filter the fact table on zoom and pan.
- Expanded functionality of the Iframe API (Beta).
- Introduces optional higher data throughput Arrow Transport (Beta).
- Expanded functionality for Embedded UI Customization (Beta).
- Adds database-level Global Custom Expressions (Beta).
- Adds cohort support for crosslinking (Beta).
- Fixed application of Configuration UI changes to discretion charts’ legends.
- Fixed custom SQL expression support for custom data sources.
- Fixed various cases for upgrading legacy charts to the New Combo chart.
- Fixed various minor UI issues.
- Adds IF EXISTS support to DROP USER and DROP ROLE statements.
- Significant speedup for POINT and fixed-length array imports and CTAS/ITAS--generally 5-20X faster.
- Fixes an issue where null values and empty arrays were not imported correctly for text array columns.
- Correctly account for compressed size of 8-bit and 16-bit dictionary encoded text columns and all geo type columns to avoid overly aggressive punting of queries to CPU when columns of these types are used in a query.
- Prevent overflow of columns with widths less than 4 bytes when columnar projection used in a query, which could lead to a crash.
- Improves the performance of density accumulation stats gathering when using min/max/1stStdDev/2ndStdDev ranges, up to 50x speedups in some instances.
- The PNG encoding step of a render request is no longer a blocking step, thereby improving render concurrency.
- Fixes a potential timeout in the Vulkan renderer in a multi-GPU server configuration if GPU contention caused by many queries/renders executing at the same time exists. The timeout can manifest as a VulkanDeviceLost error that can lock up renders for a minute before the renderer can recover.
- Fixes a minor memory leak when caching parsed Vega JSONs.
- Fixes a possible ghosting artifact in multilayer/multi-GPU renders when individual layers use different subsets of available GPUs when rendered.
- BETA - Adds custom expressions to table columns.
- BETA - Adds Crosslink feature with Crosslink Panel UI.
- BETA - Adds Custom SQL Source support and Custom SQL Source Manager.
- Adds support to hide deprecated chart types from add/edit chart menu.
- Improved speed of user role verification.
- Fixes minor issues in the Parameter manager UI.
- Fixes New Combo chart binning migration.
- Fixes issues in the global side navigation.
- Fixes fetching a cohort count when a custom SQL filter contains a parameter.
- Fixes Vega chart color-utils to always fetch latest color palettes.
- Fixes filter removal after changing and resetting the default value if you use a SQL custom dimension as a parameter.
- Fixes minor UI styling issues.
- Fixes custom source selection in Vega source selector.
- Fixes cross-filter support with geo-joined bounding boxes.
- Fixes resetting of parameter definitions when canceling from inside the chart editor view.
- Fixes new cases of chart title helper support.
- Fixes New Combo Bar label automatic formatting.
- Added a new SHOW ROLES command for viewing directly assigned and effective user roles.
- Fixed a regression in Release 5.8.0 where purging idle render sessions can result in server crashes with the OpenGL renderer.
- Fixes a regression in Release 5.8.0 where poly renders can result in segfaults.
- User role checks that block dashboard load and login are completed much more quickly.
Release 5.8 officially enables the new Vulkan backend renderer as the default, replacing the OpenGL renderer. For more information on Vulkan, reasons for the change, and troubleshooting, see Vulkan Renderer.
The legacy OpenGL backend renderer is still available as a fallback, but will be deprecated and removed in subsequent releases, so it is advisable to use the legacy OpenGL renderer only if you have a blocking issue with Vulkan. To disable the Vulkan renderer and enable OpenGL, add the following flag to your server configuration:
renderer-use-vulkan-driver=falseIn distributed clusters, the configuration file for each node must have this parameter set to use OpenGL.
Releases 5.7 and higher require the installation of a Vulkan API loader library to support the Vulkan renderer. This library is required regardless of whether you are using the Vulkan renderer or not. If you are upgrading from Release 5.7, you have already installed this library and no further action is required.
If you see an error similar to the following when trying to start a renderer-enabled Release 5.8 server for the first time, you need to install the Vulkan API loader:
error while loading shared libraries: libvulkan.so.1: cannot open shared object file: No such file or directoryTo install the Vulkan API loader:
- On CentOS:
sudo yum install vulkan - On Ubuntu:
sudo apt install libvulkan1
For a summary of how to install the Vulkan loader on other various Linux distributions, see https://linuxconfig.org/install-and-test-vulkan-on-linux.
- Parallel executors now on by default (with default
--num-executors=2) - Spatial joins between point types using ST_Distance type are now accelerated using overlaps join hash framework, with increased in speed up to 100x.
- Window functions can run on empty partitions and can operate over tables with multiple fragments and shards.
- Queries that need to run a query step on CPU due to memory pressure or compatibility reasons now execute only that individual step on CPU. Previously, the full query was restarted and all steps ran on CPU.
- Support WIDTH_BUCKET operator for easier numeric binning.
- Natively support ST_Transform to/from all UTM zones and EPSG:4326 (Lon/Lat) and EPSG:900913 (Web Mercator).
- Support provided for file path regex filter and for file path sort order when running the COPY FROM command.
- New ALTER SYSTEM CLEAR commands enable clearing CPU or GPU memory.
- Error messages for DUMP/RESTORE commands are improved.
- Validations and error messages are improved when specifying default column values.
- More robust handling is added for long decimal strings during import.
- Non-superusers can view and interrupt their own queries.
- Rewrote query plans where certain aggregates are performed on the same expression that is grouped-by (that is, COUNT DISTINCT column where column is being grouped) to improve performance for these queries
- Fixed an issue where window functions with a preceding filter would pull all of a table’s columns into memory, even if unused by the query.
- Disabled window functions for updates because it could crash the server. Targeted to be fixed and enabled in a future release.
- Various fixes related to aggregate window functions.
- Improvements to auto-casting logic for extension functions, user-defined row functions, and table functions.
- Avoid unnecessary translation between dictionary-encoded text columns for certain classes of hash joins when the inner and outer tables’ join columns share the same dictionary, thereby improving performance.
- Fixed crash that could occur when performing a join on multifragment input when the hash table is built on GPU but the join is executed on CPU.
- Various bug fixes and performance improvements for runtime query interrupt.
- Fixed broken Immerse dashboard import in omnisql.
- The default renderer is now Vulkan. The legacy OpenGL renderer can still be used as a fallback. See the notice at the top of these release notes for more information.
- Improved the memory footprint/performance of the multi-GPU compositor by doing overlapped, tiled transfers from render GPUs to the compositor GPU. The more GPUs on the node, the bigger potential gains.
- Added extra logging in the event of a timeout or Vulkan device lost error.
- Migrated numeric binning in Immerse to use the new WIDTH_BUCKET SQL operator, significantly improving binning/histogram performance in some cases.
- BETA: Added a new global side navigation.
- BETA: Added support for 3D terrain in 3D scatter chart.
- BETA: Support for hiding Immerse UI elements in iframed app usage is added.
- BETA: Crossfilter referencing in custom filters is supported.
- Fixed an issue where the parameters dropdown would not autocomplete.
- Fixed incorrect file headings in chart map export.
- Fixed multilayer legend value for Choropleth chart with geo heat map.
- Fixed display of long dashboard titles.
- Fixed dashboard duplication permission case.
- Fixed hit-test error on a custom measure.
- Fixed SQL syntax error on Pointmap chart with custom SQL group-by dimension.
- Fixed rendering of Scatter Plot and Pointmap charts after hovering over points.
- Fixed various minor UI bugs affecting overall usability.
- Fixed SAML login to a specified database based on login URL with database name.
- Fixed role retrieval for usernames with @ and other symbols ($ & + , ; : / = ?).
Query Capabilities
- Added support for default column values. When creating tables or adding new columns to existing tables, you can now specify column default values. This works in both SQL and with Thrift APIs.
- Added support for APPROX_QUANTILE, with performance and functionality similar to APPROX_MEDIAN, which is APPROX_QUANTILE called with a 50% quantile argument.
- Per-kernel interrupt performance significantly improved, now on by default. Queries can be interrupted using Ctrl + C in omnisql, or by calling the interrupt API.
Performance
- The Arrow data frame API now differentiates between where the query is run and where the Arrow result is requested. For example, a query that must run on CPU can be pushed to GPU for Arrow IPC sharing, and a query that would normally run on GPU can still run on GPU, even if the user requests the Arrow buffer in CPU memory via IPC sharing.
- Improved performance of high-cardinality group-by queries. Large cohorts in Immerse should show substantial performance improvement.
- Parallel executors now in public beta (set with
--num-executorsflag). In future versions of OmniSci, the number of executors will be set to 2 by default.
Administrative
- Added a new REASSIGN OWNED command, which changes ownership of database objects (tables, views, dashboards, etc.) from a user or set of users in the current database to a different user.
- Added a new SHOW USER DETAILS command for introspecting user information.
- Error messages have been updated to remove redundancy and otherwise extraneous content.
Geospatial
- Added support to ST_Transform for transformations from 900913 (web mercator) to 4326 (lon/lat).
- ST_Contains and ST_Intersect joins and filters can now run fully accelerated on dynamically constructed points using ST_Point(lon, lat).
- Geospatial projection support for points is now more robust, faster, and widely supported. For example, ST_SetSRID(ST_Point(id,id),4326) is now supported in projections.
- Major enhancements to ODBC driver, including support for Geospatial types.
- Added FlatGeoBuf import/export support, which is about 3x faster than shapefile and 8x faster than geojson.
- Improved handling of geospatial columns in intermediate results and temporary tables. Previously, a "columnar conversion not supported" error would be thrown in joins involving multi-fragment geospatial tables. Now, the geospatial column can be zipped up, allowing joins and similar operations to proceed.
Enhanced AWS Permissions and Session Management
- Added support for using IAM roles or server permissions when importing data from AWS S3. Admins can enable the use of IAM roles when running on an EC2 instance. Credentials can also be configured on the server either through AWS environment variables or credential files. Enable this option with the
allow-s3-server-privilegesserver configuration. - Added support for AWS session tokens through omnisql and Thrift import APIs.
- Improved messaging related to an error that can occur when the sample data insert script downloads data to a path outside the server data directory.
- Fixed an issue where the server can encounter an error on startup due to a pre 4.0 release migration bug.
- Fixed a race condition that can occur when SELECT queries and auto-vacuuming execute concurrently.
- Projections without limits now should not re-compile if query literals change, significantly increasing performance
- Arrow over-the-wire query requests were always being executed on CPU, now they will run on default device type (and can be overridden by the user with a query hint)
- Fix crash that could occur with empty Arrow result sets
- omnisql now returns error status codes if a single command is called and the command fails. This allows omnisql to be more easily embedded into scripts with error checking.
- Resolved an issue where join table reordering could fail for some geojoins, resulting in an error message or crash.
- Resolved a crash that occurred when inserting a NULL value into a geospatial column in distributed mode.
- Various bug fixes and performance improvements for geospatial types in Insert Into As Select / Create Table As Select queries in distributed mode.
- Resolved a crash that occurred when updating a BOOLEAN array column.
- Resolved null handling issues when grouping by BIGINT columns.
- Resolved an issue which prevented CURRENT_TIME() or NOW() from being used in aggregate queries.
- Resolved an issue involving CASE statements with string literals in one of the case branches, where the returned results could be incorrect.
- Comparison between full array and indexed array columns will throw an appropriate error. Previously, SQL with such a comparison could cause the server to crash.
- Resolved an issue where NULLS LAST in an ORDER BY clause could cause an incorrect ordering with respect to sign (e.g. negatives before positives).
- Resolved an issue where CTAS/ITAS queries with small returns in the SELECT statement could enter an infinite loop in distributed mode.
- Resolved an issue preventing the owner of a database from dropping that database.
Release 5.7 includes the official beta release of the new Vulkan-backed renderer, which will replace the current OpenGL renderer. For more information, see Vulkan and the reasons we are making the switch, see Vulkan Renderer (Beta).
Because Vulkan will be the default renderer in Release 5.8, OmniSci strongly recommends using it now. The OpenGL renderer will be deprecated and removed in subsequent releases. The Vulkan renderer is reliable and stable, and switching to Vulkan now can help reveal any unforeseen issues. Finding such issues early, while an easy fallback exists, ensures a smooth, less risky transition.
To enable the beta Vulkan renderer, set the
renderer-use-vulkan-driver configuration parameter to true.In distributed clusters, the configuration file for each node must have the parameter set.
The Vulkan library is required for Release 5.7, regardless of which renderer you use. If you do not install the renderer, you will see the following error when trying to start a renderer-enabled server for the first time:
error while loading shared libraries: libvulkan.so.1: cannot open shared object file: No such file or directoryIf you receive this error, you need to install the Vulkan API loader:
- CentOS:
sudo yum install vulkan - Ubuntu:
sudo apt install libvulkan1
For a summary of how to install the Vulkan loader on various Linux distributions, see: https://linuxconfig.org/install-and-test-vulkan-on-linux.
- Significant speedups in large polygon renders across multiple GPUs. The larger the table, and the more GPUs, the bigger the speed increase.
- Memory footprint improvements in compositing and anti-aliasing components.
- 3-4x speedup with procedural rendering (i.e. “symbol” vega mark type)
- Better pipelining of compressed geo columns and projection math; 2x memory footprint improvement in use cases
- Added a new Airplane symbol shape for Vega that can be used in symbol/legacy symbol marks.
- Significant enhancements to Immerse parameters.
- Dashboard parameter widgets:
- Side-panel parameter controllers can be added to dashboards.
- Supports sophisticated custom dashboards with no code required.
- Parameter value display across the product:
- Chart titles and chart axes labels
- Dashboard titles
- Tooltips and Legends
- Convenience methods to use parameters within:
- Chart column selectors
- Simple filters
- Quick filters
- Enhanced parameter management:
- Show/hide hidden parameters
- Improved user interface for tracking parameters usage
- Allow parameter usage in Demo mode
- Improved parameter selection autocomplete in all chart types
- Improvements in Geo charts:
- Ability to set top color category in Pointmaps.
- Improved gradient selection in Pointmaps
- Support Zoom Level in Map Chart `Zoom to` Field
- High-precision, higher-performance lasso tool:
- Now uses an
ST_Containsfilter expression for drawn polygons instead of expensive inside-triangle expressions. - Employs a dynamic level-of-detail drawing algorithm that automatically adjusts the resolution of parts of a polygon/circle that are in view so that it matches the results of the
ST_Containsfilter exactly for map charts.
- 3D Pointmap chart (Beta release).
- Embed HTML in a text chart (Beta release).
- Updated mapd-connector with latest version of Thrift.
- Unique category support for larger unique category sets.
- Apply numeric filters using parameters.
- Fixed display of parameters on new dashboards after a session timeout.
- Fixed Choropleth hit test when hovering over a polygon.
- Fixed display of boolean values in the input field of a global filter and dropdown menu.
- Fixed parameter support in Choropleth charts with a geo join.
- Fixed reversed axes labels on New Combo chart.
- Fixed BIGINT column support in cohort builder.
- Fixed display of parameter value in filter component input.
- Fixed boolean setting of column value parameters.
- Upgrade charts with column parameters set as dimension.
- Fixed errant session clearing for non-autologin instances.
- Fixed missing support for map-move crossfilter feature flags in Combo chart.
- Fixed imperial formatting in popups.
- Fixed a multilayer popup visibility when hovering over a line or point.
- Various minor UI improvements and bug fixes.
OmniSci release 5.6.4 includes a fixed issue.
- Resolved an issue where cleaning up an expired session could cause the server to crash.
OmniSci release 5.6.3 includes a new feature and an improvement.
- ITAS and CTAS queries are now protected from entering an infinite loop when small numbers of records are returned.
- Support for encrypted credentials in connectors is decoupled from authentication through Immerse.
OmniSci release 5.6.3 includes a new feature and an improvement.
- ITAS and CTAS queries are now protected from entering an infinite loop when small numbers of records are returned.
- Support for encrypted credentials in connectors is decoupled from authentication through Immerse.
OmniSci release 5.6.2 includes new features and improvements and fixes several issues.
- Added a new 'airplane' symbol shape.
- When using INSERT to insert floats into INT columns, the numbers are rounded instead of truncated; now consistent with COPY behavior.
- Fixed an issue where UDFs created at server start time with the
-udfflag were not generating code for the GPU. - Resolved an issue where certain queries with a smaller number of groups ( < 20,000) would fail with a “Ran out of slots in the query output buffer” message.
- Reduced incidence of “Ran out of slots in the query output buffer” errors with large group-bys.
- Improved READ-ONLY configuration to never write to underlying storage.
- Resolved an issue where a left join could get incorrect results if IS NOT NULL was part of the WHERE clause for the query.
- Resolved sort order with aggregated results (negatives before positive) when NULLS LAST is specified.
- Fixed an issue causing a crash when ITAS/CTAS populates a NULL point geometry object.
- Resolved and issue where COPY FROM was restricting all other access in a distributed setup
- Improved AVG handling when NULL values are used.
- Fixed a random segmentation fault that occurred when running render unit tests.
- Immerse now accepts a set of encrypted credentials for secured authentication of a custom application.
- Improved support for creating parameters on a new dashboard.
- Resolved an issue where the record count on a zoomed map would reset after editing.
- Fixed return key support for input of column type parameters.
- Fixed the resetting of column and default value after a parameter is created.
- Fixed caching of the number chart with a chart level filter.
- Resolved issue related to support of Booleans for column value parameters.
- Fixed an issue affecting null column values in quantitative color palette scales for Heat charts.
- Batch upgrading of charts includes additional edge cases.
- Resolved issue related to support for custom color dimensions in New Combo charts.
OmniSci release 5.6.1 includes a new feature and fixes several issues.
- Fixed an issue where a rename of a user with ALTER USER USER RENAME TO USER_NEW could cause issues later when trying to restart the server, depending on role membership.
- Fixed a system crash that occurred during an INSERT TABLE AS SELECT command on a table that does not exist.
- Fixed an issue where polygons from a table which had been appended using SQLImporter might not render. Added command-line option
--noPolyRenderGroups/-nprgto disable render group assignment, to match the available WITH option on other import workflows.
- When basemap sources are not available--for example, if the user is on a private network with no external access--the user is set to offline mode and Immerse uses the lightweight map source bundled with Immerse.
- Fixed an issue to make sure that Combo charts correctly plot on autobinning when the start-of-week flag is set to Saturday or Sunday.
- Duplication of charts with an empty filter is now allowed.
OmniSci release 5.6.0 contains several new features and improvements and fixes a number of issues.
- Allowed import and export paths are now enforced. Confirm that any import or export paths are included via the configuration parameters
allowed-export-pathsandallowed-import-paths. - Loading data now more strictly enforces validation or numeric inputs. Previously, an out-of-range integer would “wrap” and be ingested as incorrect values. In Release 5.6, numerics are validated and if validation fails, the row is rejected.
- String dictionaries do not load if you exceed their capacities; previously, they inserted null for the string.
- Utility jar
mapd-1.0-SNAPSHOT-jar-with-dependencies.jaris renamed. Scripts using SQLimporter need to reference the new .jaromnisci-utility-5.6.0.jar.
- Significant speedups for point-in-polygon joins (ST_CONTAINS, ST_INTERSECTS) via optimized algorithms and range join hash framework; on by default for point-in-polygon joins.
- Approximate median function support through the new
approx_median aggregatefunction. Currently works only in nondistributed deployments and executes on CPU. - Query interrupt is now possible when the query is outside of core kernel execution (for example, copy/import, CTAS, ITAS, query data fetch, result reduction). On by default and controlled by new flag
enable-non-kernel-time-query-interrupt. - Query interrupt during core kernel execution has been optimized so that enabling it with the existing
enable-runtime-query-interruptflag should have little to no impact on query performance; actual impact depends on the query pattern. - Added partial column specification for INSERT and INSERT FROM SELECT, such that data can be inserted and appended to a subset of columns in a table. NULL values are added to columns not specified in the statements.
- Updated the CREATE TABLE command
max_rollback_epochsoption with a default value of 3. This reduces the number of epochs/data versions that are stored. When combined with new space reclamation on delete, optimizes space utilization. Setting the option with a high value stores more epochs and reduces data compaction and space reclamation. To update older tables, use theALTER TABLE <table> SET MAX_ROLLBACK_EPOCHS=<value>command. - Added automatic metadata updates and vacuuming. On UPDATE and DELETE queries, OmniSciDB updates metadata, vacuums, and re-uses space as specified. Automatic metadata updates are turned on by default. See
enable-auto-metadata-updatein Configuration Parameters. Automatic vacuuming on delete occurs when deletes or updates on variable length columns exceed the vacuum threshold. For more information, seevacuum-min-selectivityin Configuration Parameters.
To ensure optimal space usage, ensure the table’s MAX_ROLLBACK_EPOCHS option is set to 3 or lower. This is automatically set by default for the CREATE TABLE command on new tables created in release 5.6 and higher.
For existing databases, OmniSci recommends using
ALTER TABLE <table> SET MAX_ROLLBACK_EPOCHS=3 to cap the space usage, and OPTIMIZE TABLE [<table>] WITH (VACUUM='true'); to reclaim space. This primes the system for efficient space management.- OmniSciDB now enforces permitted filepaths for import/export operations by default. The default import and export paths (
{data directory}/map_importand{data directory}/map_exportrespectively) are allowed by default, so no change is required when doing an import/export from Immerse or for sample data import on server startup. However, when using other commands with user-provided paths (such as COPY FROM) provided paths must be under an allowed root path. See the Configuration Parameters for more information.NOTE: For distributed systems, allowed paths only need to be in the configuration file on the aggregator. However, for consistency and future compatibility, OmniSci recommends that the allowed paths also be added to the leaf omnisci.conf file. - Significantly improved OmniSciDB startup time. Removed extra metadata seeks/reads during startup. Systems with high epoch counts or large column counts will have marked improvements.
- Validation of numeric inputs is more strictly enforced on load. Previously an out-of-range integer would “wrap” and be ingested as some incorrect value. Numeric-typed columns are now validated, and if validation fails, the row is rejected.
- String dictionary reaching capacity during import now throws an error and stops the load instead of inserting nulls in the dictionary-encoded column.
- Significant performance improvements for load into high-cardinality dictionary-encoded text columns via optimized dictionary hash table resizing.
- Improved load performance on systems with many cores. Default max number of threads to use is capped at 32, instead of 2X the number of cores, which can cause contention on systems with many cores.
- Improved the performance of left join queries with filters by searching for left-hand side filters that can be executed before the join, potentially reducing the cardinality of the join.
- Support date format
%m/%d/%ywith 2-digit year on data import. - Improved the performance of common multicolumn sort patterns.
- Chunksize default increased to 2 GB to allow for more records per fragment, particularly when variable-length columns are present (none-encoded strings, arrays, and geospatial types). This allows certain query patterns, such as large joins that might previously span multiple fragments, to run faster.
- Entitlements (introduce simple row level security via SAML attributes).
- Accept session-token when using S3 import, WITH option is s3-session-token.
- Upgrade to calcite 1.25.
- Improved the accuracy of ST_Contains and ST_Intersects at small geospatial scales.
- Allow Narrowing casts in SQLImporter.
- Upgrade to sqlite 3.34.
- Allow SAML and LDAP to provide authentication without having to also provide authorization.
- Utility jar renamed and versioned: omnisci-utility-5.6.0-SNAPSHOT.jar.
- Added support for CURRENT_TIMESTAMP, CURRENT_DATE and CURRENT_TIME functions
- Improved READ-ONLY support.
- UDTFs now allow for a ColumnList argument type, which allows a variable number of arguments to be used for inputs.
- Improved the granularity of the session lock, allowing local users to access the system even if third-party authentication providers have failed or are experiencing high latency.
- Many bug and performance improvements.
- Fixed issue causing a crash when UDTFs were composed.
- Fixed an issue where definition of CPU UDTFs could lead to all subsequent queries running on CPU.
- Resolved an issue where selecting from a view with a self join could cause a crash.
- Improved the ability of the parser to handle complex UPDATE and DELETE queries.
- Added human-friendly error messages to UPDATE and DELETE queries.
- Resolved an issue where specifying warmup queries would cause the aggregator to crash in distributed mode.
In distributed mode, specify warmup queries for the aggregator and all leaf nodes to work correctly.
- Resolved an issue where an equijoin between array columns could crash. Now, an attempt to join throws an exception if the join is not supported.
- Resolved an issue where window functions in a subquery could cause a crash.
- Resolved an issue where window function aggregates could crash if used with filters.
- Improved sorting performance of and resolved a number of stability and correctness issues.
- Resolved an issue in distributed mode where subqueries over replicated tables could return duplicate results.
- Resolved several potential deadlocks between the rendering engine and the query engine when multiple queries or types of queries are in process.
- Fixed an issue where SHOW CREATE TABLE would show the wrong table name after a table with shared dictionaries was renamed.
- Resolved an issue where a COPY TO command with a subquery could fail.
- Improved the quality of density accumulation rendering by incorporating a technique that uses 64-bit atomic operations instead of 32-bit when calculating standard deviation. This allows for a greater range of colors when using statistical measures to color the densities. The 32-bit technique was prone to uncaught overflows that resulted in a much more clamped color range.
- Performance improvement when using a Mercator projection explicitly for marks using the Vega projection property.
- Multi-GPU compositor performance and memory footprint is improved through better pipelining and reduced copy operations.
- New Arrow stock symbol shape can be used in symbol/legacy symbol marks.
- Line and polygon stroke rendering improved by 30-40% by simplifying the math to build the stroke geometry.
- Significant improvements to the quality of point/symbol rendering, achieved by improvements to procedural anti-aliasing technique and to render sample capturing when drawing procedural symbols.
- Polygon geometry imported via the SQLImporter path can now be rendered.
- Fixed a bug where decimals are not properly updated in vega scales after an empty render.
- Fixed a rare CUDA misaligned address crash when rendering multilayer vega with points, geo, and xforms in consecutive vegas.
- Resolved a small host memory leak in the renderer.
- Fixed a polygon render issue where fully transparent polygons can inadvertently affect the renders of visible neighbors.
- Fixed a crash when attempting to render a line mark layer with opacity explicitly set to 0.
- Fixed a potential race condition when DROP or CREATE is run in between renders or while render is in flight. The race condition could result in a crash when executing the get_result_row_for_pixel endpoint.
- Corrected an issue with multilayer rendering with multiple density accumulation layers where all density layers incorrectly applied the color palette from a single layer.
- Fixed a crash that occured when rendering a line mark in a multi-GPU-configured server/leaf when the number of GPUs required to render the lines jumps from one to more-than-one GPU when rendering consecutive vegas.
- Corrected the stroke rendering of ellipse symbols.
- Corrected possible skew in symbol rendering when using the angle property.
- Custom SQL dimensions, measures, and filters can now be parameterized, enabling more flexible and powerful analysis:
- Support for column, attribute, numeric, and custom parameter types
- New Parameter Manager where parameters are created, modified, and deleted.
- Parameter panel on the dashboard supports dynamic edits of the parameter values, and the ability to link/unlink parameters across dashboard tabs.
- All charts have been upgraded with the new custom SQL modal, which contains better SQL validation and ability to inject parameters and columns into the SQL.
- Chart-level error handling and recovery.
- Median added as a supported aggregation type across all charts and aggregate filters.
- A new angle measure has been added to Pointmap and Scatter charts, allowing orientation data to be visualized with wedge and arrow icons.
- When you mouse over a point, line, or poly on Pointmap, Choropleth, or Linemap charts, a popup box appears, where you can copy the popup information to your clipboard. If the popup includes a URL, you can click the URL to open the web page.
- The Other category can be toggled off from the categorical color legend on map charts.
- Immerse charts can now be configured with a Saturday or Sunday start of week by using the servers.json
ui/start_of_weekflag. Supported start of week days include Saturday, Sunday, and Monday (default behavior). - Zoom and panning performance has been significantly improved on New Combo charts.
- A new confirmation modal has been added to the Reset Dashboard Settings action to help prevent accidental loss of work
- On multilayer charts, unchecking layers from the master layer view now persists those changes and prevents unchecked layers from being displayed.
- Fixed issue on charts where click-based cross-filters from binned time dimensions were applied incorrectly.
- Fixed issue on Table charts where the chart did not highlight the row correctly after filtering on a boolean value.
- Fixed issues where Map and Stacked Bar chart data did not refresh correctly with the dashboard set to auto-refresh.
- Fixed issue in the global filter panel where the calendar picker was hidden behind charts when the dashboard was scrolled down.
- Fixed issue on map charts where color legends were not displaying correctly for multilayer charts.
- Fixed issue in New Combo charts where brush filters were removed when filter sets were switched.
- Resolved issue in New Combo charts where the range and focus charts were out of sync when there were multiple sources on the chart.
- Fixed issue in New Combo charts where newly created columns were not available for data selection.
- Fixed issue on new combo charts where x-axis labels were not updating correctly when sorting time columns.
- On New Combo charts, brush filters are now set correctly on duplicated charts.
- Fixed issues on Table charts where formatting was applied incorrectly to custom measures and date dimensions.
- Resolved issue on New Combo charts where bars with certain custom SQL measures in percentage mode did not render correctly.
OmniSci release 5.5.6 fixes an issue.
- License key input removes extraneous characters from pasted text.
OmniSci release 5.5.5 includes a new feature and fixes an issue.
- Align license management for Enterprise Edition and Free Edition.
- Resolved sign-on issues after idle session timeout expiry.
OmniSci release 5.5.2 includes new features and fixes several issues.
- SQLImporter allows text-based WKT or WKB fields to be imported into an appropriate geo column.
- A new command-line option, log-user-id, enables logging internal numeric user IDs instead of textual user names.
- OmniSciDB Arrow distribution is upgraded to v2.0.0.
- Resolved an issue where concurrent CTAS runs could deadlock.
- Resolved an issue where replacing an existing dashboard could cause the server to crash.
- Corrected behavior of multicolumn sort with NULL values.
- Resolved an issue when using ANY with a decimal array.
- Resolve an issue when loading BIGINT data into an ARRAY.
- Resolved an issue when using OFFSET with no LIMIT.
- Fixed incorrect left join-to-inner join optimization rule.
- Improved handling of CASE statement literals when dictionaries are used.
- Improved overflow checking.
- Fixed issues for persistence of deleted rows after OPTIMIZE.
- Improved locking for CONNECT to be less blocking.
- Fix a multilayer Vega render bug in which two consecutive renders with minimal changes in the code (for example, a change in layer ordering for) can result in a cryptic exception, but the Vega code is otherwise legal.
OmniSci release 5.5.1 includes a new feature and fixes several issues.