Migrating Vega Code to Dynamic Poly Rendering

In Release 5.2, the polygon cache for rendering was deprecated and will be completely removed in a subsequent release. Any poly cache rendering in your code must be reworked to use dynamic poly rending. This topic describes how to migrate poly cache code to dynamic, cacheless rendering.

Caching poly buffers has two main drawbacks, both of which have a significant impact on memory:

• The cache cannot span multiple GPUs.

• The entire table is cached, regardless of the filter in the query.

In contrast, dynamic poly rendering can utilize all available GPUs and only uses the data that passes any filters.

To move to dynamic poly rendering, determine if you are using the poly cache, and then adjust your code if needed.

HEAVY.AI strongly recommends that you also remove the render-poly-cache-bytes option from your server configuration file, if used. This will help prevent startup warnings or errors in subsequent releases of HEAVY.AI.

Am I Using the Poly Cache?

It may not be immediately obvious if you are using poly cache rendering because no flag is used to enable it. Instead, poly cache rendering is enabled according to the SQL code used in a poly-formatted data block of Vega code. If the query ultimately projects or results in a POLYGON/MULTIPOLYGON column, the cache is not used and no code changes are requried.

However, if the query does not reference a POLYGON/MULTIPOLYGON column, but projects a rowid column, then poly caching is in use.

The following Vega code has a poly query that uses the cache system:

...,
"data": [ 
    {
      "name": "polys", 
      "format": "polys", 
      "sql": "SELECT rowid from zipcodes WHERE ST_XMax(heavyai_geo) > -160.0 AND ST_XMin(heavyai_geo) <
              160.0 AND ST_YMax(heavyai_geo) > -60.0 AND ST_YMin(heavyai_geo) < 60.0"
     }
   ], 
...

The sql property of the polys data block, which uses "format": "polys", projects a rowid column. This activates poly caching, even though the geo column (heavyai_geo in this case) is used in the filter.

To convert this Vega code to dynamic, cacheless rendering, change the SQL query:

"sql": "SELECT heavyai_geo from zipcodes WHERE ST_XMax(heavyai_geo) > -160.0 AND ST_XMin(heavyai_geo) < 
        160.0 AND ST_YMax(heavyai_geo) > -60.0 AND ST_YMin(heavyai_geo) < 60.0"

You can keep rowid in this query and use it for later hit-testing. It's presence does not affect dynamic rendering. See the simple projection query in Example 1 for an example.

Alternatively, you can check the INFO logs to see if poly caching is used. Look for a LOG statement similar to the following:

Caching <number> bytes (<number> for vbo/ibo) for poly query: <sql> on gpu <number>
Poly cache on gpu <number> now using <number> of <number> bytes.

or

Cannot cache <number> bytes (<number> for vbo/ibo) for poly query: <sql> on gpu <number>. 
There is currently <number> of <number> total bytes used in the poly cache.

If you find either of these, then a poly cache render query is used.

Migrating Your Code

To migrate from poly cache to dynamic poly rendering, you output a POLYGON/MULTIPOLYGON column in the SQL query of a poly-formatted data block of your Vega code. In the following examples, the heavyai_geo column is a MULTIPOLYGON.

Example 1 - Simple projection query

Cached (only rowid is output):

..., 
"data": [
     {
       "name": "polys",
       "format": "polys",
       "sql": "SELECT rowid from zipcodes WHERE ST_XMax(heavyai_geo) > -160.0 AND ST_XMin(heavyai_geo) < 
               160.0 AND ST_YMax(heavyai_geo) > -60.0 AND ST_YMin(heavyai_geo) < 60.0"
     }
   ],
 ...

Dynamic (heavyai_geo is now projected, along with rowid):

..., 
"data": [
     {
       "name": "polys",
       "format": "polys",
       "sql": "SELECT heavyai_geo, rowid from zipcodes WHERE ST_XMax(heavyai_geo) > -160.0 AND ST_XMin(heavyai_geo) < 
               160.0 AND ST_YMax(heavyai_geo) > -60.0 AND ST_YMin(heavyai_geo) < 60.0"
     }
   ], 
...

Example 2 - Join query using a WITH subquery

Cached (no geo column is projected in the outer query of the join):

...,
"data": [ 
     {
       "name": "choropleth",    
       "format": "polys",
       "sql": "WITH colors AS (SELECT pickup_building as key0, avg(tip_amount/fare_amount) as color 
               FROM taxi_factual_closestbuilding WHERE (fare_amount > 0 AND tip_amount >= 0 AND tip_amount/fare_amount<1) 
               GROUP BY key0) SELECT nyc_buildings.rowid, colors.key0 as key0, colors.color as color 
               FROM colors, nyc_buildings WHERE (colors.key0 = nyc_buildings.bld_id) AND 
               (ST_XMax(nyc_buildings.heavyai_geo) >= -74.06993970987278 AND 
               ST_XMin(nyc_buildings.heavyai_geo) <= -73.87446838332451 AND 
               ST_YMax(nyc_buildings.heavyai_geo) >= 40.64111330399987 AND 
               ST_YMin(nyc_buildings.heavyai_geo) <= 40.87934187208219)"
     }
   ],
...

Dynamic (the geo column is projected in place of rowid):

...,
"data": [
     {
       "name": "backendChoropleth",
       "format": "polys",
       "sql": "WITH colors AS (SELECT pickup_building as key0, avg(tip_amount/fare_amount) as color 
               FROM taxi_factual_closestbuilding WHERE (fare_amount > 0 AND tip_amount >= 0 AND tip_amount/fare_amount<1) 
               GROUP BY key0) SELECT nyc_buildings.heavyai_geo as heavyai_geo, colors.key0 as key0, colors.color as color 
               FROM colors, nyc_buildings WHERE (colors.key0 = nyc_buildings.bld_id) AND 
               (ST_XMax(nyc_buildings.heavyai_geo) >= -74.06993970987278 AND 
               ST_XMin(nyc_buildings.heavyai_geo) <= -73.87446838332451 AND 
               ST_YMax(nyc_buildings.heavyai_geo) >= 40.64111330399987 AND 
               ST_YMin(nyc_buildings.heavyai_geo) <= 40.87934187208219)"
     }
   ], 
...