Getting Started
Last updated
Last updated
HeavyRF is a separately licensed module addition to the HEAVY.AI Enterprise platform Release 6.3.0 and higher. To obtain a license, contact your Account Executive.
Set ui/enable_custom_source_manager to true in Heavy Immerse to enable creation of custom data sources.
Use of a special parameter ${Crossfilter.tablename} allows access to any crossfilter in SQL. This is commonly used to access crossfilters for antennas (filter out specific antennas). However if you wish to interactively test the importance of different clutter parameters, ${Crossfilter} can also be used on the clutter table.
Advanced knowledge of the HEAVY.AI platform, data science, and ETL workflows and tools.
ETL workflows and tooling are not covered in this topic.
Obtain terrain or clutter data. This can be as simple as pure terrain elevations or as complicated as raw LiDAR data.
Obtain antenna data. Minimum requirement is data with 2D positions and an antenna identifier.
Define scalar parameters. Scalar parameter defaults are used when data is missing. Example, for antenna data, decide on defaults for height, frequency, power, and angle (heading and tilt).
The data described above is used to generate the HeavyRF table function SQL query consisting of two cursors (antenna and terrain) and a number of scalars.
You can generate the SQL manually as defined in HeavyRF Table Functions and apply it as a custom source on the Pointmap chart, or by using the Pointmap chart Custom Source Editor in Immerse (Pointmap > Sources > Select Data Source > +Create Custom Source > Transforms > tf_rf_prop or tf_rf_prop_max_signal).
The following is a Pointmap chart generated using the tf_rf_prop_max_signal table function.
Terrain elevation data is currently acquired using three different techniques, resulting in data with varying characteristics:
Additional sources of terrain data are generated from LiDAR flown on planes. Because this has required dedicated flights and relatively expensive sensor systems, it has not been widely available until recently. However, in 2022 the USGS completed coverage of most of the continental US, including all major urban areas, in a program known as 3DEM. This data varies in resolution, but is guaranteed to be at least 9 points per square meter, or approximately 1 foot in horizontal resolution. In addition, it can distinguish building and tree canopies from "bare earth."
LiDAR-based elevation models are the most accurate and can be used to distinguish buildings and vegetation directly, so are generally preferred where available. They are available across the US and in many other developed countries, but generally not yet in the developing world.
Terrain Recommendation: Use LiDAR-based terrain data, which provides the best accuracy, is generally available, and can distinguish between buildings, tree canopies, and "bare earth". LiDAR data may require additional ETL processes prior to importing/linking.
USGS National LIDAR data archive on AWS S3 (recommended source)
Terrain Tiles (multisource) on AWS
Terrain Processing Methods
Conventional terrain data is normally stored in raster file formats such as geoTIFF. These can be directly read into all releases of HEAVY.AI since 5.11.
You can use two methods to make this data available for use in HEAVY.AI:
Import using standard COPY FROM statements or Heavy Immerse data connectors.
Linked to the data using HeavyConnect. Linking to the data creates a cache and can be used for datasets larger than available GPU RAM.
In HEAVY.AI release 6.1.0 and later, file 'globbing' is only supported locally. This means that if you have an archive with 1000 terrain files on s3, you cannot access it using syntax such as "COPY terrain FROM 's3://my-bucket/.tif"; you must first copy the files locally. Then, _"_COPY terrain FROM '/my/local/path/.tif" will work.
Compression of multiple raster files in a single archive is not supported. This is not an issue with any of the sources mentioned above.
Boundary clipping of data is not supported during import. Logic for selecting tiles and clipping must generally be done either in a script or with regex path filters.
Many forms of terrain elevation data are currently available. Most of these can be directly read by HEAVY.AI, but are sometimes organized in ways that require some scripting to efficiently download and ingest.
Specify UDTF Terrain Parameters
This example includes high-resolution building height data from photogrammetry and LiDAR in the 'sf_bld_heights_m' table.
The UDTF currently requires that latitude and longitude be specified using columns. Use the OGC standard ST_X and ST_Y functions to extract longitude and latitude from the geom point column.
Antenna files are extremely diverse, with varying levels of detail, and have not been standardized across the industry. For global coverage, there is an open data project known as "Open Cell ID" that provides basic information. However, it does not indicate carrier, frequency, or power characteristics. These files are generally rather small. Python provides a wide set of methods to clean up and normalization them into formats better-suited for a relational database.
The minimum requirements are data with 2D positions and an antenna identifier.
HEAVY.AI expects antenna power in dBm, but some sources provide it in watts.
The power P(dBm) in dBm is equal to 10 times the base 10 logarithm of 1000 times the power P(W) in watts (W) divided by 1 watt (W):
dbm = 10 * log10(1000 * watts)
You can more easily perform this conversion using Pandas instead of in the database.
Configure the Antenna Cursors
The RF Propagation UDTF requires a set of antennas. These have at least an identifier, which can be anything unique, and a location in 2D.
XY can be specified in geographic coordinates of longitude and latitude, or in planar coordinates. A boolean scalar parameter called geographic_coords (see below) specifies which type of coordinates are being used.
Similarly, there are two options for antenna height specification: as height above ground in meters ("AGL") or as absolute height in meters above mean sea level ("AMSL"). The boolean scalar parameter rf_source_z_is_relative_to_terrain specifies which of these you are using, which typically depends on data available. In this example, the City of San Francisco provided AGL heights.
All antennas can have a default antenna Tx power and Frequency, which is convenient if, for example, you are planning for a specific type of equipment rollout. However, these can also optionally be specified per antenna. To do so, add tx_power_watts and/or tx_freq_mhz to the antenna cursor. This also depends on available data. In representing current conditions with a variety of antenna types, the optional attributes are preferred.
Review the schema for your table and make sure you have at least XY, and optionally Z, power and frequency per antenna.
Specify UDTF Antenna Parameters
Required parameters
Optional parameters
The third major configuration step is to set the scalar parameters.
The first two parameters are metadata for the data already specified above (terrain and antenna cursors). Because raw values are passed into the UDTF, no horizontal or vertical SRID information is otherwise available.
The next important and powerful parameter determines the output simulation horizontal resolution in meters. Generally, this needs to be smaller than the distance between major obstacles to avoid artifacts. Computational complexity of the simulation increases with the square of this value, as do data volumes. A pragmatic compromise may be necessary, depending on available time and memory.
Three other parameters control the complexity of the simulation. First, you can bound the maximum distance computed from any antenna. This is generally a function of frequency and urban density. Higher frequencies such as 5G mmWave in more urban areas can be well simulated with a max_ray_travel_meters of a few hundred meters to a few kilometers. Midrange frequencies such as 4G LTE may require up to dozens of kilometers, especially in suburban or rural areas.
A basic optimization of the simulation is to not calculate signal too weak to be of interest. This varies with cell phone technology generation and receiver characteristics, so here we default to -120dBm. However for 3G or 4G signal and older phones, a value of -100dBm might be sufficient and save computation time.
As described above, antennas can all have individual heights, if provided in the antenna cursor. However, when not provided, or when there is no data for a particular observation, a fallback value can be provided here.
Because a "ray tracing" technique is used to simulate, an important parameter is the number of rays emitted per source. For an omni-directional antenna, 360 rays are needed for 1 degree of horizontal precision, or 720 rays for 1/2 degree. For a tri-sector antenna with 120 degree horizontal coverage angle, 120 rays would provide 1 degree of horizontal precision.
To increase ray tracing efficiency, you can automatically generate rays when simulated bins are not getting appropriate coverage. These advanced parameters do not normally need adjustment.
Add the combined UDTF to the Custom Source of the Pointmap Chart to generate the HeavyRF simulation. All Pointmap chart features are available with HeavyRF.
Each section of the UDTF can be executed individually. Verify the parameters in each section using Jupyter notebook before proceeding to the next.
Start with minimal clutter data to ensure the SQL and antenna data is correct in the UDTF. Layer on increasingly complex clutter data to generate the desired RF propagation.
To generate complex HeavyRF scenarios, contact your Account Executive.
