Getting Started with Open Source

HeavyDB is an in-memory, column store, SQL-based, relational database designed from the ground up to run on GPUs. Developers are encouraged to contribute to this Open Source project to expand and enhance HeavyDB capabilities.

Licensing

HEAVY.AI Open Source is licensed under the Apache License, Version 2.0.

Third-Party Licenses

The repository includes a number of third party packages provided under separate licenses. Details about these packages and their respective licenses are at ThirdParty/licenses/index.md.

Cloning

You can clone a copy of HeavyDB from GitHub ( https://github.com/omnisci/heavydb).

Installing

Click the link for instructions on how to use a script to install HeavyDB dependencies for your chosen platform.

• Installing CentOS 7 Dependencies

• Installing macOS Dependencies

• Installing Ubuntu Dependencies

• Installing Dependencies from Arch User Repository

This is the complete list of HeavyDB dependencies.

This list describes the required dependencies, but you do not have to install these dependencies individually. Follow the links above for instructions on how to configure HeavyDB dependencies for your chosen platform.

Package	Min Version	Required
CMake	3.3	yes
LLVM	3.8-4.0, 6.0	yes
GCC	5.1	no, if building with clang
Go	1.6	yes
Boost	1.65.0	yes
OpenJDK	1.7	yes
CUDA	8.0	yes, if compiling with GPU support
gperftools		yes
gdal	2.3	yes
Arrow	0.11.0	yes

Dependencies for omnisci_web_server and other Go utilities are in the ThirdParty/go directory. See ThirdParty/go/src/mapd/vendor/README.md.

Installing CentOS 7 Dependencies

HeavyDB requires a number of dependencies that are not provided in the common CentOS/RHEL package repositories. A prebuilt package containing all of these dependencies is provided for CentOS 7 (x86_64).

Use the scripts/mapd-deps-prebuilt.sh build script to install prebuilt dependencies.

These dependencies are installed in a directory under /usr/local/mapd-deps. The mapd-deps-prebuilt.sh script also installs Environment Modules in order to simplify managing the required environment variables. Log out and log back in after running the mapd-deps-prebuilt.sh script in order to active Environment Modules command, module.

The mapd-deps environment module is disabled by default. To activate for your current session, run the following command:

module load mapd-deps

To disable the mapd-deps module, run the following command:

module unload mapd-deps

The mapd-deps package contains newer versions of packages such as GCC and ncurses that might not be compatible with the rest of your environment. Disable the mapd-deps module before compiling other packages.

CUDA

It is best to install CUDA and the NVIDIA drivers using the .rpm, using the instructions provided by NVIDIA. The preferred .rpm (network) method ensures you always have the latest stable drivers, while the .rpm (local) method allows you to install without Internet access.

The .rpm method requires DKMS to be installed, which is available from the Extra Packages for Enterprise Linux (EPEL) repository:

sudo yum install epel-release

Reboot after installing to activate the NVIDIA drivers.

Environment Variables

The mapd-deps-prebuilt.sh script includes two files with the appropriate environment variables:

• mapd-deps-<date>.sh (for sourcing from your shell config)

• mapd-deps-<date>.modulefile (for use with Environment Modules, yum package environment-modules).

  These files can be found in the mapd-deps install directory, usually /usr/local/mapd-deps/. Either of these can be used to configure your environment: the .sh can be sourced in your shell config, while the .modulefile must be moved to the modulespath.

Building Dependencies

Use the scripts/mapd-deps-centos.sh script to build the dependencies. Modify this script and run it if you want to change dependency versions or to build on alternative CPU architectures.

cd scripts
module unload mapd-deps
./mapd-deps-centos.sh --compress

This completes installation of dependencies. Go to the Building section to continue.

macOS

The shell script scripts/mapd-deps-osx.sh automatically installs and/or updates Homebrew, then uses it to install all dependencies. macOS must be completely up to date and Xcode must be installed before you run the script. You can find and install Xcode on the Apple App Store.

CUDA

mapd-deps-osx.sh automatically installs CUDA via Homebrew and adds the correct environment variables to your ~/.bash_profile.

Java

scripts/mapd-deps-osx.sh automatically installs Java and Maven via Homebrew and adds the correct environment variables to your ~/.bash_profile.

This completes installation of dependencies. Go to the Building section to continue.

Installing Ubuntu Dependencies

Most build dependencies required by HeavyDB are available via APT. Certain dependencies such as Thrift, Blosc, and Folly must be built as they either do not exist in the default repositories or have outdated versions. A prebuilt package containing all these dependencies is provided for Ubuntu 18.04 (x86_64). The dependencies are installed to /usr/local/mapd-deps/ by default; see Environment Variables for how to add these dependencies to your environment.

Ubuntu 16.04

HeavyDB requires a newer version of Boost than the version provided with Ubuntu 16.04. The scripts/mapd-deps-ubuntu1604.sh build script compiles and installs a newer version of Boost into the /usr/local/mapd-deps/ directory.

Ubuntu 18.04

Use the scripts/mapd-deps-prebuilt.sh build script to install prebuilt dependencies. These dependencies are installed to a directory under /usr/local/mapd-deps. The mapd-deps-prebuilt.sh script generates a script named mapd-deps.sh containing the required environment variables. Source this file in your current session (or symlink it to /etc/profile.d/mapd-deps.sh) to activate it:

source /usr/local/mapd-deps/mapd-deps.sh

Some installs of Ubuntu 18.04 might fail while building with a message similar to:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

This is a known issue in 18.04 that should be resolved in Ubuntu 18.04.1. To resolve on 18.04:

sudo rm /etc/ssl/certs/java/cacerts
sudo update-ca-certificates -f

Environment Variables

Add the CUDA and mapd-deps lib directories to LD_LIBRARY_PATH; add the CUDA and mapd-deps bin directories to PATH. The mapd-deps-ubuntu.sh and mapd-deps-prebuilt.sh scripts generate a script named mapd-deps.sh containing the environment variables that need to be set. Source this file in your current session (or symlink it to /etc/profile.d/mapd-deps.sh) to activate it:

source /usr/local/mapd-deps/mapd-deps.sh

CUDA

Recent versions of Ubuntu provide the NVIDIA CUDA Toolkit and drivers in the standard repositories. Use the following command to install:

sudo apt install -y nvidia-cuda-toolkit

Reboot your system after install to activate the NVIDIA drivers.

Building Dependencies

The scripts/mapd-deps-ubuntu.sh and scripts/mapd-deps-ubuntu1604.sh scripts build the dependencies for Ubuntu 18.04 and 16.04, respectively. The scripts install all required dependencies (except CUDA) and build the dependencies that require it. Modify this script and run it if you want to change dependency versions or to build on alternative CPU architectures.

cd scripts
./mapd-deps-ubuntu.sh --compress

This completes installation of dependencies. Go to the Building section to continue.

Installing Dependencies from Arch User Repository

The following uses yaourt to install packages from the Arch User Repository.

yaourt -S \
    git \
    cmake \
    boost \
    google-glog \
    extra/jdk8-openjdk \
    clang \
    llvm \
    thrift \
    go \
    gdal \
    maven

VERS=1.21-45
wget --continue https://github.com/jarro2783/bisonpp/archive/$VERS.tar.gz
tar xvf $VERS.tar.gz
pushd bisonpp-$VERS
./configure
make -j $(nproc)
sudo make install
popd

CUDA

You can install CUDA and the NVIDIA drivers with the following commands:

yaourt -S \
    linux-headers \
    cuda \
    nvidia

Reboot after install to activate the NVIDIA drivers.

Environment Variables

You must add the CUDA bin directories to your PATH variable. A convenient way to do so is to create a new file, /etc/profile.d/mapd-deps.sh that contains the following:

PATH=/opt/cuda/bin:$PATH
export PATH

This completes installation of dependencies. Go to the Building section to continue.

Building

OmniSci uses CMake as its build system. The following commands create and navigate to a new build directory, set the build type to debug, then build the OmniSci application using up to four simultaneous jobs.

cd ~/heavydb // Go to the root directory of your HeavyDB clone.
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=debug ..
make -j 4

You can use the following optional configuration parameters to enable or disable features in your build.

-DCMAKE_BUILD_TYPE=release	Build type and compiler options to use. Options are debug, release, RelWithDebInfo, MinSizeRel, and unset
-DENABLE_ASAN=off	Enable address sanitizer. Default is off.
-DENABLE_AWS_S3=on	Enable AWS S3 support, if available. Default is on.
-DENABLE_CALCITE_DELETE_PATH=on	Enable Calcite Delete Path. Default is on.
-DENABLE_CALCITE_UPDATE_PATH=on	Enable Calcite Update Path. Default is on.
-DENABLE_CUDA=off	Disable CUDA. Default is on.
-DENABLE_CUDA_KERNEL_DEBUG=off	Enable debugging symbols for CUDA kernels. Will dramatically reduce kernel performance. Default is off.
-DENABLE_DECODERS_BOUNDS_CHECKING=off	Enable bounds checking for column decoding. Default is off.
-DENABLE_FOLLY=on	Use Folly. Default is on.
-DENABLE_IWYU=off	Enable include-what-you-use. Default is off.
-DENABLE_JIT_DEBUG=off	Enable debugging symbols for the JIT. Default is off.
-DENABLE_PROFILER=off	Enable google perftools. Default is off.
-DENABLE_STANDALONE_CALCITE=off	Require standalone Calcite server. Default is off.
-DENABLE_TESTS=on	Build unit tests. Default is on.
-DENABLE_TSAN=off	Enable thread sanitizer. Default is off.
-DENABLE_CODE_COVERAGE=off	Enable code coverage symbols (clang only). Default is off.
-DENABLE_JAVA_REMOTE_DEBUG=on	Enable Java Remote Debug. Default is off.
-DMAPD_DOCS_DOWNLOAD=on	Download the latest master build of the documentation / omnisci.com/docs. Default is off. Note: this is a >50MB download.
-DPREFER_STATIC_LIBS=off	Static link dependencies, if available. Default is off.

Testing

HeavyDB uses Google Test as its main testing framework. Tests reside under the Tests directory.

The sanity_tests target runs the most common tests. If using Makefiles to build, you can run the tests using the following command:

make sanity_tests

AddressSanitizer

You can activate AddressSanitizer by setting the ENABLE_ASAN CMake flag in a fresh build directory. At this time, CUDA must also be disabled. In an empty build directory run CMake and compile:

mkdir build && cd build
cmake -DENABLE_ASAN=on -DENABLE_CUDA=off ..
make -j 4

Now you can run the tests:

export ASAN_OPTIONS=alloc_dealloc_mismatch=0:handle_segv=0
make sanity_tests

ThreadSanitizer

You can activate ThreadSanitizer by setting the ENABLE_TSAN CMake flag in a fresh build directory. At this time, you must also disable CUDA. In an empty build directory, run CMake and compile:

mkdir build && cd build
cmake -DENABLE_TSAN=on -DENABLE_CUDA=off ..
make -j 4

OmniSci uses a TSAN suppressions file to ignore warnings in third party libraries. Source the suppressions file by adding it to your TSAN_OPTIONS env:

export TSAN_OPTIONS="suppressions=/path/to/omnisci/config/tsan.suppressions"

Now you can run the tests:

make sanity_tests

Packaging

HeavyDB uses CPack to generate packages for distribution. Packages generated on CentOS with static linking enabled can be used on most other recent Linux distributions. To generate packages on CentOS (assuming starting from top level of the heavydb repository), execute the following commands:

mkdir build-package && cd build-package
cmake -DPREFER_STATIC_LIBS=on -DCMAKE_BUILD_TYPE=release ..
make -j 4
cpack -G TGZ

The first command creates a fresh build directory, to ensure there is nothing left over from a previous build.

The second command configures the build to prefer linking to the dependencies' static libraries instead of the (default) shared libraries, and to build using CMake's release configuration (enables compiler optimizations). Linking to the static versions of the libraries reduces the number of dependencies you must install on target systems.

The last command generates a .tar.gz package. You can replace TGZ with, for example, RPM or DEB to generate a .rpm or .deb, respectively.

Using

You can use the startomnisci wrapper script to start HeavyDB in a testing environment. The script performs the following tasks:

• Initializes the data storage directory via initdb, if required.

• Starts the main HeavyDB server, omnisci_server.

• Offers to download and import a sample dataset, using the insert_sample_data script, if the --sample-data flag is provided.

If you are in the build directory, and it is a subdirectory of the heavydb repository, you can run startomnisci with the following command:

../startomnisci

Starting Manually

If you prefer to use the startup commands individually, run the following commands from the build directory.

Initialize the data storage directory. You only run this command one time.

mkdir data && ./bin/initdb data

Start the HeavyDB server:

./bin/omnisci_server

Optionally, insert a sample dataset by running the insert_sample_data script in a new terminal:

../insert_sample_data

You can now start using the database. You can use the omnisql utility to interact with the database from the command line:

./bin/omnisql -p HyperInteractive

where HyperInteractive is the default password. The default user (omnisci) is used if you do not provide a user name. See omnisql.

Coding

Your contributed code should compile without generating warnings by recent compilers on most Linux distributions. Changes to the code must follow the C++ Core Guidelines.

clang-format

A .clang-format style configuration, based on the Chromium style guide, is provided at the top level of the repository. Format your code using a recent version (6.0+ preferred) of ClangFormat before you submit your changes.

To use clang-format:

clang-format -i File.cpp

clang-tidy

A .clang-tidy configuration is provided at the top level of the repository. Lint your code using a recent version (6.0+ preferred) of clang-tidy before you submit your changes.

clang-tidy requires all generated files to exist before you run the utility. The easiest way to accomplish this is to run a full build before you run clang-tidy. OmniSci provides a build target that runs clang-tidy.

To use clang-tidy:

make clang-tidy

• clang-tidy might make invalid or overly verbose changes to the source code. OmniSci recommends that you first commit your changes, run clang-tidy, then review its recommended changes before you amend them to your commit.

• The clang-tidy target uses the run-clang-tidy.py script provided with LLVM, which might depend on PyYAML. The target also depends on jq, which is used to filter portions of the compile_commands.json file.

Contributing

OmniSci welcomes and encourages your contributions to our Open Source project. You can read the roadmap for ideas on where you might contribute, and review the list of known issues and enhancements you can implement immediately.

Contributor License Agreement

To verify the intellectual property license granted with contributions from any person or entity, OmniSci must have a Contributor License Agreement (CLA) on file signed by each contributor. After you make a pull request, a bot will notify you if a signed CLA is required and provide instructions for how to sign it. Read the agreement carefully before signing and keep a copy for your records.