Skip to content

Amazon Redshift Integration


Amazon Redshift is a cloud-based data warehouse service designed for large data data sets stored by a column-oriented DBMS principle. Amazon Redshift integrates easily with Row64 by wiring to Row64 RamDb through Python.

Integration Overview

The basic connection process is to use row64tools to push updates from Amazon Redshift to Row64. An overview is available here:
https://pypi.org/project/row64tools/

If sub-second speeds are required, you might want to look at connecting using real-time streaming, which is covered under the streaming help:
Row64 Stream Overview V3.5

Continuous Update

Cron jobs are the simple and production-proven Linux tool for continous update.
Here's a simple example on how to set them up:
https://www.geeksforgeeks.org/linux-unix/how-to-setup-cron-jobs-in-ubuntu/

All you need to do is take the integration .py file and set up a cron job to run at your data refresh rate, from every day to every 20 seconds.



If your update rate is faster than 60 seconds, be sure to update your Row64 config in:
/opt/row64server/conf/config.json

so that "RAMDB_UPDATE" is set to match the update speed.

Install Pip Libraries

To set up Amazon Redshift integration, install the following pip libraries: 

pip install row64tools
pip install pandas
pip install python-dotenv
pip install psycopg2-binary

Setup For Security

Any security process that works in Python and Linux can be used to secure your data credentials. Our integration code is written to be easily modified to fit your exact needs. The default example is to use .env files to set Linux environment variables. An overview of that approach is here:
https://upsun.com/blog/what-is-env-file/

The most popular Python library for .env is here:
https://pypi.org/project/python-dotenv/

The Amazon Redshift integration code assumes you create a .env file at the same location as your .py file. It sets the variables: DBHost, DBPort, DBUsername, and DBPassword.

Download Amazon Redshift Integration

Row64 Integrations can be downloaded from the github project:
https://github.com/Row64/Row64_Integrations

The Amazon Redshift integration is in the sub-folder:
https://github.com/Row64/Row64_Integrations/tree/master/AmazonRedshift

Note

The integration .py files are intended to be modified or used as a starting point to fit your specific needs.


More help and background information on this database connector can be found at:
https://docs.aws.amazon.com/redshift/latest/mgmt/python-connect-integrate-pandas.html

If you have any problems or requests, please log them at:
https://github.com/Row64/row64tools/issues