Live Layers – Real Time Data Updates with the CartoVista REST API and Python

Tips&Tricks
Author

Want to use live-time data for a layer on your map?

With CartoVista, you can easily integrate live-time data to a layer using the CartoVista REST API and a simple Python and batch script. The layers will automatically update depending upon how frequently you opt to run your Python script. 

The Data

For this tutorial, we will be using the USGS’s Earthquake real-time data API on Turkey to collect daily updates on Earthquakes that take place in the country for the past 30 days as of the update. We will be using Python’s Pandas, GeoPandas, JSON, and Requests libraries to do the following:

  • Collect the data from the USGS
  • Store them into a geodataframe
  • Create a shaped zipfile with the date of extraction
  • Post the data layer to CartoVista
  • Write a Python script to update the layer in CartoVista
  • Write a batch script to run the Python script daily

Libraries and Packages

To begin, let’s set our working directory and import the relevant libraries as shown below.

Imports and Working Directory for Data Collection.

Acquiring The Data

Now, we will create a GET REQUEST to collect earthquake data in Turkey for the past month as of the day we make the request. Since the data is GeoJSON and we want the features, we will collect JSON data and inspect the features to view the entirety of the data (earthquakes in the past 30 days).

Acquiring Earthquake Data from USGS

Data Cleaning

The data we collected is in JSON format, but we want a dataframe as a shapefile so that we can map it on our CartoVista web map. We will convert the JSON data into a geodataframe with a loop. Inside the loop, we will do the following:

  1. Set the coordinate reference system to 4326
  2. Separate the latitude and longitude coordinates
  3. Filter the data to acquire the magnitude, location, time, status and geometry (latitude and longitude)
  4. Convert the time with .strftime() to get date (time) of earthquake in “mm-dd-yyyy” format
  5. Filter with the ‘place’ feature to acquire earthquakes only happening in Turkey 
Filtering and Storing Turkey's Earthquake Data into a Dataframe

Exporting the Data Locally

We’ve got ourselves some data on earthquakes in Turkey for the past month! However, our data is not exported and stored in a CartoVista accepted format (zipped Shapefile or CSV). To keep things simple and easy to distinguish when storing in our working directory, we will name our file the same as the date the script is run. The extension of this file will be a shapefile since it won’t require any geocoding in CartoVista. Finally, we will export this dataset with new quakes appended daily and store them as a zip file.

Creating and Storing Shapefile Locally

Uploading Data to your CartoVista Workspace

The data is ready on our machines. We can move to using the CartoVista API to make our data layer live. But first, we need to upload the layer to our CartoVista workspace as a data layer.

Upload a layer to the CartoVista workspace

Generating Your CartoVista API Key

Additionally, we need our API key to be able to send data into our workspace. This can be accessed by selecting on the “Users” tab in our workspace, selecting on our user’s name and selecting the “API key.” Here we can create our API key and copy it to our clipboard. Ensure that the key is “active.”

Use the CartoVista REST API to update the layer

Let’s head back to our Python script. We will now store the server URL and API key into variables. The file we just uploaded to our workspace containing the earthquake data will also be stored as a variable and opened for reading based on where it is stored in our local computer. 

It’s time to make a POST REQUEST. We wil use two CartoVista API endpoints :

  1. getFromFileDescription: this endpoint will allow us to send the file and get as a response the structure of the file description in CartoVista
  2. updateFromFileDescription: we will send this file description in this endpoint to finalize the update.

 The CartoVista Swagger interface shows you the list of all endpoints, and for each endpoints, the list of all parameters.

So, in recap, here’s what it looks like for the first endpoint:

python script API parameters

To finalize our update, we will conclude our Python script by making the second POST REQUEST. To check if the REQUEST is valid, we can print it so see if there are any errors. 

 

python script API parameters second endpoint

Automating the Script

The script is ready to go and replicable for any live-time data, but we need to be able to automate the process of running the script frequently without having to open the Python script and manually run it. For this, we wrote a simple batch script that is scheduled to run the Python script everyday at 9:00 a.m. with the Task Scheduler on Windows. Depending upon your needs for updates, you can set the task scheduler to run your script as frequently as you need (daily, weekly, monthly, etc.).

Final Map - Live-Time Map of Earthquakes in Turkey

The following map visualizes earthquakes as they happen daily across Turkey as of February 2023 using the USGS Earthquake Data API. Clusters are created for regions where earthquakes happen in large volume. Check daily after 9:00 a.m. EST for updates! 

Discover our Turkey-Syria Earthquakes Story Map
cartovista logo

Want to create maps like this yourself?

Try for free