Automate OS Premium Data Downloads Using the OS Download API
We’re excited to reveal a much anticipated development in how Ordnance Survey users can access and automate product data downloads.
As part of our drive to make downloads more accessible, we have improved the reach of the Operating system download API include Premium data as good as Open data. This complements the existing offering to give users a full and comprehensive list of products to download through the service. Before we understand what this means, let’s take a look at the OS Download API service in general terms and what it is for.
The service provides a mechanism for you to download the product of your choice, without the need for user interaction, such as manual download through a web browser (on Operating system data center) or by requesting a paper version. The API service offers you the following functionalities:
- Find out which products (as data packages) are available for download
- Request metadata about data packages available for download
- Identify and request data packages for a specific area of interest (AOI)
- Identify and request data packages for a specific format
- Once you have chosen what you want to download, lets you automate said download
This service works in conjunction with OS Data Hub, so all a user would need to do as a prerequisite for using the API service is create a data package (or modify an existing data package). ) of their choice on OS Data Hub (Get help with data packages on OS Data Hub).
After the download API service is explained, the rest of this article will describe an example of using the service to automate the download of a premium data package. There are different methods and tools available to do this and for this example we will be using ETL (extract, transform, load) software called FME to basically ‘call’ the API service and create a workflow to automate downloads.
In our very first step, we’ll need to activate an OS Download API instance by going to the OS Data Hub API page and adding the OS Download API service to a project. .
This will create a API key (given as url) which you can access / copy – take note of this. It will be necessary.
Still in OS Data Hub, it is now a good idea to create or select an existing data package and take note of the Data packet ID that is generated. This can be found in gray text, just below the title of the data packet in the list of data packets window.
Finally, and still in the data packet, write down the Version ID that is generated. This is located at the beginning of the file name of individual file downloads.
With this instance of the service now generated and a data packet ready, we can now start using FME to use the call and automate the download.
To begin with, we’ll use the “Creator” and “HTTPCaller” transformers to initiate a call to the operating system’s download API service. Think of this as a placeholder that we are now asking the API to fill with the information it has.
Specifically in the settings of the ‘HTTPCaller’ transformer, we have to pay special attention to the Request url. This box can be filled in with the API key that you kept in step 1.
As we also know the Data package ID and version ID, we can now start modifying the URL in this area to include a specific data package and version that we have in mind.
To do this, we must:
- Add the word ‘/ dataPackages /’ after ‘v1‘
- Then add the Data packet ID
- Then add the word ‘/ versions /
- Then add the Version ID
- This should then continue with the API key as usual, i.e.? key =
(Nb, the image contains FME user settings for entering information, so it differs from the steps but the principles still apply.)
Connect a ‘JSONFragmenter’ transformer to the HTTPCaller transformer. The reason is that the response provided by the API is embedded in a JSON file. This transformer will allow us to split this file into components that we need to perform a download.
In the settings of the ‘JSONFragmenter’, try to match the image, then run the workflow. The result will provide you with a few columns to examine. You are most interested in the ‘URL’ column where you can view and copy a specific URL for downloading the file.
Connect a new ‘HTTPCaller’ transformer to the ‘JSONFragmenter’ transformer and in the ‘Request URL’ area of the new ‘HTTPCaller’ transformer settings, add the specific URL that you may have copied in step 3.
At this point, you may want to configure the remaining part of the ‘HTTPCaller’ settings, including the ‘Output folder’ box, that is, where you want to save the file, such as a file location. folder on your local hard drive.
Running the workflow now will download your data package.
Step 5 – Optional
You can use the additional attributes that were exposed in step 3. By checking the ‘fileName’ and ‘size’ columns against the downloaded file that you have, you will ensure that the download worked correctly and did not. been corrupted during the Download process.